Return To Portal Search and Retrieve NOAO Data PI & Co-I Data Access Data compression Archive file names
Downloading data directly from the NOAO Archive using cURL
The tutorial on how to Search and Retrieve NOAO Data describes the standard procedure used to search for data in the archive and to stage it for retrieval using FTP.

Alternatively, you can fetch your data using cURL, a command line tool that allows batch downloads of data directly from the archive without going through the FTP staging area. In certain circumstances, this may be faster or more robust than standard FTP. Here we describe the procedure for using cURL to retrieve data from the NOAO Archive.

Scripts and certificates that you will need in order to retrieve data this way are available in the NOAO Archive contributed software repository.

1. Obtain a copy of cURL

cURL is a command line tool for transferring files using URL syntax. It is provided by most Linux systems but supports other systems as well.

2. Read the secure_get instructions is a script that uses cURL and an SSL certificate (which is necessary if the data are proprietary) to retrieve a list of archive files specified by their URIs (Uniform Resource Identifiers). The procedure for creating a list of URIs for archive data is described in step 5, below.

3. Download the secure_get script

4. To retrieve proprietary data, get an SSL certificate

The SSL certificate encodes your username and login information so that the secure_get script can retrieve your proprietary data from the NOAO Archive.
  • You must be logged in to the Archive in order to get an SSL certificate to retrieve proprietary data. Go to the Archive home page and click the Login link at the upper right in order to sign in.
  • At the Archive's contributed software repository, click on Get your certificate. Your certificate will download automatically, and be saved on your local disk as an encrypted file with a name like username.cert (where username will be your Archive username, e.g., copernicusn.cert if you are Nicolaus Copernicus).
  • SSL certificates are valid for a 2 week period, then expire for security reasons.

If you wish to retrieve only public (non-proprietary) data, you can skip this step.


5. Query the NOAO Archive the data you wish to retrieve

Search the NOAO Archive for the data you wish to retrieve following standard procedures described in the tutorial on how to Search and Retrieve NOAO Data:
  • Fill out the Query Form and click Search
  • On the Results page, select the data you wish to retrieve by clicking the checkboxes for individual data sets, or by using the Selection pulldown menu to select all results.
Next, instead of staging your data for FTP retrieval:
  • Click Download selected rows as VOTable.
This will save the search results as an xml file with a name like rows_as_votable_1303942326_4420.vot.xml

6. Retrieve your data using curl, your SSL certificate, and your query result VOTable

From your local computer, execute the following commands:
    % chmod +x
    % ./ rows_as_votable_1303942326_4420.vot.xml --cert copernicusn.cert
Substitute the actual name of your downloaded VOTable and SSL certificate.

If you are retrieving only non-proprietary data, you can leave out the --cert option.