Top-header-tutorial1
Return To Portal Search and Retrieve NOAO Data PI & Co-I Data Access Data compression Archive file names
How principal investigators can get their NOAO data

This tutorial explains how principal investigators (PIs) of NOAO observing programs can access and retrieve data from the NOAO Science Archive. The main steps are:

  1. Archive registration
  2. Login
  3. Find your data
  4. Stage your data for retrieval
  5. Retrieve your data by FTP
  6. Working with your downloaded data:
    • uncompressing data
    • restoring filenames

As PI, you can also authorize co-investigators to access your proprietary data:



1. Archive registration

You do not need to register or have an account in order to retrieve non-proprietary data from the NOAO Science Archive. However, if you are PI for an NOAO observing program, you must be registered in order to access and retrieve your proprietary data.

PIs and co-Is of approved NOAO observing programs are automatically registered with the Archive and assigned a username. This is generally your last name followed by your first initial, e.g., Tycho Brahe would be "brahet". When you are first registered with the Archive, you will receive an e-mail with your username and password. If you cannot remember your username or password, or are not sure if you are registered, there are links on the Login page that will send you reminders.

PIs are automatically granted access to their proprietary data when they sign into the Archive with their username. In future semesters, when you are PI for new NOAO observing programs, proprietary access to those data will automatically be assigned to your existing Archive username. You do not need to re-register or do anything else to ensure access. Co-investigators must be authorized by the PI in order to access proprietary data.

If you encounter problems signing in to the Archive, or with accessing proprietary data, please contact sdmhelp@noao.edu.

In the past, the NOAO Archive used the National Virtual Observatory (NVO) single sign-on (SSO) prototype for user registration. This has now been replaced with the new NOAO Archive login system. However, if you have previously used your NVO username to sign in to the NOAO Archive, you can still do so. At the bottom of the Login page, you can click the link marked "NVO SSO Log in". This will take you to the NVO login page at https://sso.us-vo.org/, where you can sign in.

However, even if you have previously used your NVO login with the NOAO Archive, you will have also been assigned a new NOAO Archive username, as described above. We encourage you to switch over to using your NOAO Archive username instead of the NVO login.

Login


2. Log in to the Archive

There are several ways to log in to the Archive:

  • Go to the Archive Portal and click on the Log In link near the top right side of the page.
  • Or, click the button marked Proprietary data access (login required).
Proprietary
In the login form that appears, enter your username and password. If you do not know or have forgotten these, you can click the appropriate links and you will be sent a reminder.

If you wish to sign in with your NVO user ID instead, click the link at the bottom of the sign-in form. This will direct you to the NVO single sign-on (SSO) page, where you can log in. Then, you should be redirected back to the NOAO Archive. In the past, this was the standard method for accessing proprietary data from the NOAO Archive, but we now recommend that you use your assigned NOAO Archive username and password instead.

Login_form

Once you are logged in, you should see your username at upper right portion of the page, next to the Log out link.

Logout


3. Find your data

  • In the top menu toolbar, select Search NOAO Data to go to the search form.
  • From the Search Type menu, select Search My Data. This restricts the search to only those data that you own. If you have not already logged in, you will be directed to the NVO sign-on page, then back to this form.
Searchtype_searchmydata
Dataproducts

4. Stage your data for retrieval

The results of a query are displayed in a table, which may span many pages if you have many files. There are several ways to retrieve your data. Here, we describe the steps for sending the files from your query results to a staging area and retrieving them.

A single query may return up to a maximum of 50000 data sets, but there is an upper limit of 1000 files that can be staged for retrieval at any one time. Your observing program may have resulted in more than 1000 (or even 10000) files, particularly if you are using an instrument with a rapid observing cadence and high data rate, like NEWFIRM, or if you are retrieving all raw and pipeline-reduced data products (the DECam, Mosaic and NEWFIRM pipelines can produce several reduced data products for each raw image). If you wish to download more than 1000 files, you will have to do this in steps; we will describe that process below.

Selectmydata

Staging your data:

  • Above the table of query results, use the Selection menu to "Select all rows". (You may also select subsets of files; see help on data selection for more information.)
  • Click: 'Stage selected rows' (also above the Results table)
  • You should be automatically taken to the Staging Area. You can go there any time by clicking on the Staging Area tab at upper left.
  • Here you will be able to monitor progress of staging (e.g., in the 'Image Staging Status' box). Staging is generally very quick, even for a large number of files.
Stagingarea_status

If you wish to retrieve more than 1000 files:

You have several options.
  • Restrict your query: Perhaps the easiest option is to restrict your query using other qualifiers, such as a range of dates. For instruments with high data volumes, like NEWFIRM, you may wish to query your data one or two nights at a time in order to limit the number of files to 1000 or less.
  • Stage subsets of your files: You can use the Selection menu at the top results grid to select or deselect files one page (20 files) at a time. At the top of the query results table, you can navigate among the pages of results, selecting the files that you wish to stage, up to a limit of 1000 files. You may find it useful to sort or categorize your data according to the parameters in one of the query result columns (e.g., Proposal ID, Start Date, Filter, Data Product Type, etc.). When you have selected the files you want, hit the button marked Stage selected rows.
  • Bulk download using 'cURL': As an alternative, you may fetch your data directly from the Archive using a utility like 'cURL'. This bypasses the use of the FTP staging area, and can be used to retrieve more than 1000 files at a time.

5. Retrieve your data by FTP

You can use any FTP client to login to the hostname shown above to download your data from the FTP site and staging directory given above, using the FTP username and password that were provided.

Note: You must use plain FTP (and not SFTP) to download your data from the staging area.
Be sure to select binary file transfer, or your data may be unreadable after downloading!

The Download tips in the Staging Area also give instructions for using lftp, which provides faster parallel transfer and can considerably speed up download times for large volumes of data. You may need to install lftp, which is available from most standard software repositories, and to download an lftp configuration file and save it as ~/.lftprc. Then, follow the instructions provided in the Download tips.


6. Working with your downloaded data

Data compression:

FITS data stored in the NOAO archive are compressed in order to save space and to speed download transfer. Before 2010, data were compressed using gzip. From semester 2010A on, the Archive has begun to use "tile compression", which is a method of handling data compression within the FITS standard (rather than externally compressing existing FITS files). The NOAO Archive is now using the Rice compression algorithm to create tile-compressed FITS images; this is substantially faster than standard gzip and achieves greater compression factors. The tile-compressed data files are recognizable by their ".fz" extension.

Archive filenames:

Data from NOAO telescopes and instruments are assigned unique filenames when they are stored in the NOAO Science Archive. However, the new archive filenames are generally not very informative. If you are the PI of an observing program, you may find it useful to rename the data files that you retrieve from the archive to the names they had at the telescope.

How to give co-investigators access to your data

The Principal Investigator of an NOAO observing program may authorize selected co-investigators to access and download proprietary data. To do this, select PI Admin from the top menu bar. You must be logged in with your NVO username to use this feature; if you have not yet logged in, you will be directed to the NVO sign-on page, and then back to the PI Admin page. Piadmin

Piadmin_authorization_after On the left side of the PI Admin page you will see a list of the NOAO programs for which you are PI. Select one of these programs, and at right you will see a list of the project co-investigators. (Co-investigator information may not be available for older NOAO programs where the proprietary periods for the data have already ended.)

To authorize co-investigators to access proprietary data from this program, check the boxes next to their names. Then, click the button marked "Authorize and notify seleced Co-Is". A pop-up window will ask you to confirm that you really want to make this authorization. When you answer yes, the NOAO Science Archive will send an e-mail to the selected co-investigators notifying them that they have been authorized to access data. If they have already registered with the NVO, they will be able to log in, then find, stage and retrieve your data immediately. If they have not yet registered, they will be sent an invitation to register, and can follow the steps described in the tutorial on user registration.