Return To Portal Search and Retrieve NOAO Data PI & Co-I Data Access Data compression Archive file names
Working with compressed data from the NOAO Science Archive
FITS data stored in the NOAO Archive are compressed in order to reduce storage space and to speed downloading to users. Data archived prior to observing semester 2010A were compressed using the familiar gzip algorithm. New data from semester 2010A onward are stored using "tile compression", which is described here. If you download data from the NOAO Archive, you may receive files compressed in either format, depending on when the data were taken and archived.

What is the '.fz' format?

Astronomers have long used Unix commands like gzip to compress their data. Gzip was designed to work with text files and does a poor job at best with astronomical data. Other programs like bzip2 may do a better job of compressing data, but are woefully slow. More fundamentally, a gziped FITS file is no longer FITS and is unreadable with many familiar FITS tools and applications.

Given all this, the FITS working group created the 'Tile Compression' standard in 2001. Tile Compression is a nifty way to handle the bookkeeping for compressing data within the FITS standard itself. This is similar to other graphics standards like jpeg, gif and png, which include compression as an integral part of the format. Tile Compression has long been supported by the widely used CFITSIO library.

The '.fz' extension indicates a FITS tile-compressed file, for example: a001.fits.fz.

One very useful feature of this method is that the image headers remain readable for tile-compressed data. The division of the image into small rectangular tiles also permits rapid access on a line-by-line basis without having to uncompress the rest of the pixels.

Users will find most '.fz' files will have been compressed using the Rice algorithm. Rice compression has been used for many years to achieve high compression ratios for space missions. It is also very fast. Rice has been benchmarked in the NOAO Archive at about 10 times as fast as gzip.

How do I read and write '.fz' files?

FPACK and FUNPACK are convenient, stand-alone tools for compressing and uncompressing FITS files using the tile-compression standard, analogous to gzip and gunzip. They are available for all platforms supported by CFITSIO.

Note: Older versions of FUNPACK may show an error when processing, reporting "FITSIO status = 415; unknown error status". If this happens, please try upgrading to the latest version from the HEASARC distribution.

If you have received tile-compress '.fz' files from the NOAO Archive, these may be uncompressed into the original FITS format using the FUNPACK command:

      % funpack cp5001776.fits.fz
      % ls -l cp*
      -rw-r--r-- 1 bits bits 2396160 Jan 28 18:04 cp5001776.fits
      -rw-r--r-- 1 bits bits 1198080 Jan 28 18:03 cp5001776.fits.fz
By default, FUNPACK and FPACK retain the original file in the same directory. As usual, there are multiple options:
      % funpack -H
      funpack, decompress fpacked files.  Version 1.4.1 (Jan 2010) CFITSIO version 3.240
      usage: funpack [-E <HDUlist>] [-P <pre>] [-O <name>] [-Z] -v <FITS>
      more:   [-F] [-D] [-S] [-L] [-C] [-H] [-V]
      Flags must be separate and appear before filenames:
      -E <HDUlist> Unpack only the list of HDU names or numbers in the file.
      -P <pre>   Prepend <pre> to create new output filenames.
      -O <name>  Specify full output file name.
      -Z          Recompress the output file with host GZIP program.
      -F          Overwrite input file by output file with same name.
      -D          Delete input file after writing output.
      -S          Output uncompressed file to STDOUT file stream.
      -L          List contents, files unchanged.
      -C          Don't update FITS checksum keywords.
      -v          Verbose mode; list each file as it is processed.
      -H          Show this message.
      -V          Show version number.
      <FITS>  FITS files to unpack; enter '-' (a hyphen) to read from stdin.
      Refer to the fpack User's Guide for more extensive help.

Since FPACK creates these files, there are even more options to select for different compression algorithms and so forth. Please see the User's Guide.

Support for tile-compression is built-in to the CFITSIO library. A program linked against a recent version of CFITSIO already has the possibility of reading (or writing) '.fz' files:

The IRAF FITSUTIL package now includes support for FITS Tile Compression:

As use of this format expands, we anticipate more community software packages will feature support for FITS Tile Compression. As with jpeg, the ultimate goal is not to separately compress and uncompress each file to restore an original FITS file, but rather to have the ability to maintain the data in its compressed state throughout a processing workflow.

For additional information:

More information about FITS tile compression, with pointers to documents that describe other important features of this procedure, can be found in an article by Rob Seaman titled What is FITS Tile Compression? in the March 2010 NOAO Newsletter .