|Working with compressed data from the NOAO Science Archive|
What is the '.fz' format?
Astronomers have long used Unix commands like gzip to compress their data. Gzip was designed to work with text files and does a poor job at best with astronomical data. Other programs like bzip2 may do a better job of compressing data, but are woefully slow. More fundamentally, a gziped FITS file is no longer FITS and is unreadable with many familiar FITS tools and applications.
Given all this, the FITS working group created the 'Tile Compression' standard in 2001. Tile Compression is a nifty way to handle the bookkeeping for compressing data within the FITS standard itself. This is similar to other graphics standards like jpeg, gif and png, which include compression as an integral part of the format. Tile Compression has long been supported by the widely used CFITSIO library.
The '.fz' extension indicates a FITS tile-compressed file, for example: a001.fits.fz.
One very useful feature of this method is that the image headers remain readable for tile-compressed data. The division of the image into small rectangular tiles also permits rapid access on a line-by-line basis without having to uncompress the rest of the pixels.
Users will find most '.fz' files will have been compressed using the Rice algorithm. Rice compression has been used for many years to achieve high compression ratios for space missions. It is also very fast. Rice has been benchmarked in the NOAO Archive at about 10 times as fast as gzip.
How do I read and write '.fz' files?
FPACK and FUNPACK are convenient, stand-alone tools for compressing and uncompressing FITS files using the tile-compression standard, analogous to gzip and gunzip. They are available for all platforms supported by CFITSIO.
|Note:||Older versions of FUNPACK may show an error when processing, reporting "FITSIO status = 415; unknown error status". If this happens, please try upgrading to the latest version from the HEASARC distribution.|
If you have received tile-compress '.fz' files from the NOAO Archive, these may be uncompressed into the original FITS format using the FUNPACK command:
% funpack cp5001776.fits.fz % ls -l cp* -rw-r--r-- 1 bits bits 2396160 Jan 28 18:04 cp5001776.fits -rw-r--r-- 1 bits bits 1198080 Jan 28 18:03 cp5001776.fits.fzBy default, FUNPACK and FPACK retain the original file in the same directory. As usual, there are multiple options:
% funpack -H funpack, decompress fpacked files. Version 1.4.1 (Jan 2010) CFITSIO version 3.240 usage: funpack [-E <HDUlist>] [-P <pre>] [-O <name>] [-Z] -v <FITS> more: [-F] [-D] [-S] [-L] [-C] [-H] [-V] Flags must be separate and appear before filenames: -E <HDUlist> Unpack only the list of HDU names or numbers in the file. -P <pre> Prepend <pre> to create new output filenames. -O <name> Specify full output file name. -Z Recompress the output file with host GZIP program. -F Overwrite input file by output file with same name. -D Delete input file after writing output. -S Output uncompressed file to STDOUT file stream. -L List contents, files unchanged. -C Don't update FITS checksum keywords. -v Verbose mode; list each file as it is processed. -H Show this message. -V Show version number. <FITS> FITS files to unpack; enter '-' (a hyphen) to read from stdin. Refer to the fpack User's Guide for more extensive help.
Since FPACK creates these files, there are even more options to select for different compression algorithms and so forth. Please see the User's Guide.
Support for tile-compression is built-in to the
library. A program linked against a recent version of CFITSIO already
has the possibility of reading (or writing) '.fz' files:
The IRAF FITSUTIL package now includes support for FITS Tile Compression: