Thoughts on Data Compression:

1. Thoughts on Data Compression: • Data compression of high spectral IR data is very important. Others think so too (ORA, ARL, etc). • Data compression can have applications in a number of areas: • downlink • rebroadcast • distribution • archive • The nature of the data must be understood to best compress and analyze the impact on products • Due to the natural redundancy in the spectra, lossy data compression should be considered (if needed). • UW/CIMSS is uniquely qualified to do the impact of lossy compression on a number of derived products (retrievals, clouds, etc).

2. Hyperspectral IR Data Compression Where we were then… Where we are now… Where we are going…

3. Hyperspectral IR Data Compression • Where we were then… • Guessed at lossy and lossless values for hyperspectral data, hoped for 10X and 2X • Only had experience with (time consuming) principle component work • No standard datasets for comparisons of various methods

4. Hyperspectral IR Data Compression • Where we were then… • Knew there were several potentail uses: • Downlink • Rebroadcast • Archive (both short and long-term) • No optimized algorithms for hyperspectral IR sounder data • No studies done to demonstrate compression impacts on products

5. Hyperspectral IR Data Compression • Where we are now… • Several optimized algorithms for hyperspectral IR sounder data • Begins of studies demonstrating compression impacts on products • Developed and have shared standard datasets for comparisons of various methods • Can compare to and more experience with principle component work

6. Hyperspectral IR Data Compression • Where we are now… • Preliminary estimates for lossless values for hyperspectral data via a number of methods, showed that at least 2X should be very possible • Preliminary estimates for lossy values for hyperspectral data via a number of methods, showed that 5-6X should be very possible • Modified wavelet-based algorithms so we can compress on “irregular” (non-2^n) grids

7. Hyperspectral IR Data Compression • Where we are now… • Understand more of how the compression ratio is a function of bit depth • Developed a new pre-processing step to improve the compression ratio of any existing scheme. • Began impact of lossy compressions on products

8. Hyperspectral IR Data Compression • Where we are now… • Investigated several state-of-the art algorothms on hyperspectral data (ISO image compression standards – JPEG2000, JPEGLS) • Adopted and tested several lifting schemes: • 45 in 1-D • 9 of which were also converted to 3-D • Working with a new Cooperative Institute: CREST

9. Hyperspectral Data Compression • Where we are going… • Continue to study the impact of lossy compressions on data and core (and new) products • Validate results on near global observations • Process in instrument “count-space”, not just the radiance space (eg, “raw data”) • Process on interferometric data, not just grating-type. • Reach out to others with similar interests/needs.

10. Hyperspectral Data Compression • Where we are going… • Trade-off studies (CPU, memory, Compression Ratio, etc) to recommend what types of compression algorithms are strong candidates for GOES-R • Continue PCA work

11. NOAA/NESDIS GOES-R Hyperspectral Sounder Data Compression Studies Bormin Huang, Allen Huang, Alok Ahuja and Kevin Baggett Cooperative Institute for Meteorological Satellite Studies University of Wisconsin-Madison March 2004 UW-Madison

12. Satellite Satellite “Archive” 2:1 10:1 Users GRB SYSTEM “Downlink” 2:1? “Re-broadcast” 6:1? SOCC/ Backup Any GRB site “Data Pool” “Data pool” of full GOES (the compete level 1b data) would not be compressed, nor rebroadcast. Full GOES would be available for ground-based transfers.

13. How might we move from 100 to 24 Mbps for a rebroadcast? • Maintain a "data pool" concept for land access and archive of • uncompressed sensor data • Users could then "test what is not being sent“ for their own application(s) • One method going from ~100 Mbps to 24 Mbps: • 10X for the 0.5 km visible band • 6X for the "1km” bands • 2X for the IR bands of the ABI • 6X for the HES-IR • Means approximately half of the band width would be used by the imager and half by the sounder; each image could be sent out

14. Selected AIRS granules on Sept. 6, 2002 GOES will not see data from two of the most extreme (polar) datasets

15. AIRS Radiance Field Five of the sample AIRS granules (IR window)

16. AIRS Radiance Field Five of the sample AIRS granules (IR window)

17. Lossless CompressionComparison of Methodologies JPEG-LS and JPEG-LS give the best compression ratios

18. Lossless Compression JPEG-LS at various bit-depths The compression ratio is a function of the bit-depth

19. Lossy Compression JPEG-LS RMS error A compression ratio of 5 is less than the instrument noise. A pre-processing step improves the performance.

20. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

21. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

22. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

23. Summary • What has been done • Generic Data cubes prepared for HES compression studies • Various 3D wavelet transforms tested on real data • Various 3D wavelet tree coding developed and tested • Regular-sized vs. irregular-sized wavelet tree encoders/decoder have been implemented • Lossless and Lossy compression implemented • Predictor based encoders tested on real data • PVQ implemented and investigated on real data • Results 1. The performance rank from best to worst in terms on compression ratios is given in the order of JPEG-LS, JPEG2000,3D SPIHT, CALIC, BWT and 3D EZW. 2. JPEG-LS and JPEG2000 are applied for lossy compression and both have rms errors below the noise threshold.