AIRS Data Decimation for Climate Trending

1. AIRS Data Decimation for Climate Trending 30 March 2004 Hartmut H. Aumann

2. The excellent data quality demonstrated by AIRS during the first 18 months of data promises some potentially very interesting contributions to climate trending. Key to successful climate trending using AIRS data is to make the data readily available to the climate research community. The AIRS spectrometer produces 1.3Mbits/sec = 14 GBytes/day The AIRS PGE produces about 12 GBytes/day L2 37 GBytes/day L1b If one day of L1b data is requested from the DAAC over a 1MByte/sec internet connection, it would take 14 hours to transmit. The expected 7 years of AIRS data would require 80 TBytes storage. At an internal 10MByte/sec transfer rate it would take 92 days to read. Access difficulties will hamper the effective and wide spread use of the AIRS data, no matter how good the data are.

3. The approved AIRS project plan identifies three methods to deal with • this issue: • Write out the data more efficiently than is done currently without • loss of information (presentation by Evan Manning) • Generate level 3 products which provide globally mapped means • suitable for climate research (presentation by Steph Granger) • Generate spatial, spectral and/or temporal level 1b data subsets • suitable for climate research

4. Generate spatial, spectral and/or temporal level 1b data subsets suitable for climate research. Goal: A 7 year record which can be read into a Unix computer at 10 MBytes/sec in one day = 1 TByte Currently there are at least four types of subsets being generated, which could be evaluated, augment and promoted to routine production status at the DAAC. We discuss the four and propose augmentations for routine production at the DAAC

5. NESDIS BUFR file NESDIS Radiance 0.5x2 degree gridded product (all channels,center fov) NESDIS Radiance 3x3 degree gridded product (entire “golfball”) NESDIS AIRS Level 2 Support file @ 0.5 x 2 degree gridded product UMBC clear ocean subset JPL calibration clear ocean subset

6. JPL calibration clear ocean subset The JPL calibration clear ocean subset was optimized to support the long term trending of the AIRS radiometric and spectral calibration. This file is generated every day. It saves for every “reasonably clear” AIRS footprint 70 “key” AIRS channels (2 per module ++) the matching mean and stdev vis1, vis2 and vis3 the matching AMSU 1,2,3,4,5,6 and 15 the matchup with the RTGSST. Typical 100,000 points per day Average size (not bzip compressed) 18 Mbytes/day Clear biased data subset and climate application may conflict Factor 1800 less than the current 32 GBytes/day Seven year volume 0.05 TByte

7. The AIRS sst2616 tracks the RTG.SST at night with 0.64K bias and 5mK/year stability. rms=0.4K uncorrected night bias -0.62K 0.35K of the cold bias is due to the day night bias in the RTG.SST and the skin effect. The remaining 0.28K bias is due to residual cloud contamination

8. UMBC clear ocean subset Every fairly “clear” ocean spectrum is saved for all 2378 channels in rtp format plus the ECMWF matchup T(p), q(p) and TSurf Typically 20,000 spectra per day 500 Mbytes/day in 240 files Clear biased data subset and climate application may conflict Factor 60 reduction in data volume compared to current level 1b Seven year volume 1.3 TBytes

9. NESDIS 0.5x2 degree gridded product All 2378 channels are saved for the first spectrum which falls into the 0.5 x 2 degree grid for ascending and descending orbits. Unbiased 720 x 90 gridpoints * 2 * 2378 points * 2 bytes/point = 616 MBytes in two files per day Factor 54 volume reduction compared to current L1b Seven year volume 1.6 TBytes All 2378 channels for first 3x3 golfball in 3x3degree box same.

10. NESDIS AIRS BUFR file For every second AMSU footprint all 15 AMSU channels and 340 AIRS channels are saved (i.e. every 0.53 seconds) Each day 163,000 points * 370 samples * 2 bytes/sample = 120 MBytes/day Factor 270 decrease in data volume Seven year volume 0.3 TBytes Currently this file is not saved at NESDIS

11. Enhanced 0.5x2 degree gridded product Save 400 channels (with AMSU and vis) from the first 3x3 golfball which falls into the grid for ascending and descending orbits. unbiased 720 x 90 gridpoints * 2 * 9 * 400 points * 2 bytes/point = 933 MBytes in two files per day Factor 34 volume reduction compared to current L1b Seven year volume 2.4 TBytes

12. Augmented “clear” subset For every day save for every “reasonably clear” land and ocean AIRS footprint all AIRS channels the matching mean and stdev vis1, vis2 and vis3 the matching AMSU 1-15 the matchup with the surface analysis (RTGSST over ocean). Typical 150,000 points per day Average size (not bzip compressed) 800 Mbytes/day Clear biased data subset and climate application may conflict Factor 40 less than the current 32 GBytes/day Seven year volume 2.0 TByte

13. For climate every eighth day may be adequte

14. 8th day sampled 0.5x2 degree gridded product Save all 2378 channels (plus AMSU and vis) from the first 3x3 golfball which falls into the grid for ascending and descending orbits. Unbiased 720 x 90 gridpoints * 2 * 9 * 2400 points * 2 bytes/point = 5.6 GBytes in two files every eighth day Factor 34 volume reduction compared to current L1b Seven year volume 1.8 TBytes

15. Proposal/Plan • With the version 4.0 PGE we include the production of • at least five subsets in support of climate (not counting L3) • All channels all footprints compacted to 2 bytes. • compressed to 1 byte. 2400 bytes per spectrum • All channels all footprints 80 static principle components • saved as 400 bytes per spectrum. • The NESDIS 0.5 x 2 degree gridded product for every day • The NESDIS 3x3 degree gridded product for first golf ball • for all 2389 channels • The all channels “clear” land and ocean set every day • The climate data subsets are reprocessed to generate a • complete record starting in September 2002.