An overview of real-time radar data handling and compression

1. An overview of real-time radar data handling and compression Valliappa Lakshmanan National Severe Storms Laboratory & University of Oklahoma http://cimms.ou.edu/~lakshman/ lakshman@ou.edu

2. Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary lakshman@ou.edu

3. Operational radar network • Weather Surveillance Radar 1988 Doppler (WSR-88D) • 142 radars in CONUS • 11 in Hawaii Alaska • S-band • Reflectivity to 460km • Velocity to 230km lakshman@ou.edu

4. Operational radar data • Each radar operates according in one of several volume coverage patterns • NWS forecast offices choose VCP for “their” radars • All VCPs involve several elevation angles • Resolution (current) • 0.95 degrees • 1km (ref) • 0.25 km (velocity) • Tilt in 20-60s • Volume in 4-10 min • Super-resolution • 0.25km x 0.5 deg • Faster VCPs lakshman@ou.edu

5. Radar data: levels • Level-I (or I&Q data) • Signal processed at radar • 64 values at each range gate • Not transmitted/stored routinely • Level-II (moment data) • Reflectivity, Velocity, Spectrum Width • 3 scalar values at each range gate (pixel) • Distributed via LDM (more on this later) • Level-III (products) • Produced by algorithms in the Radar Product Generator • Distributed via NOAA Port • Somewhat obsolete, now that Level-II is available in real-time • Better products can be created in real-time off-site lakshman@ou.edu

6. Size of radar data • Focus on Level II • Size of radar data depends on the VCP • Clear-air VCPs have fewer (4-5) tilts collected more slowly • 7.1 MB in 990 seconds for VCP 31 • 0.6 GB per day • Bandwidth: 7.3 kB/s • Storm-type VCPs have more tilts, therefore more data • 15 MB in 460 seconds for VCP 12 • 2.8 GB per day • Bandwidth: 33.4 kB/s • A CONUS radar processing system needs to process 142 radars • On a typical storm day, 1/3 of CONUS radars are in storm mode: 200 GB • Mean: 60.6 GB/day when compressed • Bandwidth: 736 kB/s (peak: 1771 kB/s) lakshman@ou.edu

7. Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary lakshman@ou.edu

8. Real-time Distribution lakshman@ou.edu

9. Distribution: components • Radar Interface and Data Distribution System (RIDDS) • Taps into radar stream at RPG • Transmits over NWS network to regional server • Regional servers on Abilene network • Load dependent on geography • Southern region average bandwidth: 484 kB/s • Western region average bandwidth: 74 kB/s • Real-time data may be requested via Local Directory Manager (LDM) • Developed by Unidata • Peer-to-peer realtime data distribution • Top-tier providers: Oklahoma, Purdue, Maryland • Transmission is in chunks of radials (all 3 moments) • 50 radials at a time • Compressed using bzip2 lakshman@ou.edu

10. Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary lakshman@ou.edu

11. bzip2! • Custom compression methods can work better than bzip2 • Example: V. Lakshmanan, “Lossless coding and compression of radar reflectivity data,” in 30th International Conference on Radar Meteorology, pp. 50–52, American Meteorological Society, (Munich), July 2001 Not further compressible gzip bzip2 Linear prediction, Run length encoding, Huffman coding lakshman@ou.edu

12. So why bzip2? • Order of improvement of custom compression over bzip2: 10% • Maintenance of compress/uncompress software has significant costs • Every user of radar data would have to implement uncompress code • Costs are spread out over many organizations: really adds up • 10% decrease in bandwidth/space not worth the increased cost • Would be different if this would bring bandwidth below some cost point • Example: T1 line or dial-up? • Choose among publicly available, non-proprietary compression formats • gzip, bzip2, etc. • Burrows-Wheeler (bzip2) performs best, so chosen lakshman@ou.edu

13. Preprocessing • Some preprocessing done at radar site to improve compressibility • Level I: signal-to-noise threshold • Level II: height threshold • Both done at radar site • Part of radar system • No cost on part of users • Compression algorithm is still bzip2 • Theoretically “lossy” compression • But loss is on part of image not useful to end-users (JPEG?) lakshman@ou.edu

14. Compressibility • But … • Compression has uses beyond saving storage and bandwidth • The ideas behind compressibility have wider use lakshman@ou.edu

15. Instrument artifacts Mostly good data Instrument artifact lakshman@ou.edu

16. Quality control of radar data • QC needs to be done before data are fed to automated algorithms • Done by means of machine intelligence algorithms • Helps distinguish ground clutter, biological contamination etc. • But … • Hard to distinguish between stratiform rain and instrument artifacts • Both fill the radar volume • With reflectivity values around 30-50 dBZ • Velocity and spectrum width do not help • Texture-based quality control doesn’t help • Reflectivity fields in both cases are locally smooth lakshman@ou.edu

17. Shannon entropy • Theoretical maximum compressibility based on information content • No need to actually compress file • Simply compute Shannon’s information measure (entropy) • The information content in the two fields (globally) is different • Stratiform rain has high information content • Even if most of the radar volume is not filled • Instrument artifacts have low information content • Even though most of the radar volume is filled • Can easily detect changes in entropy • Reject instrument artifacts even if it comes during a time period of rain lakshman@ou.edu

18. Real-time radar data handling and compression Data characteristics Real-time handling Compression Summary lakshman@ou.edu

19. Summary • Radar data is transmitted routinely in real-time to interested users • Compressed in chunks of 50 radials each • Peer-to-peer sharing (software: LDM) • Compressed using bzip2 • Custom compression techniques are not cost-effective • Perform better than bzip2 • But incremental (10%) improvement not worth the increased cost • Compression ideas have other uses • For example in quality control of radar data lakshman@ou.edu