1 / 26

Efficient Reduction and Compression of Weather Radar Data in Universal Format

Efficient Reduction and Compression of Weather Radar Data in Universal Format. W. David Pan Dept. of Electrical & Computer Engineering, Univ. of Alabama in Huntsville ONR Summer Faculty Research Fellow, Naval Research Laboratory, Monterey, CA Paul R. Harasti

nemo
Download Presentation

Efficient Reduction and Compression of Weather Radar Data in Universal Format

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Efficient Reduction and Compression of Weather Radar Data in Universal Format W. David Pan Dept. of Electrical & Computer Engineering, Univ. of Alabama in Huntsville ONR Summer Faculty Research Fellow, Naval Research Laboratory, Monterey, CA Paul R. Harasti Visiting Scientist Programs, University Corporation for Atmospheric Research, Boulder, CO Michael Frost, Qingyun Zhao, John Cook Marine Meteorology Division, Naval Research Laboratory, Monterey, CA Timothy Maese Basic Commerce and Industries, Moorestown, NJ Lee J. Wagner Atmospheric Propagation Branch, SPAWAR Systems Center, San Diego, CA

  2. Outline • Problem • UF files • Challenge • Header compression • Radar data compression • Software developed and test results • Further work

  3. Radar Data Assimilation System • Naval Research Lab (NRL) Monterey is developing • a radar data assimilation system to • enhance the safety of at-sea ship & aircraft operations by • predicting short term changes in in-situ hazardous weather • providing decision makers with tools to exploit or mitigate those changes. • The system will • take advantage of Navy vessels with weather radar capability, e.g., • SPS-48E/G: HWDDC(Hazardous Weather Detection and Display Capability) • SPY-1 Tactical Environmental Proc. • have the capability to send radar data back to FNMOC for incorporation into the numerical weather prediction model

  4. Impacts of HWDDC • Flight safety • Reduction of weather-related aviation mishaps • Flight and deck operations • Avoidance of weather-related disruptions • Ship navigation • Avoidance of storms, areas of poor visibility • Maintenance of wind over deck

  5. Weather Extraction Computer 5.4 MB / File x 12 Files / Hour Weather Radar Data Files (UF Format) Source: Tim Maese et al., “Hazardous Weather Detection and Display Capability for US Navy Ships,” 23rd AMS Conf. on IIPS, 2007.

  6. Challenge: Minimized Bandwidth Use FNMOC Battlegroup SIPRNet 3 UF files / hour / ship Ship 2 Ship n … Ship 1

  7. Radar Data Files in Universal Format File Name File Size • UF files generated approx. once every 5 mins • File size is 5.4 Mega Bytes / file • File consists of records • Each record consists of headers and data of different types • Data set available (252 UF files collected on Feb. 22, 2006)

  8. Challenge 1 MB The smaller, the better ! 2.4 5.4 MB 1.7 1.3 PAQ Goal: Can we compress each UF file down to < 1 MB ? Bzip2 Gzip UF File Size Note: PAQ required 14 minutes per UF file – unacceptable for near-real-time applications

  9. Divide and Conquer Significant redundancies in headers Headers of neighboring records change very slightly Headers interleaved with data in original UF files Render the UF files hard to compress Separating headers from data Reorganizing the UF file make it easier to compress Goal Large compression on headers Lossless recovery of headers guaranteed Data reduction w/o impacting COAMPS (Coupled Ocean/Atmosphere Mesoscale Prediction System) Near real-time (low delay, small memory, etc.) Our Strategies

  10. Data (DZ) Data (SN) Data (VE) Data (SW) Data (VV) FH(DZ) FH(SN) FH(VE) FH(SW) FH(VV) LUHB MHB DH 19 21 21 19 21 9 3+2*5 45 Headers and Radar Data Headers: MHB: Mandatory Header Block LUHB: Local Use Header Block DH: Data Header FH: Field Header ________________________ Data Fields:DZ: Reflectivity (dBZ) SN: Signal-to-Noise Ratio (dB) VE: Mean Radial Velocity (m/s) SW: Spectrum Width (m/s) VV: Valid Velocity Flags Elevation . . . Range Ray . . . Data stored in a record: Azimuth Ray Length:

  11. Breakdown Header Data

  12. Mandatory Header Blocks uf: [2x1 char] recordLength: 941 nonMandPosition: 46 localPosition: 46 dataPosition: 55 recordFile: 0 volumeScan: 1 rayNumber: 0 recordInRay: 1 sweepNumber: 0 radarName: [8x1 char] shipName: [8x1 char] latDegrees: 20 latMinutes: 46 latSeconds: -27776 lonDegrees: -154 lonMinutes: -57 lonSeconds: 5312 antennaHeight: 40 year: 6 month: 2 day: 22 hour: 15 minute: 42 second: 50 timeZone: [2x1 char] azimuth: 0 elevation: 12 sweepMode: 1 fxa: 0 sweepRate: 5760 genYear: 2006 genMonth: 2 genDay: 22 genShip: [8x1 char] missingFlag: 1 uf: [2x1 char] recordLength: 941 nonMandPosition: 46 localPosition: 46 dataPosition: 55 recordFile: 1 volumeScan: 1 rayNumber: 1 recordInRay: 1 sweepNumber: 0 radarName: [8x1 char] shipName: [8x1 char] latDegrees: 20 latMinutes: 46 latSeconds: -27776 lonDegrees: -154 lonMinutes: -57 lonSeconds: 5312 antennaHeight: 40 year: 6 month: 2 day: 22 hour: 15 minute: 42 second: 50 timeZone: [2x1 char] azimuth: 64 elevation: 12 sweepMode: 1 fxa: 64 sweepRate: 5760 genYear: 2006 genMonth: 2 genDay: 22 genShip: [8x1 char] missingFlag: 1 different entry different entry Record 1 Record 0 different entry different entry

  13. Macro Headers Record n: merge the headers (zero padding for empty headers) Data (SW) Data (VV) Data (DZ) Data (SN) Data (VE) FH(SW) FH(VV) FH(DZ) FH(SN) FH(VE) LUHB MHB DH FH(DZ) FH(VE) FH(SN) FH(VV) FH(SW) LUHB MHB DH FH(DZ) FH(VE) FH(SN) FH(VV) FH(SW) LUHB MHB DH . . . FH(DZ) 00…………………0 FH(SN) LUHB MHB DH Record index FH(DZ) FH(SN) 00…………………0 LUHB MHB DH 168 words = 336 bytes

  14. Direct Compression 0 The macro header can be compressed to about 8 KB

  15. Bit-wise XOR Properties Truth Table x 0 1 y 0 0 1 1 1 0 (Recovery) Differential Operation x FH(DZ) FH(SN) FH(VE) Record n FH(VV) FH(SW) LUHB MHB DH Bit-wise Exclusive OR y FH(DZ) FH(VE) FH(SN) FH(SW) FH(VV) Record n+1 LUHB MHB DH Sparse vector with most of the entries being 0 Diff. Record n+1 • XOR operations run much faster than subtractions on computers • XOR is safe – no worry about exceeding dynamic range due to subtraction

  16. Lossless Reconstruction x FH(DZ) FH(VE) FH(SN) FH(SW) FH(VV) Record n LUHB MHB DH Sparse vector with most of the entries being 0 Diff. Record n+1 y FH(DZ) FH(VE) FH(SN) Record n+1 FH(SW) FH(VV) LUHB MHB DH D0 = H0 = = H0 H0 D1 = H0 H1 H1 H1 H2 D2 = H1 H2 H2 . . . . . . . . . Sparse and thuseasy to compress Differential Macro Header Perfectly Recon. Macro Header Macro Header

  17. Differential Macro Header Sea of 0’s The DMH can be compressed to about 2.5 KB -- about 3 times more compression than on MH

  18. Compressed Header Sizes < 3 KB Over 600:1 compression! (from 1,677 KB) Time

  19. Data Reduction • Can now focus on data compression • Headers can be squeezed to negligibly small sizes • Blank out data if values are below certain thresholds, as determined by the QC (quality control) requirements • DZ <= 5 dBZ, or • SN <= 10 dB • Some data types (e.g., SN) in the original UF files are for used for QC only • Drop these data will further reduce the bandwidth load • Transmit only three types of data (DZ, SW, VE) Header Data 70%

  20. UF Compression Software Package UF Reader MH DMH ‘uf.dat’ XOR Header Extractor BZIP2 compression UF File Data Extractor Thresholding ‘uf.dat.bz2’ ‘ufzip’: Encoder Channel DMH UF Writer MH XOR-1 Recon. UF File ‘uf.dat’ BZIP2 de-compression Data ‘ufunzip’: Decoder

  21. No Thresholding Avg. Size of Compressed File (DZ, SW, and VE) = 0.4 MB

  22. Thresholding < 0.25 MB/file Sizes of Compressed UF Files

  23. UF File 70 UF File 190 UF File 1 Original DZ Data Recon noTH Recon withTH

  24. Linux Screenshots Encoder Compressed file size (in bytes) Took 3 sec to compress an UF File Decoder Took only 1 sec to reconstruct the UF File

  25. Summary 1 MB 5.4 MB 1.7 1.3 0.5 MB 2.4 0.12 MB Threshold + only DZ,VE,SW transmitted(Effective Reduction > 40: 1) PAQ Bzip2 Gzip Estimated upper bound on compressed UF file sizes -- more than meet the goal! UF File Size

  26. Future Work Further Work • The developed UF file compression software package will be tested for approval for ship installation by the Integrated Test Facility laboratory, Space and Naval Warfare Systems Center (SPAWAR), San Diego, CA. Acknowledgment • ONR Summer Faculty Research Fellowship

More Related