1 / 34

Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA)

Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA). Gary Blackburn, Steven G. Carson, John K. Williams NCAR Research Applications Laboratory. April 20, 2006. Outline. CCR and goals Requirements addressed Operational concept Impact on external systems

eros
Download Presentation

Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Design Approach Review NEXRAD Turbulence Detection Algorithm (NTDA) Gary Blackburn, Steven G. Carson, John K. Williams NCAR Research Applications Laboratory April 20, 2006

  2. Outline • CCR and goals • Requirements addressed • Operational concept • Impact on external systems • Dependencies on external changes • Agency benefits • NTDA algorithm diagram • Development approach • Real-time demonstration • Data flow diagrams and AEL description • Resource usage evaluation • Test approach • Integration challenges • Proposed schedule • Baseline documents to be updated • Known issues

  3. Configuration Change Request • CCR #NA06-09601“IMPLEMENTATION OF THE NEXRAD TURBULENCE DETECTION ALGORITHM (NTDA) IN THE NEXRAD OPEN RPG” • Goals • Supply turbulence data for the FAA AWRP’s Graphical Turbulence Guidance Nowcast (GTGN) product, scheduled for deployment in Spring 2010 • Display via WARP, CIWS, ITWS, and AWIPS (separate CCRs)

  4. Requirements addressed • Original WSR-88D turbulence detection requirement • Commercial Aviation Safety Team Turbulence JSAT Report (January, 2001) • FAA Aviation Weather Mission Needs Statement #339 Attachments 2 & 5 (June 2002) • Commercial Aviation Safety Team JSIT report Enhancements 64, 65, 68, 72, 76 (July 2004) • TAC website: “Turbulence analysis techniques” is NEXRAD Active Technical Need TN-17 • ~1300 FAA requirements for turbulence and convection • However, there is currently no specific agency requirement for NTDA.

  5. Operational concept Polar Data Available to WARP, CIWS, ITWS, AWIPS NEXRAD Turbulence Detection Algorithm CONUS EDR Grids (NSSL mosaic at NCEP) EDR Grids Available to Other Users Convective Weather Nowcasts Integrated Turbulence Diagnosis and Nowcast In situ EDR GTG Nowcast (ADDS) Satellite Radar Reflectivity NWP model (WRF) GTG = Graphical Turbulence Guidance, ADDS = Aviation Digital Data Service (see http://weather.aero)

  6. Graphical Turbulence Guidance The current GTG clear-air turbulence forecast product with overlaid in situ turbulence reports from United Airlines aircraft • Operational GTG: http://adds.aviationweather.gov • Experimental GTG: http://weather.aero

  7. Impact on external systems • Data transmission issue • How will data be transmitted/collected? • Polar EDR and confidence data from each tilt (1 km range x 1 azimuth spacing) should be transmitted immediately to minimize latency • AWIPS, “Level 2.5” product? • Data collection, archival • Display development considered a separate issue for users (separate CCRs for WARP, etc.)

  8. Dependencies on external changes • None known.

  9. Agency benefits • FAA • Direct benefit from products and displays • Indirect benefit through commercial weather information providers, improved aviation situational awareness • AFWA • Similar to FAA benefits • NWS • EDR data may be useful for convection longevity diagnosis, or in NWP models

  10. NTDA Algorithm Diagram(Fuzzy Logic approach)

  11. Development approach • NTDA implemented in ANSI C • Tested prototype in summer 2005 using LDM data from 16 NEXRADs • 90-95% adherence to “Radar Product Generation System (RPG) C CODING STANDARDS, October 21, 2001” • Raised CODE NTDA memory limit to 100 MB (current usage ~45 MB) • No adaptation parameters for NTDA • Currently uses CLAPACK, g2c, sunae.c (NASA) • CODE prototype uses Build 7.0, public edition • Will switch to Build 8.0, NWS edition when available

  12. Dual-purpose implementation Common Operations and Development Environment / Open Radar Products Generator (CODE/ORPG) Turbulence Remote Sensing Realtime Demonstration System (TRS/RDS) Application: CPC099/TSK001 ntda_alg Application: NCARTurbDetectAlg NtdaAlgs API NtdaAlgs_init NtdaAlgs_FreeMemory NtdaAlgs_CalcEdrFieldsInit NtdaAlgs_EDR_SW_MEAN NtdaAlgs_CalcEdrFieldsFree Algorithm: ANSI C NtdaAlgs.cc Algorithm: ANSI C NtdaAlgs.cc Identical Code in Both Environments

  13. NTDA Real-Time Demonstration, Summer 2005

  14. Why include in WSR-88D baseline? • Ability to use ORPG intermediate products • Reduce the effort required to adapt the NTDA to accommodate ORDA upgrades • Polar EDR data could be made broadly available to users (displays, NWP models, commercial vendors) • Keep NTDA developers “in the loop” on baseline upgrades • Involve ROC, OS&T, and other experts in design, implementation, and approval decisions

  15. 303 COMBASE_ELEV ELEVATION_DATA 162 NTDA_EDR_IP ELEVATION_DATA 160 NTDA_EDR ELEVATION_DATA NTDA_FP 99/02 ELEVATION 163 NTDA_CONF_IP ELEVATION_DATA 299 RECCLDIGDOP ELEVATION_DATA 161 NTDA_CONF ELEVATION_DATA Data flow diagram NTDA_ALG 99/01 ELEVATION

  16. NtdaAlgs – Data flow, Part 1 SW Multiplier Table One time At startup NtdaAlgs_init params struct Application: NCARTurbDetectAlg or ntda_alg NtdaAlgs struct Every Elevation range gates artifact removal field dimensions spectrum width (SW) radial velocity (VE) azimuths & elevations combined sweep sun exclusion NtdaAlgs_CalcEdrFieldsInit

  17. array dimensions reflectivity (DZ) radial velocity (VE) spectrum width (SW signal-to-noise (SNR) power-ratio (PR) EDR EDR confidence SW variance SW residuals variance SW residuals sigma SW confidence / SW variance SW confidence / SW residuals SW confidence / SW sigma SW confidence / SW SW confidence / SNR SW confidence / Alt-DZ SW confidence / PR coverage fraction confidence NtdaAlgs_EDR_SW_MEAN NtdaAlgs – Data flow, Part 2 Every Elevation Application: NCARTurbDetectAlg or ntda_alg Netcdf file (TRS/RDS) Or Intermediate Products (ORPG)

  18. allocate temp arrays compute unsmoothed EDR^2 using range-dependent scaling of SW sun spike removal smooth EDR^2 using 2D confidence-weighted mean reflectivity-derived SW confidences compute final EDR confidence; encode all output fields as short int smooth SW confidence if CombinedSweep compute SW variance compute variance of SW residuals from local linear fit free temp arrays Processing sequence

  19. AEL: Initialization • Initialize quantities for this elevation • OBTAIN Doppler bin size (meters) • OBTAIN range to first Doppler bin (meters) • OBTAIN unambiguous range (meters) • COMPUTE Doppler data array size • COMPUTE reflectivity array size • COMPUTE array of ranges to reflectivity bins • OBTAIN arrays of azimuth and elevation values • for all radials in this elevation • OBTAIN flag indicating whether this is a split-cut • OBTAIN information needed to perform sunspike removal • INITIALIZE reflectivity (DZ), signal-to-noise (SNR), • power-ratio (PR) arrays to “missing data” • OBTAIN DZ & SW arrays from COMBBASE_ELEV input buffer • CALCULATE SNR from DZ • CALCULATE PR from SNR • COMPUTE arrays of ranges and azimuth spacing • COMPUTE interpolate hi-res table of SW multipliers to • construct array of SW multipliers for each range gate • COMPUTE table of confidence as a function of range • COMPUTE processing window size for EDR smoothing • COMPUTE processing window size for SW variance calc • COMPUTE processing window size for SW residuals calc • COMPUTE processing window size for SW confidence smoothing • COMPUTE remove artifacts from SW field • INITIALIZE EDR and confidence Intermediate Product Buffers

  20. AEL: Sun spike removal 2. Perform sunspike removal COMPUTE azimuth and elevation of sun as seen from radar location at current time DO FOR ALL azimuths IF beam azimuth & elevation are too close to sun DO FOR ALL ranges COMPUTE set DZ,SW,SNR,PR to “missing data” END DO END IF END DO

  21. AEL: Reflectivity-derived confidence 3. Calculate reflectivity-derived SW confidence DO FOR ALL azimuths COMPUTE sin and cos of beam elevation angle DO FOR ALL ranges COMPUTE spectrum width as floating-point value IF reflectivity is not “missing data” COMPUTE SNR as floating-point value COMPUTE spectrum width confidence due to SNR from fuzzy-logic interest map COMPUTE spectrum width confidence due to combination of altitude & DZ from fuzzy-logic interest map COMPUTE spectrum width confidence due to PR from fuzzy-logic interest map COMPUTE overall SW confidence due to all DZ-derived sources by combining component confidences using geometric average ELSE // reflectivity is “missing data” COMPUTE set all component confidences to “missing data” COMPUTE set overall reflectivity-derived SW confidence to “missing data” END IF END DO // all ranges END DO // all azimuths 4. Smooth reflectivity-derived SW confidence IF this elevation is a “split-cut” COMPUTE smooth DZ-derived SW confidence to account for temporal incoherence END IF

  22. AEL: SW linear variance 5. Calculate variance of SW residuals after 2D linear fit COMPUTE 2D linear fit of SW at each grid point using SVD COMPUTE SW residual = difference between actual and fit SW at each grid point COMPUTE SW residual variance COMPUTE Z-score of central residual

  23. AEL: “Raw” EDR and confidence 6. Calculate EDR using range-dependent scaling of SW DO FOR ALL azimuths DO FOR ALL ranges IF SW is not “missing data” COMPUTE SW as floating point value COMPUTE SW confidence due to SW residuals variance using fuzzy-logic interest map COMPUTE SW confidence due to SW residuals sigma-score (standard deviations) using fuzzy-logic interest map COMPUTE SW confidence due to all SW-derived confidences by combining component confidences using geometric average COMPUTE EDR confidence due to SW = product of DZ-derived SW confidence times SW-derived SW confidence COMPUTE unsmoothed EDR from range-dependent scaling of SW COMPUTE unsmoothed EDR confidence = EDR confidence due to SW COMPUTE unsmoothed EDR squared = square of Unsmoothed EDR END IF // SW not “missing” END DO // ranges END DO // azimuths

  24. AEL: Conf.-weighted mean EDR 7. Smooth EDR squared using confidence-weighted 2D mean DO FOR ALL azimuths DO FOR ALL ranges COMPUTE size of averaging window in azimuth and range based on range from radar DO FOR ALL azimuths within averaging window DO FOR ALL ranges within averaging window IF EDR squared is not “missing” COMPUTE sum of confidences COMPUTE sum of EDR squared * confidence END IF END DO // ranges within window END DO // azimuths within window COMPUTE fraction of averaging window containing non-missing data COMPUTE confidence of averaged EDR squared as the average of the confidences within window COMPUTE confidence-weighted average of EDR squared END DO // ranges END DO // azimuths

  25. AEL: Final product computation 8. Calculate final EDR and confidence and encode for output DO FOR ALL azimuths DO FOR ALL ranges COMPUTE EDR = sqrt(EDR smooth squared) COMPUTE CONF = EDR smooth squared confidence COMPUTE confidence due to coverage fraction using fuzzy-logic interest map COMPUTE CONF = CONF * (coverage fraction conf) IF SW not “missing” COMPUTE encode EDR and CONF as short int using scale factor and bias ELSE // SW is “missing” COMPUTE set EDR and CONF to “missing value” END IF COMPUTE store final EDR and CONF values in output arrays END DO // ranges END DO // azimuths

  26. Resource usage evaluation • Test hardware • dual 3.4 GHz Xeon processors • 4 GB RAM, 2 MB FSB • Red Hat Enterprise Linux 3 • Software: CODE Build 7.0, public edition • Approach • ran ORPG/CODE using 3 test cases of archived Level II data • stripped-down NTDA (“best case”) and full version • recorded total CPU time for NTDA • evaluated size of BZIP2 compressed linear buffers (full 40-length buffer size = 29510272 bytes)

  27. Case 1: KMKX 20050605 0700 UTC NTDA CPU time per volume: 32s or 83s NTDA compressed linear buffer sizes (length 40) Turbulence EDR_IP: 497539 bytes Confidence CONF_IP: 530829 bytes VCP 121 Reflectivity (first tilt) NTDA EDR NTDA confidence

  28. Case 2: KMKX 20050925 1900 UTC NTDA CPU time per volume : 23s or 60s NTDA compressed linear buffer sizes (length 40) Turbulence EDR_IP: 360212 bytes Confidence CONF_IP: 381658 bytes VCP 21 Reflectivity (first tilt) NTDA EDR NTDA confidence

  29. Case 3: KMKX 20050915 1600 UTC NTDA CPU time per volume: 15s or 23s NTDA compressed linear buffer sizes (length 40) Turbulence EDR_IP: 170930 bytes Confidence CONF_IP: 159110 bytes VCP 32 Reflectivity (first tilt) NTDA EDR NTDA confidence

  30. Test approach • Summer 2006 real-time demonstration • continued statistical verification, pilot feedback • case studies of poor performance events • Run CODE version on LDM and archived data • Perform extensive testing • test CODE output to verify match with prototype • stability/regression testing • acceptance testing • Make sure test suite includes volumes from each VCP

  31. Integration challenges • NTDA developed/tested on Build 7.0/8.0, but intended for Build 10.0 • important to anticipate Build 10.0 changes that may impact the NTDA where possible • Data format and transmission mode have not yet been determined

  32. Proposed schedule • SREC presentation: 2 May • Summer 2006 demonstration, algorithm tuning: 1 June – 15 September • Algorithm freeze: 15 September • Code freeze: 1 October • Intensive code testing/hardening period: 1 October – 30 November • Second DAR: 15 October • IRR: 5 December • Code delivery: 5 January 2007

  33. Baseline documents to be updated • System Specification (SS) – Yes • Software Requirement Specification (SRS) – Yes • Algorithm Enunciation Language (AEL) – Yes • Product Specification – Yes • Interface Control Document(s) (ICD(s)) – Yes • Software Description Document (SDD) (VISIO diagrams) – Yes

  34. Known issues • VCP 121 (MPDA) is not handled ideally • adjust or eliminate use of PowerRatio confidence • Beam indexing is not yet accommodated • REC-based confidence is not yet integrated • In CODE, NTDA is eventually killed, e.g., ERROR:NtdaApp:NtdaApp_Read_COMBBASE_ELEV: Mon Mar 13 16:19:26 2006 COMBBASE_ELEV pointer is NULL; ERROR:NtdaApp:NtdaApp_Read_COMBBASE_ELEV: Mon Mar 13 16:19:26 2006 RPGC_get_inbuf status = 1; ERROR:NtdaFpApp:NtdaFpApp_ReadInputs: Mon Mar 13 16:19:27 2006 NTDA_EDR_IP input buffer pointer is NULL; ERROR:NtdaFpApp:NtdaFpApp_ReadInputs: Mon Mar 13 16:19:27 2006 RPGC_get_inbuf failed for NTDA_EDR_IP RPGC_get_inbuf status = 1; ...

More Related