1 / 27

Verification of WAFS Global Icing Products

Verification of WAFS Global Icing Products. Jennifer Mahoney 1 and Sean Madine 1,2 1 NOAA/Earth System Research Laboratory 2 Cooperative Institute for Research in the Atmosphere. This work was completed on behalf of the U.S. Federal Aviation Administration

miyo
Download Presentation

Verification of WAFS Global Icing Products

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Verification of WAFS Global Icing Products Jennifer Mahoney1 and Sean Madine1,2 1NOAA/Earth System Research Laboratory 2Cooperative Institute for Research in the Atmosphere This work was completed on behalf of the U.S. Federal Aviation Administration Presented at: WAFS Workshop on the Use and Visualisation of Gridded SIGWX Forecasts Paris, September 14-15 2009

  2. Acknowledgements This work was completed on behalf of the U.S Federal Aviation Administration Project Team • Project Lead: Sean Madine • Project Analyst: Chungu Lu • Programming Support: Mike Kay • Program Chief: Jennifer Mahoney

  3. Outline of Discussion • Motivation, Background, and Challenges • Approach and Assessment Strategy • Creation and validation of CloudSat Icing Product (CLIP) • Forecast Analyses • Comparison of WAFC U.K. and WAFC U.S. Global Icing • Comparison of WAFC Global Icing to Significant Weather (SigWx) Product • Summary

  4. Motivation • Icing forecasts are used in strategic planning • One example: Planning for amount of fuel carried on board aircraft to manage possible icing encounters. • The WAFC Washington (U.S.) and the WAFC London (U.K.) have developed Global Icing Products, updated frequently, that may improve icing information that is provided to the aviation community.

  5. Background • Project Goal: Evaluate the quality of the WAFC Global Icing Forecasts; with resource and time constraints: • Altitudes that typically contain icing (i.e., mid-levels) • Scoped evaluation to Nov 2008 – Jan 2009; winter in northern hemisphere • Domains: Global, North Pacific, North Atlantic; with a North Atlantic focus • Evaluation criteria : 1) accuracy of the products should be at least as good as the human generated Significant Weather (SigWx) forecasts produced today, and 2) WAFC U.S. and U.K icing products need to be compatible

  6. Challenges • Obtaining a representative observation dataset • Voice pilot reports (PIREPS) for icing are sparse in time and space • PIREPs poorly sample at flight levels relevant to operational decisions (enroute flight levels are typically above the occurrence of icing) • PIREPs disproportionately sample icing conditions • Exploring new techniques for creating a proxy observation of icing in the atmosphere

  7. Assessment Approach • Compare the quality of the WAFC U.K icing product to the WAFC U.S. icing products • Measure the quality • Qualitative examination of compatibility (i.e., incompatible means ambiguity of the forecast in the planning process) • Compare the quality of the WAFC Global Icing Products to the Significant Weather Product • Measure the quality

  8. Assessment Strategy • Build an icing verification proxy based on satellite products and use for evaluating the WAFC icing forecasts over global data-sparse domains • Development of CloudSat Icing Product (CLIP) • Tuned and validated CLIP with respect to the Current Icing Potential (CIP), current U.S. operational icing analysis • Study period: November 2007 through January 2008 (Note: Same season, 1 year earlier) • Use CLIP to assess the quality of the WAFC global icing products and SigWx products over global domains • Study period: November 2008 through January 2009

  9. CloudSat Icing Product (CLIP) Strengths • CloudSat cloud profiling radar very effective at diagnosing existence of clouds, particularly at levels relevant to our study • CLIP highly accurate at detecting no cloud and therefore ‘no icing’ • Information for CLIP allows assessment of forecast risk (miss) and forecast efficiency (false alarms) • PIREPs don’t allow the false alarm measure because of the imbalance of yes/no reporting • CLIP’S radar input provides • An objective measure of cloud, at high resolution • Continuous reporting along orbiting track • An improvement upon the subjective nature and sparseness of using PIREPs

  10. Comparison of the WAFC Global Icing Products

  11. Analysis of WAFC Global Icing Products • Many stratifications examined • Geographic domain Global, N. Atlantic, N. Pacific • Flight level • All, FL100, FL140 • Cloud type (as diagnosed by CloudSat) • All, Convective, Non-convective • Algorithm threshold • 0.1, 0.3, and 0.5 • Lead time • All leads 6, 12, …, 36-h individually

  12. Analysis of WAFC Global Icing Products • Main thread of investigation • North Atlantic domain • Domain of interest during assessment winter time period and included significant amount of air traffic density • Altitudes 10,000 - 20,000 ft • Operationally significant altitudes and typically contain ice • Non-convective clouds • Products definition specifically includes non-convective icing • Max icing attribute • Most compatible WAFC forecasts

  13. Domains Global, North Atlantic, North Pacific; 3 CloudSat orbits, each orbit ~1.5h,

  14. WAFS Maximum Icing 12 January 2009 UK/FL100 US/FL100 UK/FL140 US/FL140

  15. Vertical Cross Sections Along CloudSatPath Atlantic Domain WAFS-UK Cloud Classification WAFS-US CLIP 12 January 2009

  16. WAFS Maximum Icing 23 December 2008 UK/FL100 US/FL100 UK/FL140 US/FL140

  17. Vertical Cross Sections Along CloudSatPath Atlantic Domain 23 December 2008

  18. Comparison of Performance: Flight Level (Statistics for 12- and 18-hr leads, Atlantic domain, Non-convective clouds, Max icing attribute) Product POD False Alarm Rate (1-PODn) ROC Thresholds: 0.1, 0.3, and 0.5

  19. Comparison WAFC Icing Forecasts in Operational Context CLIP No Icing No Cloud No Icing Warm Cloud No Icing Cold Cloud Yes Icing Yes 12367 15356 2940 3640 UK (fcst) No 6684 54837 6430 3375 No Icing No Cloud No Icing Warm Cloud No Icing Cold Cloud Yes Icing 8987 11576 3661 2251 Yes US (fcst) No 10063 58617 5709 4764 Atlantic domain, 0.1 Threshold, FL100/140, 12- and 18-hr leads, Max icing intensity, Non-convective clouds Blue – Agreement Yellow – False alarms Red - Misses

  20. Comparison Summary • The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk

  21. Comparison of the WAFC Global Icing Products to the SigWx Product

  22. WAFS Mid Level SIGWX Product • Thunderstorms/CB • Tropical Cyclones • Mod/SevTurb (CAT or IC) • Mod/Sev Icing • Tropopause Heights • Jetstreams (80kt and above depicted) • Jet Depth • Volcanic Eruptions • Widespread Sand/Dust storms • Release of Radioactive Materials NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx Product was evaluated

  23. Analysis - Main Thread • Atlantic domain • Altitudes 10,000 - 20,000 ft • Non-convective clouds • Max icing attribute

  24. Comparison of Performance: SigWx Chart (FL100/140, SigWx issuances/lead, Atlantic domain) Product POD False Alarm Rate (1-PODn) NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx product was evaluated ROC Thresholds: 0.1, 0.3, and 0.5

  25. Comparison Summary • The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk

  26. Summary • The U.K. icing product slightly outperforms the U.S. icing product in an operational; For similar FAR values, the U.K. has higher POD • Qualitatively, the WAFC U.K. and U.S products are incompatible in volume extent and intensity of potential • Both U.K. and U.S. products perform better than the SigWx Product in the operational context; For similar FAR, both WAFC products have higher POD values • Based on this study and understanding of the algorithms, the automated WAFC product could be significantly improved with respect to volume coverage and calibration of potential

  27. Appendix • False alarm ratio = yn/yy+yn • False alarm rate = 1-PODn = yn/nn+yn • PODn = nn/nn+yn • POD = yy/yy+ny 2x2 Contingency Table

More Related