1 / 40

Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA

Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA 14 May 2009 COPC Meeting NAVO Stennis Space Center. Outline. Participants and resources Recent dropout examples COPC Action Item (AI) 2008-1.5 COPC AI 2008-2.14 Dropout team findings

kendall-tran
Download Presentation

Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Update on Dropout Related COPC Action Items Presented by Dr. Bradley Ballish Co-Chair JAG/ODAA 14 May 2009 COPC Meeting NAVO Stennis Space Center

  2. Outline • Participants and resources • Recent dropout examples • COPC Action Item (AI) 2008-1.5 • COPC AI 2008-2.14 • Dropout team findings • Summary • Next steps • Background slides COPC Dropout Update

  3. Participants and Resources • NCEP’s Jordan Alpert • NCEP’s DaNa Carlis • NCEP’s Krishna Kumar • NCEP’s Bradley Ballish • NCEP is investing in an additional FTE/contractor • NRL’s Rolf Langland • FNMOC’s Chuck Skupniewicz • The NCEP dropout team meets weekly with Steve Lord and reports quarterly to the NCEP/OD COPC Dropout Update

  4. Recent Dropout Examples FNMOC Dropout GFS Dropout To view how the model errors change with time at different levels,use the the website of NCEP’s DaNa Carlis: http://www.emc.ncep.noaa.gov/gmb/dcarlis/directory, and click on Dropout_cases/south_hem/10apr200900z_dropoutSH for the above GFS dropout, see the next slide for differences in 48 hour GFS and ECMWF forecasts, where the forecast differences suggests multiple initial condition errors COPC Dropout Update

  5. 48 Hour GFS • forecast error • versus verifying • GSI analysis • 48 Hour ECMWF • forecast error • versus verifying • ECWMF analysis • 48 hour GFS and • ECMWF forecast • differences, which • often show several • large difference • areas in the SH COPC Dropout Update

  6. AI 2008-1.5 COPC Action Item 2008-1.5: Develop a monitoring system to analyze differences between the NCEP and FNMOC global models and the ECMWF global model in real-time and make this real-time system available to OPCs as a daily tool. Status • Experts from NCEP, FNMOC and AFWA have had brainstorming meetings and email exchanges • Initial requirements have been defined COPC Dropout Update

  7. AI 2008-1.5 Draft Requirements • NCEP and FNMOC will generate internal warnings when their global models have extreme localized differences from ECMWF forecasts on a 1x1 degree grid • NCEP and FNMOC will develop a real time warning system to show when integrated forecast differences of their global model with the ECMWF (such as height correlations) exceed normal limits • FNMOC and NCEP will develop graphical tools to allow 24x7 staff to study the divergence in forecasts to show likely analysis problem areas COPC Dropout Update

  8. AI 2008-1.5 Next Steps • Requirements sign off (6-2009) • Identification of project manager (7-2009) • Work schedule completion (8-2009) - Design TBD - Construct TBD - Test TBD - Deploy TBD COPC Dropout Update

  9. AI 2008-2.14 • COPC Action Item 2008-2.14: CSAB will oversee the work of the “Dropout Team” to continue examining the performance dropout issue and how lessons learned can be applied to numerical model performance improvements. This activity shall include establishing periodic interface with ECMWF to exchange QC methodologies/best practices. Recommendations for QC of data, acquisition of new data sources and any other relevant issues shall be presented to the Spring 2009 COPC Status • Lessons learned have been documented • The ECMWF has been somewhat responsive to attempts at coordination but follow through has been sporadic • Recommend closing this AI COPC Dropout Update

  10. Dropout Team Findings • General aspects related to the ECMWF comparison • The NCEP GFS can avoid most dropouts when initialized by the ECMWF analysis globally or in select sensitive areas (see background slide 20) • The ECMWF analysis is 4DVAR, runs at later times with a larger data time window and has a higher resolution model • The ECMWF observational data QC is more comprehensive than NCEP’s • For almost all dropouts studied, simple, obvious data QC problems have not been found to explain the dropout • Most dropouts tend to involve subtle analysis differences in sensitive areas that grow with time in the forecasts • Observational data denial tests, where various data types are removed (such as satellite winds or select radiances), improve some cases and help identify problems • For more details, see the two papers presented at the AMS 2009 annual meeting, see: http://ams.confex.com/ams/89annual/techprogram/paper_142644.htm and http://ams.confex.com/ams/pdfpapers/142649.pdf COPC Dropout Update

  11. Dropout Team Findings • Specific deficiencies identified • Analysis: The GSI analysis agrees more with observational outliers (observations with large differences to the model background) than the ECMWF • Analysis: The GSI and FNMOC analyses have higher heights than the ECMWF analysis possibly due to: • Satellite radiance bias correction differences • ECMWF thinning of aircraft data which have an overall warm bias COPC Dropout Update

  12. Dropout Team Findings • Specific deficiencies identified • Data: The GSI and FNMOC analyses are affected more by the slower speeds of satellite winds due to QC differences • Data: Tests show that the use of non-radiance data in the GSI analysis sometimes hurts forecast skill in the Southern Hemisphere and is being researched to define the error sources • Data: OPC weather station dictionaries (locations and elevations) are inconsistent with each other and published WMO tables, impacting analyses and forecasts COPC Dropout Update

  13. Summary • Dropouts continue to be a problem for forecast skill • Proposed requirements for AI 2008-1.5 (real time analysis of NCEP and FNMOC differences to the ECMWF) were given, need approval and follow up actions • The dropout team has documented various causes of dropouts and recommends closing of AI 2008-2.14 COPC Dropout Update

  14. Next Steps • Complete monitoring tool work • Review capabilities of ECMWF observational data base (ODB) • Continue diagnostic work • Focus on • Satellite winds • Aircraft temperature data (bias correction a higher priority for ECMWF as well) • Satellite radiance data COPC Dropout Update

  15. Background SlidesProposed Plans for NCEP & FNMOC COPC Dropout Update

  16. Proposed Plans • NRL could use adjoints of the analysis/model to direct the OPCs towards better QC of satellite winds • NCEP will test bias correction of surface pressure data • JCSDA could test using satellite radiance data as QC checks on altitudes of satellite winds COPC Dropout Update

  17. Proposed Plans (continued) • NCEP will consider development of track-checking of marine surface and radiosonde data • NCEP will work with the NWS TOC to develop a website showing information on bufr data types at the TOC • JCSDA could develop graphical tools to show integrated satellite radiance data (height and total precipitable water) COPC Dropout Update

  18. Proposed Plans (continued) • NCEP will use baroclinic instability rates and ensemble tools to help analyze model sensitivity to analysis differences • AFWA could make reports every 6 months comparing station dictionary metadata entries for the OPCs with recommendations for corrections COPC Dropout Update

  19. Collaborative Dropout Work • FNMOC’s Dr. Rolf Langland and Chuck Skupniewicz have met with the NCEP staff and shared email exchanges resulting in useful information. Dialog continues • Dr. Langland’s website http://www.nrlmry.navy.mil/adap-bin/tcs_adap.cgi, with adjoint estimates of how analysis differences impact NOGAPS 24 hour forecasts, is useful in showing sensitive areas in the analysis • Dr. Langland’s studies of height biases versus the ECMWF and his finding that aircraft data in the Pacific have negative impact based on adjoint estimates are important results • OPCs have been comparing station metadata dictionaries for weather observing sites and making corrections COPC Dropout Update

  20. Trough in central Pacific shows differences between ECMWF (no dropout) and GFS (had dropout) Ovrly “patch” box ECM in this area but GSI elsewhere resulted in 16 point improvement in 5 day 500 hPa AC score COPC Dropout Update

  21. Divergence of Forecast Measures • Develop a system to warn OPCs when the current divergence (difference) in forecasts exceeds expected limits • Divergence in forecasts measures should include anomaly correlation scores and RMS differences • Metrics may be for limited areas such as the Northern Hemisphere, North American, or other theaters of interest • Metrics should include scores on heights and winds at a few pressure levels and precipitation • Normal limits could be derived from past periods that excludes poor forecast cases COPC Dropout Update

  22. Actions for Large Forecast Differences When we detect that comparable forecasts such as the 00Z ECMWF/GFS have abnormally large differences: • Alert forecasters • Check divergence in other forecasts e.g. 18Z GFS vs 00Z GFS or FNMOC vs GFS • Analyze forecast maps to find problem areas in the analysis • Check for possible data problems COPC Dropout Update

  23. Satellite Radiance Observation Plots • Codes have been developed at NCEP to plot radiance derived temperatures and other observational data differences to the model background or analysis from a given channel for a particular satellite • These are useful for analyzing dropout cases, but are too time consuming as this involves many satellite types and channels (each with different vertical profiles) • These tools depend on the accuracy of our radiance bias corrections • More work is needed to get simpler summaries e.g. height differences at standard levels from all channels integrated together from each satellite COPC Dropout Update

  24. A Detection and Warning System for Problems in Classes of Observational Data • An operational system will be developed to warn us if any class of data develops a large change in bias or RMS differences versus the model background • History has shown that classes of data can develop problems, such as US wind profilers from 9 to 11 April 2008, see the next slide. The GFS has a .67 500 hPa AC 5 day score from 18Z 9 April 2008 • Satellite radiances must be included, see the following slide • Findings of bad data must be shared with other OPCs COPC Dropout Update

  25. An error caused US wind profilers to have east longitudes or wrong locations in China COPC Dropout Update

  26. GOES 11 Channels 1-4Contribution to GSI Penalty Function Channels 1-3 show jump around 12Z 9 Apr 2009 COPC Dropout Update

  27. 5 Day Anomaly Correlation Scores at 500 hPa for Dropout Cases ECM Performs Better than GFS (NH) 2007-2008 • ECM runs (blue) are a good representation for ECMWF analysis • OVRLY runs (green) with ECM psuedo-obs over the Central Pacific drastically improve two October 2007 dropout cases (102200 & 102212). COPC Dropout Update

  28. SH 5-day 500 hPa Anomaly Correlation Scores ECM runs (blue) in the SH do almost as well as ECMWFCNTRL runs (green) improve upon GFS scores 9 of 10 times but only alleviates about half of the dropoutsInterpECMGES runs (purple) improve 4 of 10 cases over the CNTRL run COPC Dropout Update

  29. Comparison of how the GSI and ECM Analyses fit Observations • Statistics are made on how the GSI and ECM analyses fit the observations for different observation types as a function of pressure and analysis differences, for different regions • This study shows that each analysis fits certain observation types differently, but does not conclude which performs better • These analysis fits to data need to be rerun with higher resolution ECMWF analyses COPC Dropout Update

  30. GFS vs. ECM Fits to Radiosondes

  31. Satellite Wind Speed Biases OBS-ANL Apr 2008 12Z ECM (Red), GSI (Blue) m/sec Note ECM biases are mostly negative compared to GSI, meaning ECM winds are stronger than GSI winds at satellite wind locations

  32. Systematic Height Differences in the GSI vs. ECMWF • The GSI analysis systematically has higher heights than ECMWF at 200 hPa – note much red and little green • This possibly could be due large numbers of aircraft observations with warm biases, which warm the analysis and could be affecting the satellite radiance bias corrections COPC Dropout Update

  33. Rolf Langland (NRL Monterey) • shows systematic height differences between all models and ECMWF (shown is ECMWF-NCEP at 500 hPa), apparently from the satellite window coverage of ECMWF (12-h) vs others (6-h). • A Height difference plot arranged with time (October to December 2007) vs longitude, averaged over 35-65N latitudes. • The range of the systematic bias is ±12 m COPC Dropout Update

  34. Aircraft vs Sonde GSI Draws to Temps between 200-300 mb SOND Tdiff (obs-ges) Aircraft Tdiff (obs-ges) SOND Tdiff (obs-anl) Aircraft Tdiff (obs-anl) # Aircraft >> # Sondes, thus warm aircraft data overwhelms the GSI/GFS system

  35. Suru Saha’s website displays model fits to RAOBS in North America showing the GFS analysis and guess maintain a warm bias throughout most of the troposphere that may be related to large numbers of aircraft with warm biases COPC Dropout Update

  36. Proposed Aircraft Temperature Bias Corrections • In the November 2008 issue of BAMS, Ballish and Kumar analyzed systematic temperature differences between radiosondes and aircraft • Biases vary with aircraft types, pressure and the aircraft phase of flight • The next slide shows their proposed temperature bias corrections to ACARS temperatures for January 2007 COPC Dropout Update

  37. COPC Dropout Update

  38. Model Climate Impact from Aircraft Warm Temperatures • The next slide courtesy of Dick Dee of ECMWF shows the increase in the number of aircraft reports versus time at ECMWF • The temperature bias of the ECMWF analysis and background seem to be affected by the large increase in the number of aircraft temperatures along with other factors • The NCEP GSI may have more bias impact as it does not thin aircraft data and its satellite radiance bias corrections are anchored to the analysis as truth as opposed to radiosondes as truth COPC Dropout Update

  39. Model Climate Bias Impact From Warm Aircraft Temperatures Global-mean departures of analysis (blue) and background (red) from radiosonde temperatures (K) at 200hPa, and number of obs/day (x10-4, green) Global-mean departures of analysis (blue) and background (red) from aircraft temperatures (K) at 200hPa, and number of obs/day (x10-4, green) COPC Dropout Update

  40. Eady Baroclinicity Index (EBI) a c 5-day AC=0.67 d b 5-day AC=0.86 The EBI has higher amplitude and noise in the ECM analyses (b) versus operational GSI (a), but shows less amplitude at hour 24 (d) than the operational GFS (c). The EBI may be useful for studying and predicting forecast error growth rates and may help warn of a coming dropout. COPC Dropout Update

More Related