The International TIGGE Archive

1. The International TIGGE Archive Baudouin Raoult, Manuel Fuentes - ECMWF Steven Worley, Doug Schuster - NCAR Bian Xiaofeng, Li Xiang - CMA Many Others! 3rd Thorpex Int. Science Symp., May 2009

2. Background: the TIGGE database Global ensemble forecasts to around 14 days generated routinely at different centres around the world Three archive centres: CMA, NCAR and ECMWF Ten data providers: ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) Data exchanges using UNIDATA LDM, HTTP and FTP 230 TB, 1.5 billion fields, growing by > 1 TB/week, 1.7 million fields/day

3. Background, Brief Time Line • March 2005, TIGGE Workshop at ECMWF • Collect community input for TIGGE science and data requirements • Choose centralized archives for early start (ECMWF, NCAR, CMA) • September 2005, meeting of technical experts at ECMWF • Representatives from ECMWF, NCAR, CMA, and NAEFS • Create technical implementation plan to address TIGGE needs • November 2005, TIGGE Working Group at ECMWF • Present and get approval for the technical implementation plan • October 2006, first EPS data is delivered • May 2006, User access to TIGGE archive starts • February 2008, all 10 international providers sending EPS data Significant accomplishment, create 24x7 international system in less than three years 3rd Thorpex Int. Science Symp., May 2009

4. Quality assurance: homogeneity Homogeneity is paramount for TIGGE to succeed The more consistent the archive the easier it will be to develop applications There are three aspects to homogeneity: Common terminology (parameter names, file names,…) Common data format (format, units, …) Definition of an agreed list of products (Parameters, Steps, levels, …) What is not homogeneous: Resolution Base time (although most provider have a run at 12 UTC) Forecast length Number of ensemble

5. Strong governance Precise definition of: Which products: list of parameters, levels, steps, units,… Which format: GRIB2 Which transport protocol: UNIDATA’s LDM Which naming convention: WMO file name convention Only exception: the grid and resolution Choice of the data provider. Data provider to provide interpolation to regular lat/lon Best possible model output Many tools and examples: Comprehensive documentation Sample dataset available Various GRIB2 tools, “tigge_check” validator, … Scripts that implement exchange protocol

6. QA: Checking for homogeneity E.g. looking for wrong units?

7. Completeness The objective is to have 100% complete datasets at the Archive Centres Completeness may not be achieved for two reasons: The transfer of the data to the Archive Centre fails Operational activities at a data provider are interrupted and back filling past runs is impractical Incomplete datasets are often very difficult to use Most of the current tools used for ensemble forecasts assume a fixed number of members from day to day

8. QA: Checking completeness (TIGGE history)

9. Archive Status and Monitoring, Variability between providers

10. Archive Status and Monitoring, Archive Completeness PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

11. Archive Status and Monitoring, Archive Completeness PL = Pressure Level, PT = 320K θ Level, PV = ± 2 Potential Vorticity Level, SL = Single/Surface Level

12. Access to TIGGE data Data is available to Research and Education after 48 hours Self registration by agreeing to the terms and conditions Portals at CMA, NCAR and ECMWF

13. User Data Access and Support @ ECMWF TIGGE web site Documentation Model changes ECMWF portal offers: Access to offline data Aggregation along any axis (date, level, parameter, origin, ensemble, …) Provision of multi-model data on a single grid (regridding to any lat/lon grid) Sub-area selection Reduces volumes to be downloaded by many order of magnitude

14. TIGGE @ ECMWF

15. TIGGE portal at ECMWF (using Ajax)

16. TIGGE Portal: grid selection

17. TIGGE Portal: area selection

18. TIGGE @ ECMWF

19. TIGGE @ ECMWF: Scheduling users’ requests Monitoring Per user limits Global limits

20. TIGGE @ ECMWF Batch access(recommended)Perl or Python scripts

21. TIGGE@ECMWF: direct access to offline data (MARS)

22. User Data Access and Support, differences @ NCAR • User data selection and subsetting equivalent to ECMWF • 2.5 Weeks of Data Online • Archive organized in file groups by provider and forecast initialization time • Pressure Level, Θ Level, PV Level, Single/Surface Level • Multiple file download option • Direct through browser, using prepared ‘wget’ script • Parameter download available through ‘curl’ scripts • Provides mechanism to build recurring requests • Long-term file groups available by request 3rd Thorpex Int. Science Symp., May 2009

23. User Data Access and Support, differences @ NCAR • Subset data format in GRIB2 or netCDF • Online queue to monitor subset data processing progress • Subsetting completion email notification for users • TIGGE Research Support • Library listing of software tools for data access and analysis • TIGGE Users Group web forum • Community web space to share questions and findings • Provided by Unidata • Archive of Tropical Cyclone (cxml) data for participating providers • Updated 6-hourly 3rd Thorpex Int. Science Symp., May 2009

24. User Data Access and Support, differences @ CMA • Collaborated with ECMWF and CNIC • Grid Middleware • Accessing the distributed TIGGE data based on Grid middleware (GRIA, GOS) technologies • Data retrieval and Derived Production Generation • Request Monitoring and Query • Access Control 3rd Thorpex Int. Science Symp., May 2009

25. User Data Access and Support, differences @ CMA Data Retrieval Data Selection Selecting different data attributes, such as origin centre, date, base time, forecast time step, forecast type, level type, level and parameter. The different data options will be updated dynamically based on the user selection and on the constraints described in the data catalogue. Submit data request If the number of requested fields is larger than the allowed limit, the portal will suggest the users to split their data retrieval. Custom result products Area selection Grid conversion Plotting Browse and download result products

26. User Data Access and Support, differences @ CMA Production Generation (e.g. ensemble mean) Product selection Selecting the product and associated parameters based on the product catalogue Submit product generation request In development, more products later Request Monitoring and Query Check the real-time updates of the information about requests Access the detailed monitoring information of requests for a period of time, including request creation time, request finish time, process status, request condition, detailed processing information and result download. Query monitoring information according to the request date, request user id, request type and result status. Access Control Several kinds of users (admin, data retrieval, product retrieval, …)

27. User Metrics at ECMWF

28. User Metrics at ECMWF : User Metrics at ECMWF

29. User Metrics at ECMWF : Data volumes

30. Introduction to Poster Session demonstrations • Poster Sessions Demonstrations • When • Tuesday, 15:00-16:30 (immediately following presentation) • Wednesday, 16:30-18:30 • Any time, please ask Baudouin, Xiaofeng or Doug • Purposes • Illustrate all features of the portals • Register users • Submit data requests • Demonstrate analysis tools 3rd Thorpex Int. Science Symp., May 2009

31. Credits Alex de Almeida Fernandes, Baudouin Raoult, Beth Ebert, Bian Xiaofeng, Dan Swank, Dave Stepaniak, David Ian Brown, Don Middleton, Dong-Joon Kim, Doug Schuster, Glenn Rutledge, Hannah Wilcox, Jean Nicolau, Jim Fraser, Joerg Urban, Julio Pablo Reyes Fernandez, Justin Cooke, Lang Hongliang, Lewis Poulin, Li Xiang, Ma Qiang, Manuel Fuentes, Michel Grenier, Okki Lee, Richard Dare, Richard Swinbank, Shintaro Yokoi, Simon Thompson, Steve Emmerson, Steve Worley, Steven Anthony, Tan Le, Tian Hao, Tom Yoksas, Waldenio Almeida, Yang Xin, Young-Youn Park, Yves Pelletier, … ... and many more