1 / 21

S2S Database at ECMWF

Explore the success of TIGGE, the development of the S2S Database, and the challenges and opportunities for future growth and collaboration.

mronald
Download Presentation

S2S Database at ECMWF

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. S2S Database at ECMWF Manuel Fuentes Products Team - ECMWF

  2. Outline • The success of TIGGE • Development of the S2S Database

  3. The TIGGE-Global database • Three archive centres: CMA, NCAR, ECMWF • Ten data providers: • ECMWF, JMA (Japan), UK Met Office (UK), CMA (China), NCEP (USA), MSC (Canada), Météo-France (France), BOM (Australia), KMA (Korea), CPTEC (Brazil) • Exchanges using UNIDATA LDM, HTTP and FTP

  4. TIGGE Archive Centres and Data Providers

  5. Keys to TIGGE success • Strong governance • Precise definition of products, format, naming convention • Quality assurance: • Homogeneity • Completeness • Tigge_check validator • Website with documentation, sample datasets, tools • Easy access to data • Granularity of a field • Batch access

  6. Users of ECMWF TIGGE Data PortalSept 2014

  7. TIGGE Current Status • Operational since 1st Oct 2006 • Addition of TIGGE-LAM 1st Jan 2013 • 1.2 PB, 4.6 billion fields • growing by ~ 550 GB/day • 1.6 million fields/day

  8. What is the future of TIGGE? • BoM stopped in July 2010 • KMA upgraded their model in Dec 2010, resumed feed in July 2011 • MetOfficeupgraded their model in July 2014, resumed feed in November 2014 • CMA upgraded their model in Sep 2014, data still being checked • NCAR stopped at the end of THORPEX • NCEP feed taken by NCDC • still troubleshooting • Many gaps at CMA in the last 6 months

  9. S2S database

  10. S2S Parameters

  11. S2S Parameters

  12. Database Development • Started in 2013 • Agreed on list of parameters to archive • Coordination with WMO: • Definition of S2S codes • Definition of reforecast templates • Definition of GRIB 2 descriptors for all data which was not in TIGGE, eg, Daily means • Created sample encodings for each of the field so other Data providers could follow by example • Agreed on data exchange protocol • Started receiving data in test mode: • Quality checks before archiving in production

  13. S2S Current Status • ECMWF • Real-time and reforecast since 1st January 2015 • NCEP • Real-time since 1st January 2015, reforecast will be complete after 1 year • JMA • fixed reforecast (800 GBytes), USB disks • CAWRC • fixed reforecast (6.5 TBytes), USB disks • Current data in the archive: • 200 GBytes (realtime) • 7.5 TBytes (reforecast)

  14. S2S Current Status

  15. Data flow ftp (Acquisition) CMA Staging area (ECPDS) Data Provider QA Archiving S2S Database

  16. WebApps Data Server: Fields

  17. Web API sample request #!/usr/bin/env python from ecmwfapi import ECMWFDataServer server = ECMWFDataServer() server.retrieve({ 'dataset' : “s2s", ‘origin’ : “ecmwf”, 'levtype' : "sfc", 'date' : "20150101/to/20150131", 'time' : "00", 'param' : “2t", 'grid' : "1/1", 'format' : "netcdf", 'target' : "data.nc" })

  18. Staff • Frederic Vitart, S2S Chair and coordinator • Manuel Fuentes, Technical coordinator • Axel Bonet, ECMWF data production • IoannisMallas, ECMWF Data Acquisition • Richard Mladek, ECMWF QA, Archiving, Cataloguing + Documentation • ShahramNajm, Enrico Fucile, GRIB and WMO • ??? , Synchronisation with CMA

  19. S2S Challenges • Heterogeneous frequency • Realtime • Reforecasts • Data Volumes • Synchronisation between archive centres • Unknown access patterns

  20. Collaboration: Benefits for CMA • Strengthen the collaboration in TIGGE • Learn about S2S data structures • It will be difficult to serve reforecasts to users (we still do not know access patterns!) • Learn by being part of the team • Setup the synchronisation ECMWF-CMA • This cannot be achieved by email • Return to CMA as contact point

  21. Thank you for your attention

More Related