Towards Automated Model Output Analysis

1. Towards Automated Model Output Analysis Charles Doutriaux Ispra 2006

2. Summary • The Problem • PCMDI’s Experience • Overview • CDAT • ESG • Collaborations • The AutoMOD Project • Portability

3. The Climate Model Data Management Issue Usually Today, Tomorrow Tremendous manual intervention, horribly inefficient by any measure, resource wasting Computers do nearly all the work and scientists do more interesting things than shoving bytes around • Data • Different formats • netCDF is not standardized • Different sites require knowledge of different methods of access • Metadata • Painful to produce • Most kept in files separate from data • unsearchable unless one is “in the know” (some kept in people’s brains) • Access control • Manual • Not formalized • Data requests/analysis • Beginnings of a formal process Beginnings of web portals • Far too much done by hand • Logging nearly non-existent • Data • netCDF is standardized to the CF model • Different sites but standardized access protocol. • Metadata • Created via batch or semi-automated processes • Kept in databases and readily searchable • Access control • Formalized • Highly granularized – down to per-file, per-person level • Data requests/analysis • Completely automated • All logging done automatically

4. Proposed SolutionAutoMODAutomated Model Diagnostic Facility • Web based-Interface • Automated Upload • Automated Atlases (pre-run/offline diagnosis) • Online Diagnosis • Searchable Database for Model, Simulations, Diagnosis • Leverage from PCMDI’s experience

5. PCMDI • Goal • Serve Climate Community • Computation Team Goals • Provide the scientific community with tools to allow them to focus on science NOT on technical aspects.

6. PCMDI Solutions • Analysis: Climate Data Analysis Tools • Data serving: ESG

7. Climate Data Analysis ToolsDesign/Philosophy • Leveraged from community’s work, for the community • Cutting-EdgeTechnology • Tomorrow’s Analysis Tools Today. • Quality driven but balanced by functional requirements. • Python based • Flexible • Efficient • Open Source, Open Community. • Knowledge sharing • Time saving • Leveraging from others work

8. Climate Data Analysis Tools (CDAT) CDAT Climate Data Analysis Tools • Python based system • Added packages by community • One environment, • Community Software

9. Scripts / VCDAT Community Contributed Packages Grid Axis Variable Dataset VCS VTK XMGRACE CDMS MA MV Numeric f2py Pyfort Python C or Fortran Cdunif.so Canvas Graphics method _vcs.so NETCDF HDF4 PP GrADS DRS CDMS XML BOXFILL ISOFILL ISOLINE MESHFILL VECTOR SCATTER TAYLOR XvsY XYvsY YXvsX OUTLINE OUTFILL CDAT’s Modularity

10. CDATExamples

11. Earth System Grid • Collaboration and data sharing • Location-independent equal-access to shared resources • (data, visualization, supercomputers, experiments, whiteboard, etc..) • Evolution from centralized data sharing to distributed data-sharing. • Allow for geographically distributed teams. • Allow researchers to focus on science not data set manipulation.

12. Site 4 Site 3 Site 1 Site 2 Catalog Catalog Catalog Catalog ESG Node ESG Node ESG Node ESG Node Data Data Data Data Federated Metadata Catalog Federated ESG Sites ESG Virtual Server HTTP User Interface Server ESG Product Request Protocol Publish to Federated Catalog

13. External collaborationsPortal examples Live Access Server Pacific Marine Environmental Laboratory GeoSPLAT British Atmospheric Data Centre

14. AutoMODAutomated Model Diagnostic Facility • Freely available • Automated Upload • Automated Atlases • Searchable Database for Model, Simulations, Diagnosis • System can evolve to incorporate new/future standard

15. Existing StructureThe “manual” approach Model Output Archive Analysis scripts (python/CDAT, IDL, Ferret, Tecplot) Analysis/MIP Guru(s) User

16. AutoMOD StructureThe “Automated” approach Model Output Archive Database Web Server (powered with CDAT) Simple HTTP FTP User

17. AutoMODMySQL Database • Variable information • Modeling Groups Information • Various versions information • Simulations Information (associated with a model version) • Working Groups Information • Unix-like Read/Write Authorization, per Group/User, Model version or simulation level

18. AutoMOD Web Interface • Apache Server, with builtin CDAT via mod_python module • MySQL interface • Permissions • Archive Content • AutoMOD Project Info (users, groups, etc…) • Online CDAT Diagnosis • Allows to restrict ressources available to the user (server swamping issues) • Use pre-loaded CDAT/Python (faster) • System 100% free and Open-Source

19. AutoMOD Web Interface

20. AutoMOD Web Interface

21. AutoMOD Web Interface

22. AutoMOD Web Interface

23. AutoMODUsers Requirements/Restrictions • Data/Metadata must adhere to strict standards • NetCDF format, CF compliant • http://www.cgd.ucar.edu/cms/eaton/cf-metadata • Data must be pre-processed by user via “output” subroutine, provided to them. • No “user-defined” Diagnosis

24. AutoMODStatus and Upcoming Tasks • Web administrative interface finished, only needs “look and feel” finish. • OCMIP2 Data in the system • First basic diagnosis in beta phase • Extensive “Atlas” and “online” diagnosis will be added through the year. • Possibility to move to ESG Data serving

25. Portability • CF Compliant data should plug-in without any changes. • Potential changes could be: • Project specific assimilation script • Adding/Removing Variable information stored into the database • Adding project specific diagnosis will obviously be needed

26. PortabilityPossible Extensions • Replace Data archive system, with Earth System Grid. • Not needed by AutoMOD at the moment, but would be for projects such as IPCC (huge datasets, more stringent security requirements)

27. Conclusions • AutoMOD: Hassle-free environment • Leveraged from PCMDI’s experience: • Powerful CDAT as diagnosis backend • Possibility of use of ESG system Easy upload of dataset • Centralized resources • Easy User Interface • Ever growing list of diagnosis • Easily Portable