The Earth System Grid

1. UCRL-PRES-148116 The Earth System Grid Presented by Dean N. Williams PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams (LLNL) http://www.earthsystemgrid.org Presented at: The “EO GRID” Workshop Frascati, Italy Earth System Grid - Williams

2. Earth System Grid (ESG): Overview • Funded by the Scientific Discovery through Advanced Computing (SciDAC), this program seeks a new paradigm in the climate change community evolving from centralized data sharing to distributed data-sharing. • Enabling geographically distributed teams of researchers to effectively and rapidly acquire knowledge and understanding of massive amounts of climate data holdings. • Multiple interfaces to ESG will allow researchers to focus on science and not issues with data receipt, format, and data set manipulation. Earth System Grid - Williams

3. ESG: Why is ESG Important to the U.S. Climate Change Program • Climate model output and quality observations are vital to providing timely assessments of climate change and impacts. • Recent U.S. and IPCC assessment efforts made it clear the lack of accessibility to model simulations is a major problem for future assessments. • Access to retrospective climate data (input and output) needed to enable a feedback mechanism to tie researchers directly back to quality control and diagnostics of models. • Researchers require access to “format independent” climate and observational data for case-study & training. • In the U.S., climate simulation can be viewed as a systems problem, requiring a team of multi-agencies and institutions working together in collaboration. Earth System Grid - Williams

4. ESG: U.S. Collaborations & Development ANL: Computational grids, & grid-based applications LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison USC/ISI: Computational grids, & grid-based applications ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing NCAR: Climate change predication and scenarios Earth System Grid - Williams

5. ESG: Requirements & Priority Matrix Earth System Grid - Williams

6. ESG: U.S. Department of Energy (DOE) Next Generation Internet (NGI) Project • ESG-I (past): • Focused on developing techniques for the high-speed data movement between sites and users (e.g., the secure highly efficient File Transfer service, called gridFTP, developed by ANL (i.e., Globus)) • Developed replica catalogs for keeping track of data locations • Developed request manages for coordinating multiple transfers • Developed a grid-enabled version of LLNL’s data analysis package Earth System Grid - Williams

7. ESG: ESG-I Architecture Earth System Grid - Williams

8. Local Disks tape system tape system Network LDAP/Sever Metadata Catalog ANL LDAP/Server Metadata Catalog SC ‘01 parallel disk system ESG: ESG-I Team Presented their work at Supercomputing 2001 RAID CLOUD LDAP/Sever Metadata Catalog LLNL TERRAIN U & V LDAP/Sever Metadata Catalog LBNL parallel disk system Earth System Grid - Williams

9. ESG: DOE SciDAC Project • ESG-II (present): • Building upon the substantial work of ESG-I • Grid-wide services supporting authentication, authorization, data discovery, and user specified analysis • Metadata services supporting remote data browsing, querying, accessing, displaying, etc. • Filtering services performing intelligent model specific analysis before delivering the results to the user • Integrate next-generation data analysis and visualization applications (such as ongoing work at LLNL and NCAR), web-based data portals and other thin clients supporting the Distributed Oceanographic Data System (DODS), and collaborative problem-solving environments. Earth System Grid - Williams

10. ESG: ESG-II Architecture Earth System Grid - Williams

11. ESG: Metadata Services ESG CLIENTS API & USER INTERFACES PUBLISHING ANALYSIS & VISUALIZATION SEARCH & DISCOVERY ADMINISTRATION BROWSING & DISPLAY HIGH LEVEL METADATA SERVICES METADATA EXTRACTION METADATA ANNOTATION METADATA & DATA REGISTRATION METADATA BROWSING METADATA QUERY METADATA AGGREGATION METADATA VALIDATION METADATA DISPLAY METADATA DISCOVERY CORE METADATA SERVICES METADATA ACCESS (update, insert, delete, query) SERVICE TRANSLATION LIBRARY METADATA HOLDINGS mirror Dublin Core XML Files Data & Metadata Catalog Dublin Core Database COARDS Database COMMENTS XML Files Earth System Grid - Williams

12. ESG services: information, replica, metadata, community authorization Data consumers ? R M CAS Computational resources Online storage systems Data producers ESG: Collaboration Network Grid and Network Infrastructure Earth System Grid - Williams

13. ESG: Example of a Web-based Data Portal (currently serving 40+ simulations of AMIP, CMIP, and PCM data, and growing) Earth System Grid - Williams

14. ESG: Example of a Client Application Earth System Grid - Williams

15. Import cdms db = cdms.open(“ldap://localhost:389/database=demo,ou=PCMDI,o=LLNL,c=US”) f = db.open( “ncep_reanalysis_mo”) ds = f(‘ts’) ESG: Example of a Script Access • The next-generation language, Python, is used to access the Earth System Grid at LLNL Earth System Grid - Williams

16. ESG: Concluding Statements • ESG is a highly collaborative effort and will allow users to quickly access data storage facilities storing petabytes of raw or processed data in an application independent manner. • Payoffs of this distributed collaborative infrastructure, would include: • distributed data-sharing • Simplified data discovery of climate data • Large-scale climate data processing and analysis • Increased collaboration among climate research scientists • Aid in climate assessments and estimates of future climate variability and trends • For more information on ESG, visit our website at: http://www.earthsystemgrid.org Earth System Grid - Williams