* With assistance from Tsengdar Lee, Mike Little, Martha Maiden, Steve Berrick , David Considine

1. Computing and Computational Science at NASA in Support of Earth System ScienceDr. Jack A. Kaye*Associate Director for ResearchEarth Science DivisionNASA Science Mission Directorate8 September 2011 * With assistance from Tsengdar Lee, Mike Little, Martha Maiden, Steve Berrick, David Considine

2. NASA’s Research and Analysis Foci • NASA missions and observations are generating tremendous amount of data. • NASA funds projects to process and analyze data • Performs analysis, extracts information, and generate science results • Models are essential tools used in this process. NASA focuses on: • Observation-Driven Modeling • Data assimilation for prediction problems • Model-data intercomparisons • Observation System Simulation Experiments • Model-Synthesized Observations • Observational data reanalysis

3. High-End Computing (HEC) Portfolio Mission Objective: Develop and deliver productive high-end computing systems and services to support NASA’s mission needs; operate and manage these resources for the benefit of Agency users, customers, and stakeholders. Customers: Any NASA funded R&A projects are entitled to request for HEC resource as described in ROSES summary of solicitation Key Science and Engineering Products: • Model simulation output & analysis products • Data reanalysis products Two Computing centers: - High-End Computing Capability (HECC) - NASA Center for Climate Simulations (NCCS) Discover Supercomputer GSFC/NCCS System Components: Compute:NASA continues to build and operate two of the world’s largest supercomputers Storage: large online and tape storage for model data Networks:High speed network peered with academia networks Services: Supercomputing runtime environment, data analysis & visualization, application development & performance optimization, code porting & tuning, large-scale data storage & sharing, local & wide area networking, 24x7 help desk & user training Pleiades, Columbia Supercomputershyperwall2 ARC/HECC

4. High-Resolution Climate Simulations with Goddard Earth Observing System, Version 5 (GEOS-5) Cubed-Sphere Model Bill Putman, Max Suarez, NASA Goddard Space Flight Center; Shian-Jiann Lin, NOAA Geophysical Fluid Dynamics Laboratory GEOS-5 scales to 13,824 cores Running with GOCART interactive chemistry at 3.5- to 10-km globally for the first time Tuning physics to better resolve tropical deep convection and hurricane structure 5-day forecasts run at 10-km globally within a 3-hour operational window 3.5-km global simulations complete 3-day forecasts in 24-hours 2009 Atlantic Hurricane Bill 3.5-km GEOS-5

5. Tools for Getting the Data You Need • NASA's Earth Science Data Systems Program is comprised of core and community elements to serve the needs of a broad and diverse community. The Advancing Collaborative Connections in Earth System Science (ACCESS) Program focuses on community elements by leveraging technologies to improve the interconnectedness between data and users. Successful community elements can be infused into the core creating a vibrant, flexible, and continuously evolving infrastructure (science.nasa.gov/earth-science/earth-science-data/). • Finding the Right Data • Reverb allows you to explore and search thousands of data products across NASA’s distributed Earth science data centers using advanced search criteria that focus on multiple data aspects (reverb.echo.nasa.gov). • Other search interfaces are tailored to your discipline-specific research community and give you additional advanced capabilities optimized for your community’s tools and practices. • Accessing Data Your Way • Advanced science-supported data services including quality screening, subsetting, reprojection, and reformatting reduce data preparation time. • Standard Web services such as OPeNDAP, Google Earth, OGC Web Map Service, and Web Coverage Service allow data to be used immediately with your tools. • Visualize and Analyze Data Before Downloading • Online tools, such as Giovanni, offer you the ability to visualize, analyze, and compare data from multiple sensors on multiple satellites without having to download the data first. After the analysis, you can select and download only the pertinent data. Giovanni provides portals tailored to multiple science discipline communities. • Get Notified About New Data • Data casting and service casting are new, lightweight notification technologies that allow you to be notified of data or services meeting your specific criteria for quality or other characteristics. A-Train Data Depot uses Giovanni to visualize multiple parameters from multiple satellites online (disc.gsfc.nasa.gov/atdd).

6. NASA Earth Exchange Data - Modeling Platform Key elements: Monitoring Modeling Forecasting Local toGlobal Funded by NASA’s Earth Science Research & Analysis, Applied Sciences, and Technology Programs

7. Challenges • Software Development Practice • Earth system scientists are not trainedinsoftware engineering. This results in poorly organized and constructed model codes. • Computational scientists are not trainedinEarth system sciences. This results in poorly executed codes and significant model errors. • Validation and Verification • Model V&V done manually results in insufficient test and evaluation. • Models and observations are producing tremendous amount of data at different time and spatial scale. This result in significant challenge comparing the two. • Tools and Environment • Productivity significantly reduced due to the inadequate tools and development environment

8. Education and Workforce Development We need educated people who can bridge the science, computing, and computational disciplines. We are thinking of short-term intensive education programs to train Earth and Space Scientists in parallel computing and software engineering. Computing and modeling centers are looking for interns to work on specific NASA problems – see http://intern.nasa.gov/ We hope the academia will share in developing approaches to help develop the cross-disciplinary workforce needed for the future.

9. Research Opportunities at NASA • Research Opportunities in Space and Earth Sciences (ROSES) solicitation the CASC community may consider: • Data systems, data management, access, and data processing • Making Earth System data records for Use in Research Environments (MEaSUREs) • Advancing Collaborative Connections for Earth System Science (ACCESS) • Advanced Information System Technology • Modeling and Data Assimilation Research: • Earth science Modeling and Assimilation Program • Atmospheric Composition: Modeling and Analysis • Heliophysics, Astrophysics Theory Programs • Computational Modeling Algorithms and Cyberinfrastructure Program • Supplemental Education Awards for ROSES Investigators • ROSES Solicitation Web site: • http://nspires.nasaprs.com/external/solicitations/solicitations.do?method=open&stack=push

10. Backup

11. NCCS’s NVIDIA GPGPUs Supporting Climate Science • GEOS-5 column physics (Matt Thompson) • Porting of radiation code, the largest of GEOS-5 column physics codes, is well underway. • Porting of cloud microphysics and the turbulence code will begin shortly. • GEOS-5 dynamics code (Bill Putman) • An 85x speedup was measured on ~1/3 of the dynamics kernel of the fvCubed Sphere. • Discover NVIDIA GPGPU system (36.4 TFLOPs Peak): • 32 general-purpose 12-core Westmere nodes, each with 48 GB memory. • 64 “Tesla” M2070 GPGPUs, each with 488 CUDA cores and 6 GB memory (28,672 Total GPGPU Streaming Cores). • Integrated into Discover software stack and file systems for quick access and code porting • PGI CUDA FORTRAN compiler is available for porting codes GPGPUs. • For more information: • http://www.nccs.nasa.gov/gpu_front.html • http://sciences.gsfc.nasa.gov/600/highlights/stories/gpus.html GMAO researchers continued porting portions of the GEOS-5 General Circulation Model for use with General Purpose Graphical Processing Units (GPGPUs), considered well-suited to “column physics,” as NCCS staff integrated the 64 NVIDIA GPGPUs, attached to 32 Westmere computational nodes, delivered in March 2011. Diagram of the JCSDA-developed Community Radiative Transfer Model (CRTM) used by GEOS-5.2 Data Assimilation System. NVIDIA M2070 “Tesla” GPGPU

12. Understanding Global Landscapes at High Resolution For the first time in Landsat history of nearly 30 years, we can now process and create quantitative information about changes in global landscapes, in a matter of hours. Near realtimequantitative updates about global landscapes Building virtual teams For monitoring crop growth, deforestation, and impacts of natural disasters

13. High-End Computing Capability HECC Project is located within the NASA Advanced Supercomputing (NAS) Division at Ames, and managed under the Agency’s Strategic Capabilities Assets Program (SCAP). It supports the modeling, simulation, analysis and decision support activities for all four NASA Mission Directorates and NESC. • Computing Systems • Pleiades: 101,120-core SGI Altix ICE (Xeon quad), 1.3 PFlops • Columbia: 4608-processor SGI Altix (Itanium2) • hyperwall2: 1,024-processor, 128-node GPU cluster • Multiple secure front ends, metadata servers, object storage servers Scientists, engineers plancomputational analyses, selectingbest codes to address NASA’s complex mission challenges Outcome: Dramatically enhanced understanding & insight, accelerated science & engineering, and increased mission safety & performance • Balanced Environment • 3 PB disk filesystem; 30 PB tape archive • Archiving 500TB – 1PB every month • High-bandwidth WAN to other Centers, external peering • Large-scale and concurrent visualization • Resources enable broad mission impact • More than 400 science & engineering projects • More than 1,200 user accounts • Typically 400-500 jobs running 24x7 • Demand for cycles extremely high NAS viz experts applyadvanced data analysis & rendering techniquesto help users explore& understand large,complex computationalresults NAS software expertsutilize tools to parallelize& optimize codes, dramaticallyincreasing simulation performancewhile decreasing turn-around time NAS support staff help users productively utilizeNASA’s supercomputing environment to rapidlysolve large computational problems

14. NASA Center for Climate Simulations (NCCS) NASA Center for Computational Sciences (NCCS) supports SMD modeling and analysis activities: Atmospheric modeling– for climate and weather research Ocean modeling– for climate, chemistry, and biology Land surface modeling – for agriculture, land use, and water resource management Space & solar modeling– forfundamental astrophysics, space weather, and gravitational wave studies Coupled models and systems of models – in support of collaborative science efforts Observing system studies – to enhance the use and design of space instruments 14

15. NCCS – Computational Science ProjectsAugust 1, 2010 to July 31, 2011 • Earth Science (95 projects) • The GEOS-5 model completed a 10-kilometer “nature run” covering 2 years and can finish 3-day, 3.5-kilometer global weather forecasts within 24 hours. • Astrophysics (14 projects) • Researchers performed for the first time axisymmetric and global 3D simulations of the magnetorotational instability-driven accretion onto magnetized stars. • For the next IPCC assessment, coupled atmosphere-ocean models performed historical simulations reaching back to the year 1000. • NCCS enables real-time forecasts and provides data services for NASA field campaigns including DISCOVER-AQ (air quality) and HS3 (hurricanes). • New analyses show the benefits of greater adoption of American and European vehicle emission standards and the potential effects of sea level rise on New York City. • The most detailed, physically motivated model X-ray luminosity functions for X-ray binaries in nearby and low-to-high-redshift galaxies are starting to match observations. • Efforts to simulate formation of planetary systems include the first theoretical models to produce planets with masses ranging from the mass of Earth to 10 times the mass of Jupiter. Planetary Science (10 projects) Heliophysics(5 projects) • 3D, multifluid MHD studies of the interaction of the solar wind with Mars described new physical processes such as density plumes and asymmetries due to ion decoupling. • Simulations are being used to assess the effects of the active solar wind on Mercury’s magnetosphere during the MESSENGER spacecraft’s orbit. • MHD simulations of the solar wind are increasing understanding of Voyager observations in the outer heliosphere and the causes of space weather.

16. NCCS Encourages Large Scale Computational Science Research with 1000+ Cores • The NCCS advanced Discover SCU7’s use for SMD’s large-scale science investigators by extending access to researchers who intend to scale codes to 1000 or more cores. • Researchers’ requests for large-scale research runs have included the following applications: • GEOS-5 fvCubed Sphere • NASA Unified Weather Research and Forecasting model (NU-WRF) • Enzo adaptive-mesh refinement code for cosmological simulations 3.5-km GEOS-5 fvCubed Sphere 2009 Atlantic Hurricane Bill Simulated radar reflectivity 2004 Atlantic Hurricane Charley 2-km NU-WRF Primordial star simulation using Enzo

17. Introducing NASA Earth Exchange Collaborative Earth Science (http://nex.arc.nasa.gov) To encourage exploration and collaboration in the Earth science community to create new ways to understand and improve life here on Earth.