1 / 38

John Drake (ORNL), Phil Jones (LANL) for the SciDAC CCSM Consortium

COLLABORATIVE DESIGN AND DEVELOPMENT OF THE COMMUNITY CLIMATE SYSTEM MODEL FOR TERASCALE COMPUTING. John Drake (ORNL), Phil Jones (LANL) for the SciDAC CCSM Consortium. Bob Malone.

inga
Download Presentation

John Drake (ORNL), Phil Jones (LANL) for the SciDAC CCSM Consortium

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COLLABORATIVE DESIGN AND DEVELOPMENT OF THE COMMUNITY CLIMATE SYSTEM MODEL FOR TERASCALE COMPUTING John Drake (ORNL), Phil Jones (LANL) for the SciDAC CCSM Consortium Bob Malone

  2. The SciDAC CCSM Consortium consists of PI: J. Drake5, P. Jones4, Site-Contacts: W. Collins7, P.Cameron-Smith3, C. Ding2, S. Ghan6, J. Larson1, W. Washington7, R. Rood8, Co-Is: J. Baumgardner4, L. Buja7, S. Chu4, M. Vertensten7, P. Duffy3, J. Dukowicz4, S. Elliot4, D. Erickson5, M. Ham5, Y. He2, F. Hoffman5, E. Hunke4, R. Jacob1, P. Jones4, J. Lamarque7, W. Lipscomb4, R. Malone4, M. Maltrud4, D. McKenna7, A. Mirin3, W. Putman8, W. Sawyer8, J. Schramm7, T. Shippert6, R. Smith4, Y.H.Tseng2, P. Worley5 1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Los Alamos National Lab, 5Oak Ridge National Lab, 6Pacific Northwest National Lab, 7National Center for Atmospheric Research, 8NASA-Goddard Space Flight Center

  3. http://www.osti.gov/scidac/updates2004

  4. CCSM3 Released June 2004 - Mission Accomplished? • CCSM Model Development • SciDAC/ CCSM Collaboration – Drake(ORNL), Jones(LANL), Collins (NCAR) • Software Engineering - Worley(ORNL), Vertensten(NCAR) • Coupler Developement - Larson(ANL), Kauffman(NCAR) • Atmospheric Dynamical Cores – Drake(ORNL), Hack (NCAR) • POP Ocean Code – Jones (LANL), Norton (NCAR) • CICE Sea Ice - Lipscomb(LANL), Hunke (LANL) • Atmospheric Chemistry – Cameron-Smith (LLNL), Lamarque (NCAR) • Land and River Modeling – Hoffman (ORNL), Bonon (NCAR) • Biogeochemistry -- Erickson(ORNL), Thornton (NCAR) • Global Change Working Group – Washington (NCAR) • Software Engineering Working Group - Worley(ORNL) Co-chair • Ice Model Working Group - Hunke (LANL) Co-chair • Performance Evaluation Research Center (SciDAC PERC ISIC) • Climate benchmarking– Worley (ORNL) • Vectorization (CRAY, NEC) • Grids and Frameworks: • SciDAC Earth System Grid • – Williams (LLNL), Middleton(NCAR) • Earth System Modeling Framework • –Suarez(NASA-GSFC), Deluca(NCAR) • Other SciDAC ISICs: CCA, TOPS, TSTT, …

  5. SciDAC Contributions to CCSM3 Release • SWE design, parallel algorithms • Coupler CPL6 and MCT, MPH libraries • Performance/ optimization / vectorization • POP 1.4.3 ocean code from LANL • CICE sea ice code from LANL • DOE computer resources for development • DOE computer resources for IPCC runs

  6. Goals of the Consortium • Performance portability for the CCSM • Open software design process • Layered software architecture to insulate modeling • Improving SWE practices • Readiness for climate change simulations • Improvement of methods and models supporting more comprehensive coupled climate simulations • Higher resolution • Dynamics algorithms • Atmospheric chemistry • Ocean and Land ecology • Carbon cycle, aerosols

  7. The State of the Software

  8. Porting and Vectorization for CRAY X1 and Earth Simulator • SciDAC Workshop on Vectorization in Feb. 2003 started the effort. • CCSM3 release includes vector mods and is used for IPCC on Japanese Earth Simulator, validated on Cray X1 • Effort spanned NCAR, NASA-Goddard, ORNL, LANL, LBNL, Cray, NEC, Fujitsu, CRIEPI Pat Worley will speak on vectorization and other performance issues tomorrow.

  9. Which application characteristics matched the vector architectures well/poorly: POP ocean code (best)- finite difference with halo-updates and conjugate gradient barotropic solver CSIM ice model (excellent) - only minor mods to finite difference with halo-updates, incremental remapping advection CAM atmosphere (good) - vector radiation/physics calculations, spectral dynamics, semi-Lagrangian advection with halo updates CLM2 (poor) - pointer data structure required a complete re-write to get an explicit loop in each processing routine. After re-write vector performance is excellent and other platforms are faster as well. CPL6 - coupler has modest vector operations (distributed sparse multiply for interpolation and field regridding) Fast network bandwidth allows good distributed performance on spectral methods and data transpositions especially important in CAM Low network latency allows fast collectives and halo updates especially important in POP Vector Adaptation

  10. Community Land Model (CLM3.0) Vectorized for Cray X1 and Earth Simulator • In addition to clumps, another set of structures, called "filters," was added to better support vectorized processing of columns and plant functional types (PFTs). • Filters group like columns or PFTs based on their process-specific categorization and are used for indirect addressing into the main data structure hierarchy. • Filters are created for snow, non-snow, lake, non-lake, and bare soil columns and PFTs by clump. When the Dynamic Global Vegetation Model is enabled, additional filters are created for naturally-vegetated PFTs. Forrest Hoffman poster presentation.

  11. Physics Load Balancing

  12. Evolution of Performance in CAMPat Worley, John Drake, Art Mirin, Will Sawyer, Bill Putman • CAM2.0 Eulerian Spectral Model at T42L26 • Hybrid MPI/OpenMP programming paradigm • Cache friendly chunks, load balance, improved algorithms • FV Core on IBM NHII in lat-vert decomposition • NASA- DAO’s mod_comm replacing PILGRIM

  13. Performance of POP and CLM

  14. Simulations

  15. AOMIPElizabeth Hunke • Arctic Ocean Model Intercomparison Project run 1948-2002. • It's global, 0.4 deg, coupled POP2.0 and CICE3.1_beta.

  16. IPCC Fourth Assessment Computational support for IPCC simulations using CCSM3

  17. Plans for C4MIP SimulationsBiogeochemistry Working Group • Start at a T31 resolution. • Simulations will be "F" runs with active atmosphere and land models using data ocean model and thermodynamic ice run through the coupler. • Software engineering work is needed on the data ocean and CSIM ice models so that they can cycle through 25 years of forcing data for spin-up. • As spin-up progresses, turn on CO2 flux to the atmosphere with advection, complete implementation of land use change for agriculture, add fossil fuel emissions forcing, etc. • As time is available on the Cray X1, the CASA' and CN simulations for C4MIP could be run concurrently at ORNL.

  18. Eddy-Resolving Ocean Obs 2 deg 0.28 deg 0.1 deg

  19. Atmospheric Chemistry for A2 ScenarioJ-F Lamarque, S. Walters

  20. Subgrid OrographySteve Ghan and Tim Shippert

  21. Subgrid scheme

  22. Progress on Subgrid Orography • Scheme applied to CAM and CLM • Euler dycore • Finite-volume dycore (1-D and 2-D domain decomposition) • Developmental branch updated to cam2_0_2_dev55 (IPCC physics) • Bit-for-bit agreement between SP, SMP, SPMD • Restarts bit-for-bit agreement • Bit-for-bit agreement with dev trunk if subgrid scheme turned off • Energy conservation demonstrated • Load balancing within nodes for both dycores • Load balancing between nodes for both dycores (FV 1-D only) • Runoff distributed according to elevation of river transport model surface elevation • Simulations at T42 and 2x2.5º resolution • Starting IPCC downscaling simulations

  23. Downscaled Temperature

  24. March snow

  25. North American Glaciers

  26. River Discharge

  27. SWE and Model Development

  28. Community Atmospheric Model Plans (May04) • Continue Optimizations for IPCC • Vector version for Cray X1 and NEC SX7 (June 2004) • IBM p690 improvements • Increased resolution for spectral cores • SLT 2-D decompositions October 2004 • SLD reduced grid Dec 2004 • SLD lagrangian vertical coordinate June 2005 • SLD new advection (incremental remapping?) October 2005 • Toward a chemical atmosphere model • Increased resolution (B ->D) for Lin-Rood Oct 2004 • WACCM configurations, chemistry, vectorization Oct 2004 • Advection optimization for a Chemical atmosphere Dec 2004 • IPCC Simulations with subgrid orography Jan 2005 • T170 Eulerian and FV (D-grid) downscaling of selected IPCC runs • Implications of SP-CAM • Revision of software design and the Unified Model • Block/ESMF dycores Dec 2004 • Utility layer standardization March 2005 • Single executable CCSM (with NCAR) Completion date: October 2004. • A prototype code is done. • Integrating parallel netCDF into Ziolib Complete: Nov 2004. • Incorporating ZioLib into CAM: Completion date: December 2004.

  29. Unified Model Architecture • Issues: • Model componentization • Utility layer • Extensibility • Performance • Single or multiple • executables • Stand alone execution • Unit testing • CPL6 -- Implemented, Tested, Deployed based on the Model Coupling Toolkit and the MPH3 multi-processor handshaking library • ESMF and CCA prototypes • Single Executable tests underway Version 2.0.2 Released 11 July 2004

  30. Distributed array In (X,Z,Y) index order Remapped on staging PEs In (X,Y,Z) index order I/O staging PEs write in parallel height (Z) latitude (Y) Z-decomposition longitude (X) MPI gather, global transpose and single-PE write MPI gather and global transpose Total write times with ZioLib Remapping times with ZioLib Parallel I/OWoo-Sun Yang, Chris Ding Parallel decompositions of ZioLib Tested implementation in CAM Parallel performance of Ziolib Parallel NetCDF in conjunction with SciDAC SDM Center

  31. Ocean BiogeochemistryS. Elliot, S. Chu, M. Maltrud, D. Erickson • Ocean ecology model additions to Doney-Moore-Lindsey • Implemented in POP2.0 • Replace POP1.4.3 with POP2.0 • Use to drive DMS flux with atmosphere coupling (other chemicals as well) • LANL Ecosystem Model • nutrients (nitrate, ammonium, iron, silicate) • phytoplankton (small, diatom, coccolithophores) • zooplankton • bacteria, dissolved organic material, detritus • dissolved inorganic carbon (DIC), alkalinity • trace gases (dimethyl sulfide, carbonyl sulfide, methyl halides and nonmethane hydrocarbons) • elemental cyclings (C,N,Fe,Si,S)

  32. Atmospheric ChemistryP. Cameron-Smith, J.F. Lamarque, S. Walters, … • Gas-phase chemistry with emissions, deposition, transport and photo-chemical reactions for 89 species. • Experiments performed with 4x5 degree Fvcore – ozone concentration at 800hPa for selected stations (ppmv) • Mechanism development with IMPACT • A)    Small mechanism (TS4), using the ozone field it generates for photolysis rates. • B)     Small mechanism (TS4), using an ozone climatology for photolysis rates. • C)    Full mechanism (TS2), using the ozone field it generates for photolysis rates. Zonal mean Ozone, Ratio A/C Zonal mean Ozone, Ratio B/C

  33. Polar and THC

  34. HYPOP • Arbitrary Lagrangian-Eulerian vertical coordinate • Keep Lagrangian in deep ocean • Remap to z-coordinate in mixed layer • CSU SciDAC • New time stepping/mode splitting • Progress • Model currently working in z-coord mode • Examining vertical grid generators • Testing

  35. CICE • Incremental Remapping for Sea Ice and Ocean Transport • Incremental remapping scheme that proved to be three times faster than MPDATA, total model speedup of about 30% --added to CCSM/CSIM • CICE3.0 restructered for vector Community Sea Ice Model • Sensitivity analysis and parameter tuning test of the CICE code • Automatic Differentiation (AD)-generated derivative code

  36. SciDAC DOE Milestones for FY05 Q1: (LANL) • Non-interacting ocean biogeochemistry (ecosystem + trace gases) model implemented in the current version of the Parallel Ocean Program (POP) ocean GCM and executed as part of a coupled experiment within CCSM3 to simulate the response of ocean biogeochemistry to climate. Q2: (LLNL) • Code implemented in CCSM3 coupler and all components to exchange carbon and sulfur species; modify radiation scheme in the atmospheric component of the model to include the effects of sulfur species. Q3: (ORNL) • Initial testing of all model changes to include the sulfur cycle and the carbon cycle in coupled climate model simulations Q4: (ORNL) • Coupled simulations of the climate system including the carbon and sulfur cycles (POP2/biogeochemistry, CAM3/sulfur+carbon) with an interactive land surface model component.

  37. Block Buster Sequel: SciDAC II is coming …

  38. Go Provides Uses Provides Provides Uses Uses Provides Provides Provides Provides Provides Provides Provides Provides Provides Provides Uses Uses Uses Uses Uses Uses Uses Uses Uses Uses CPL6 AL_Coupling DP_Coupling CLM3 Advection Utilities CAM3 Carbon Model Stepon Physics Atm Chemistry Libraries Dynamics Component Model Architecture

More Related