1 / 26

SCEC Capability Simulations on TeraGrid

SCEC Capability Simulations on TeraGrid. Yifeng Cui San Diego Supercomputer Center. SCEC Computational Pathways. SCEC Capability Simulations on Kraken and Ranger.

tejana
Download Presentation

SCEC Capability Simulations on TeraGrid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SCEC Capability Simulations on TeraGrid Yifeng Cui San Diego Supercomputer Center

  2. SCEC Computational Pathways

  3. SCEC Capability Simulations on Kraken and Ranger • ShakeOut-D: 600 x 300 x 80 km domain, 100m resolution, 14.4 billion grids, upper frequency limit to 1-Hz, 3 minutes, 50k time steps, min surface velocity 500m/s, dynamic source (SGSN), velocity properties SCEC CVM4.0, 1 terabyte inputs, 5 terabytes output • ShakeOut-K: 600 x 300 x 80 km domain, 100m resolution, 14.4 billion grids, upper frequency limit to 1-Hz, 3 minutes, 50k time steps, min surface velocity 500m/s, kinematic, velocity properties SCEC CVM4.0, 1 terabyte inputs, 5 terabytes output • Chino Hills: 180x125x60km, 50m resolution, 10.8 billion grids, 80k time steps, upper frequency limit to 2-hz, using both SCEC CVM4 and CVM-H velocity models • Latest simulation completed within 1.8 hours for ShakeOut-D run on 64k Kraken XT5 cores. ShakeOut-D 2-hz benchmark achieved sustained 49 Teraflop/s. Source: Yifeng Cui, UCSD

  4. Validation of Chino Hills Simulations • Goodness-of-fit at 0.1-0.5 Hz for synthetics relative to data from M5.4 Chino hills earthquake. • Seismogram comparisons of recorded data (black traces), CVM-S synthetics (read traces) and CVM-H synthetics (blue traces)

  5. SCEC Capability Simulations Workflow • Inputs are in TB size with spatial and temporal locality • Input partitions are transferred between TG sites, • Simulation outputs are backed up on TACC Ranch and NICS HPSS. • Visualization done on Ranger

  6. initial stress input settings media input yes Serial or parallel source partitioning and split options Serial or parallel mesh partitioning and options source fault input no if >2 0-4 source mode 0-3 media mode if 0 or 2 Read in Adapting SCEC Applications to DifferentTeraGrid Architectures Read in Spatial Locality Temporal Locality media partition if 2 save partition save partition if 1 if 1 if 0-1 if 2 if 0 or 2 if 1 solver if I/O mode 1 restart settings sfc or sfc+ vlm if >0 0-max checkpoints 0-1 MD5 mode 0-1 output mode 0-1 accumulation 0-1 performance ckpts if 1 MD5 if 1 performance measurement SAN switch Instrastructure SAM-QFS HPSS Source: Cui et al. Toward Petascale Earthquake Simulations, ActaGeotechnica, June 2008

  7. Mesh Partitioning

  8. Mesh Serial Read

  9. Mesh Partitioned in Advance • Data locality

  10. Mesh MPI-IO Scattered Read

  11. Mesh MPI-IO Contiguous Read • Data Continuity • Read XY plane and then redistribute data

  12. Comparisons of Mesh Approaches

  13. Source Partitioning

  14. Synchronous Communication

  15. Synchronous Communication

  16. Asynchronous Communication

  17. Asynchronous Communication

  18. Asynchronous Communication

  19. Asynchronous Communication

  20. SCEC Capability Simulations Performance on TeraGrid * Benchmark

  21. Other efforts in progress supporting SCEC larger-scale simulations • Single CPU optimization, for example division is very expensive, by reducing division work, we have observed performance improvements by 25-45% on up to 8k cores • Workflow: end-to-end approach to automate procedures of capability simulations • Restructuring code to prepare as SCEC community code, emphasize modularity, re-usability and ease of integration • Developing hybrid code with a two level MPI/OpenMP

  22. Acknowledgements • This work has received technical supports from varied TeraGrid sites, in particular: • Tommy Minyard and Karl Schultz of TACC • Kwai Lam Wong and Bruce Loftis of NICS • AmitChourasia of SDSC • SCEC Collaborations

More Related