260 likes | 338 Views
SCEC Capability Simulations on TeraGrid. Yifeng Cui San Diego Supercomputer Center. SCEC Computational Pathways. SCEC Capability Simulations on Kraken and Ranger.
E N D
SCEC Capability Simulations on TeraGrid Yifeng Cui San Diego Supercomputer Center
SCEC Capability Simulations on Kraken and Ranger • ShakeOut-D: 600 x 300 x 80 km domain, 100m resolution, 14.4 billion grids, upper frequency limit to 1-Hz, 3 minutes, 50k time steps, min surface velocity 500m/s, dynamic source (SGSN), velocity properties SCEC CVM4.0, 1 terabyte inputs, 5 terabytes output • ShakeOut-K: 600 x 300 x 80 km domain, 100m resolution, 14.4 billion grids, upper frequency limit to 1-Hz, 3 minutes, 50k time steps, min surface velocity 500m/s, kinematic, velocity properties SCEC CVM4.0, 1 terabyte inputs, 5 terabytes output • Chino Hills: 180x125x60km, 50m resolution, 10.8 billion grids, 80k time steps, upper frequency limit to 2-hz, using both SCEC CVM4 and CVM-H velocity models • Latest simulation completed within 1.8 hours for ShakeOut-D run on 64k Kraken XT5 cores. ShakeOut-D 2-hz benchmark achieved sustained 49 Teraflop/s. Source: Yifeng Cui, UCSD
Validation of Chino Hills Simulations • Goodness-of-fit at 0.1-0.5 Hz for synthetics relative to data from M5.4 Chino hills earthquake. • Seismogram comparisons of recorded data (black traces), CVM-S synthetics (read traces) and CVM-H synthetics (blue traces)
SCEC Capability Simulations Workflow • Inputs are in TB size with spatial and temporal locality • Input partitions are transferred between TG sites, • Simulation outputs are backed up on TACC Ranch and NICS HPSS. • Visualization done on Ranger
initial stress input settings media input yes Serial or parallel source partitioning and split options Serial or parallel mesh partitioning and options source fault input no if >2 0-4 source mode 0-3 media mode if 0 or 2 Read in Adapting SCEC Applications to DifferentTeraGrid Architectures Read in Spatial Locality Temporal Locality media partition if 2 save partition save partition if 1 if 1 if 0-1 if 2 if 0 or 2 if 1 solver if I/O mode 1 restart settings sfc or sfc+ vlm if >0 0-max checkpoints 0-1 MD5 mode 0-1 output mode 0-1 accumulation 0-1 performance ckpts if 1 MD5 if 1 performance measurement SAN switch Instrastructure SAM-QFS HPSS Source: Cui et al. Toward Petascale Earthquake Simulations, ActaGeotechnica, June 2008
Mesh Partitioned in Advance • Data locality
Mesh MPI-IO Contiguous Read • Data Continuity • Read XY plane and then redistribute data
SCEC Capability Simulations Performance on TeraGrid * Benchmark
Other efforts in progress supporting SCEC larger-scale simulations • Single CPU optimization, for example division is very expensive, by reducing division work, we have observed performance improvements by 25-45% on up to 8k cores • Workflow: end-to-end approach to automate procedures of capability simulations • Restructuring code to prepare as SCEC community code, emphasize modularity, re-usability and ease of integration • Developing hybrid code with a two level MPI/OpenMP
Acknowledgements • This work has received technical supports from varied TeraGrid sites, in particular: • Tommy Minyard and Karl Schultz of TACC • Kwai Lam Wong and Bruce Loftis of NICS • AmitChourasia of SDSC • SCEC Collaborations