220 likes | 362 Views
SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group mueller@hlrs.de. What is SPEC?.
E N D
SPEC Benchmarks for Large Systems Matthias Mueller, Kumaran Kalyanasundaram, G. Gaertner, W. Jones, R. Eigenmann, R. Lieberman, M. van Waveren, and B. Whitney SPEC High Performance Group mueller@hlrs.de
What is SPEC? The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops suites of benchmarks and also reviews and publishes submitted results from our member organizations and other benchmark licensees.
SPEC High-Performance Group • Founded in 1994 • Mission: To establish, maintain, and endorse a suite of benchmarks that are representative of real-world high-performance computing applications. • SPEC/HPG includes members from both industry and academia. • Benchmark products: • SPEC OMP (OMPM2001, OMPL2001) • SPEC HPC2002 released at SC 2002
Currently active SPEC HPG Members • Fujitsu • HP • IBM • Intel • SGI • SUN • UNISYS • University of Purdue • University of Stuttgart
Where is SPEC Relative to Other Benchmarks ? There are many metrics, each one has its purpose Raw machine performance: Tflops Microbenchmarks: Stream Algorithmic benchmarks: Linpack Compact Apps/Kernels:NAS benchmarks Application Suites: SPEC User-specific applications: Custom benchmarks
SPEC OMP • Benchmark suite developed by SPEC HPG (High Performance Group) • Benchmark suite for performance testing of shared memory processor systems • Uses OpenMP versions of SPEC CPU2000 benchmarks and candidates
SPEC OMP/CPU2000 differences • Larger working set sizes, 1.6GB for OMPM2001, 6.5 GB for OMPL2001; it is 200MB for CPU2000 • Longer run times (>1000 s/cpu for CPU2000 vs >10,000 s/cpu for OMP2000 medium) • Focus on SMP systems, and issued by HPG • SPEC OMP based on work for CPU2000, SPEC OMP mixes integer and FP in one suite • OMPM is focused on 4-way to 16-way systems • OMPL is targeting 32-way and larger systems
SPEC OMP Results • 66 submitted results for OMPM • 24 submitted results for OMPL
SPEC HPC2002 Benchmark • Full Application benchmarks(including I/O) targeted at HPC platforms • Serial and parallel (OpenMP and/or MPI) • Currently three applications: • SPECenv: weather forecast • SPECseis: seismic processing, used in the search for oil and gas • SPECchem: comp. chemistry, used in chemical and pharmaceutical industries (gamess) • All codes include several data sizes
SPECenv execution models on a Sun Fire 6800 Medium scales better OpenMP best for small size MPI best for medium size
SPECseis execution models on a Sun Fire 6800 Medium scales better OpenMP scales better than MPI
SPECchem execution models on a Sun Fire 6800 Medium shows better scalability MPI is better than OpenMP
Current and Future Work • SPEC HPC: • Update of SPECchem • Improving portability, including tools • Larger datasets • New release of SPEC OMP: • Inclusion of alternative sources • Merge OMPM and OMPL on one CD • Adoption of new benchmark codes: • Should represent a type of computation that is regularly performed on HPC systems • We currently examine CPU2004 for candidates
Conclusion and Summary • Results of OMPL and HPC2002: • Scalability of many programs to 128 CPUs • Best choice of programming model (MPI,OpenMP, hybrid) depends on: • Hardware • Program • Data set size • SPEC HPC will continue to update and improve the benchmark suites.