GridBench: A Tool for Benchmarking Grids

1. GridBench: A Tool for Benchmarking Grids George Tsouloupas Marios Dikaiakos Dept. of Computer Science University of Cyprus {georget,mdd}@ucy.ac.cy

2. Introduction • Utility of Grid Benchmarking: The experimental evaluation of Grid Infrastructures. • Researcher: • Characterization of factors of application and infrastructure performance; • provide insight to properties of Grid Architectures; • Study the dynamic nature of the Grid. • User: • Evaluate fitness of a collection of resources for running a specific application. • Investigate/troubleshoot application performance (by comparison to benchmark runs)

3. Introduction (cont’d) • Utility of Grid Benchmarking (cont’d) • Developer: • Evaluate fitness of a collection of resources for running a general class of applications, by looking into different programming modelsadigms or which type of middleware to use. • Integrator/Administrator: • Evaluate the employment of specific middleware • Evaluate scheduling and resource allocation algorithms.

4. Benchmarking the Grid: A Hierarchical Framework Targets: • Individual Resources (cluster nodes, Storage Elements) • Sites (clusters, SMPs) • Grid Constellations (i.e. multiple sites / VOs) • Middleware (e.g. mpich-g2) Benchmarks: • Micro-benchmarks –isolate basic performance characteristics • Micro-kernel Benchmarks –synthetic codes • Application Benchmarks –derived from real applications deployed on grids

5. A Simple Model • VO • SITE • Storage • NODE • NODE • … • Site • Storage • NODE • NODE • NODE • …

6. GridBench Design: Software Architecture Components: • RSL/JDL Compiler (XML benchmark descriptions to job description languages) • Orchestrator (benchmark execution, collection of results) • Benchmark component (the benchmark executable) • Monitoring component (collects monitoring information) • Archive Database (benchmark result archive) • Information provider (results to information services) • Benchmark Definition GUI • Benchmark Browser GUI

7. GridBench Design: Software Architecture

8. GridBench Design: Archival & Publication of Results • Benchmark results are archived in a native-XML database (Apache Xindice) • The Benchmark results are stored together with their GBDL definition. • Results are meaningless without the specific parameters etc. that were used • (Reference to) Monitoring data also archived. • Comprehension/Analysis of results is (hopefully considerably) enhanced when combined with infrastructure monitoring during the run. • Benchmark results related to a Site are published to the Site’s MDS for easy access by users and schedulers

9. Micro-benchmarks & Micro-kernels • Micro-benchmarks at the Worker-node level • EPWhetstone: embarrassingly parallel adaptation of the serial whetstone benchmark. • BlasBench: evaluate serial performance of the BLAS routines. • Micro-benchmarks at the Site level • Bonnie++: Storage I/O performance • MPPTest: MPI performance measurements • Micro-benchmarks at the VO level • MPPTest: MPI performance measurements (spanning sites) • gb_ftb: File Transfer Benchmark • Micro-kernels at the Site level • High-Performance Linpack • Selected kernels from the NAS Parallel Benchmarks • Micro-kernels at the VO level • Computationally Intensive Grid Benchmarks

10. GridBench Design: Meta-data GBDL Main Parts: • Components • Metrics • Parameters • Location • Co/pre-requisites • Monitors • Metrics

11. Metrics representation • Benchmark-level metrics Vs component-level metrics (e.g. CIGB completion time Vs EP component completion time) • Single-value metrics <metric name="epwhetstone_IPS“ type="value"> <vector unit="MIPS">53</vector> </metric> • Multi-value metrics <metric name="xfer-rate" type="list"> <vector unit="bps"> 2039430,2083930,1909830,2184750,...</vector> <vector unit="second" toffset="1055327287"> 0,10,20,30,...</vector> </metric>

12. Monitoring • GBDL can contain monitoring definitions. <component name="data-transfer" ID="xfer01">...</component> <monitor type="RGMA" source="ccwp71.in2p3.fr:3306" query="select * from NetworkTCPThroughput where NMIdSource='adc0003.cern.ch' and NMIdDestination='ccwp7.in2p3.fr' <parameter name="begin">comp-begin="xfer01"<ameter> <parameter name="end">comp-end="xfer01"<ameter> </monitor> • Implies that a monitoring client must be implemented/adapted (potentially several monitoring clients can be employed)

13. Compilation to RSL/JDL GBDL • XML-based GBDL to Job Description • Support for simple jobs can be through the use of simple templates. (executable, parameters and locations are transformed to simple RSL/JDL ‘scripts’) • Most benchmarks need special command-line parameter formatting, or parameter files. -> ParameterHandler for each benchmark that needs it. Paramhandler compiler RSL JDL …

14. Compilation to RSL (example) GBDL <benchmark date= name="nbody"> <component id="A" name="nbody“ type="mpi"> <location type="single"> <resource cpucount="2” name="ce1.grid.ucy"/> <resource cpucount="2” name="ce2.grid.ucy"/> </location> <parameter name="executable" type="attribute"> /bin/nbody.exec </parameter> <parameter name="nparticles" type="value"> 1000 </parameter> </component> </benchmark> RSL +(&(resourceManagerContact= “ce1.grid.ucy") (label="subjob 0") (environment= (GLOBUS_DUROC_SUBJOB_INDEX 0)) (count=2) (arguments="-n 1000") (executable="/bin/nbody.exec" )) (&(resourceManagerContact= "ce2.grid.ucy ") (label="subjob 1") (environment= (GLOBUS_DUROC_SUBJOB_INDEX 1)) (count=2) (arguments="-n 1000") (executable="/bin/nbody.exec" ))

15. Compiler GridBench UI Monitor Orchestrator Benchmark Component Executable Xindice Benchmark Component Executable Experiments User Interface CE RSL GBDL GRAM std out GRAM std out CE Archiver SE

16. Results

17. WIP and Future Work • Complete GBDL specification • Revise Metric and Monitoring representations • Implementation of more benchmarks.