VO- and Application- Centric Approaches to Service Level Agreement Marian Bubak, Jakub Moscicki,

1. VO- and Application- Centric Approaches to Service Level Agreement Marian Bubak, Jakub Moscicki, Marcin Radecki, and Tomasz Szepieniec Cyfronet AGH, Krakow, PL CERN / IT

2. Contents • VO-centric approach to SLA • Motivation • Basic requirements • SLA metrics • SLA execution • Bazaar tool • QoS from a user perspective • User-level vs system-level techniques • Tools: Ganga/DIANE • Examples of QoS metrics • Case-study:Lattice QCD 2008 • Summary

3. VO-centric Approach to SLA

4. Motivation • Large number ofVOs/usersand resources • Dynamic management is a must • Remote interactions • Limitation of automation • Policy managers want to decide about their resources • Start with human-in-the-loop SLAprocess

5. Aim: SLA based Resource Allocation

6. What is needed • Definition of meaningful and measurable SLA metrics • Communication patterns • (Re-)negotiation • Configuration validation • Tracking demands/policy changes • Complexity management and process traceability • SLA execution monitoring (including feedback from users) • So, we should • define the SLA process • and build a collaboration tool

7. EGEE Grid and Bazaar • Starting point • No standard QoS metrics • No procedures to express requirements • Resources become available in the infrastructureeven if not agreed with VO • Resource Allocation in Central Europe ROC (Bazaar Project)‏ • A procedure of tracking requests and responses to them • Registration and monitoring of SLAs between VOs and Resources Providers • Collaboration tool for tracking the process

8. Central European Region in EGEE • 8 countries, • 25 sites, • ~8000 cores, • ~850 TB storage • ~30 VOs

9. SLA Metrics • Common language for users and providers • Users: „I need to use x CPUs” • Providers prefer to speak about aggregated wall-clock time in specific period, without guarantee that resources will be available in (any) defined time • Expressive enough to satisfy users important requests • Aggregated time, parallel use, waiting time (queues), condition of environment • Configurable • providers need to have technical possibility to configure the resources according to the SLA (fabric layer need to support those requirements) • Measurable in execution time

10. Examples of SLA Metrics • Computational Resources • Guaranteed number of job slots in Local Batch System • CPUs or cores‏ • Total wall-clock time to be used in specified time period (in hours) • weekly, monthly • Access period (range of dates)‏ • Maximum wall-clock- and CPU-time of a single job (hours) • Maximum waiting time from job submission to make it running (in minutes)‏ • Average power of a single core (benchmark results like: SpecInt) • Capacity available for temporal use by a job (GB) • Memory available per core/CPU (GB) • Maximum latency between nodes in the cluster (ms)

11. Examples of SLA Metrics‏ • Grid Storage Resources • Storage quota guaranteed (GB) • Maximum latency in accessing files (optional, in ms) • Minimum bandwidth in accessing files (optional, Gb/s) • Storage quota for temporal use (optional, GB) • Time limit for temporal use of storage (optional, hours) • Period of using storage (dates from-to)‏ • General Resource QoS • Minimum resource availability (optional, in %) • Minimum resource reliability (optional, in %) • Maximum time to acknowledge trouble ticket (days) • Maximum time to resolve trouble ticket (days)

12. SLA Execution Stages in Bazaar The process is initialized by a VO by a call for resources Next, a resource providers define their proposal for SLA

13. States Transition Details Each statetransition must be confirmed by both sides Proper configuration is controlled by separated set of states

14. Bazaar Functionality • Call management - the user can perform call creation, edition and management. • SLA management including negotiation - site managers can create a contract as a response to a call. Both partners can negotiate contract conditions and track contract changes. • Notification management - system notifies a user via e-mail and user interface about actions like resource reconfiguration etc. • Feedback - VO managers can assess site's configuration and both partners can provide a general assessment of the collaboration when the contract has been completed. • Accounting and statistics - users can generate reports with resources usage statistics. In the next prototype, a tool shall enable obtaining data from EGEE accounting tools.

15. Bazaar in operation • Bazaar – a tool supporting resource allocation including SLA negotiation • Integrated with EGEE Operation Portal (CIC Portal)‏ • No cost of entry – data obtained from GOCDB and CIC-Portal VO-cards • Introduced into operations in Central European Region • Main features of Bazaar • Clear view on VOs demands for resources • Management of calls and SLAs between VOs and RCs • SLA negotiation support • E-mail notifications • Tracking of SLA changes

16. SLA in PL-Grid • PL-Grid Project • Grid operations center in Poland • 3 different infrastructures: EGI compliant (currently gLite-based), DEISA, cloud-like research grid • SLA Management in PL-Grid • We take ideas from Bazaar Project as a starting point • Develop SLA-centric model including • Impact on resources available at the technical level • Notifications on missing resources • Improvement on SLA monitoring and accounting • Integration with computational grants system

17. PL-Grid Operation Tools Architecture

18. Conclusions • Human in a negotiation loopseems to be unavoidable • SLAs should support VO and resource managers • Complexity management should be supported by Web 2.0 tools (collaboration tools with traceable processes)‏

19. QoS on the Grid with User-Level Scheduling

20. Some Grid applications • Data Analysis • extraction of (statistical) parameters from data using event loop • ATLAS experiment at LHC • Monte Carlo simulation • creation of statistical objects (e.g. histograms) or building images by generating large number of independent events • Geant4 simulations for radiotherapy in medical physics • Parameter sweep • running a large number of independent jobs in various configurations • Geant 4 regression tests • High-throughput activities • autonomous computing over long periods of time • Avian Flu Drug Search (bio-informatics)‏ • Lattice QCD (theoretical physics)‏ • High-performance, short-deadline activities • short-deadline performance peak • ITU frequency analysis for RRC06

21. QoS for scientific applications • In the Grid: the basic interaction of a user is sending jobs • efficient job/workload management plays central role • efficient scheduling often requires application-specific knowledge • which may be difficult at the system level • The system provides an appropriate QoS if it responds in an acceptable way to the user and is capable of automatically maintaining the processing goals defined by the user (measured by metrics) • Some QoS metrics (measure of user-defined goals)‏ • turnaround time • typically minimize the total execution time of the job • reliability / failure rate • response latency: time to obtain initial results • feedback from the execution • filling histograms with events -> significance of individual partial results decreases with time • prioritization/scheduling of the tasks • predictability/stability of the execution

22. Mechanisms for better QoS • In general QoS in NOT implemented on the Grid • Techniques for performance related metrics • dedication of resources (wasteful) • advanced reservations • difficult for some users who do not plan ahead interactive work • better scheduling: fast/slow queues (site configuration) • preemption: suspend lower priority job • migration: suspend and migrate elsewhere • better brokering: forecasting using monitoring systems (e.g. NWS) • Techniques for failure related metrics • metascheduling (JDL retry count, Condor) • Techniques for application-specific metrics • metascheduling (not generally implemented, e.g. out of scope of DAGs)

23. QoS Implementation Choices • QoS implementation • site service modifications • faster queues, scheduler modifications e.g. virtualization schemes with MAUI • middleware modification • checkpointing/migration, special services (e.g. GARA), Virtual Machines • system level modifications (unix kernel modules, special I/O) • user-level overlay schedulers (plot jobs, agents,...)‏ • Boundary conditions in a large Grid (e.g. EGEE)‏ • acceptance/deployment of middleware changes: very slow due organizational constraints • resource providers' constraints (site changes) • many sites cannot freely change their software (serving also non-grid users) • sysadmins do not like sudo-like programs • interfacing legacy applications

24. User-level overlay • Overlays are the only option if we talk about using existing Grid infrastructure at the large scale • LCG and EGEE Grid • the largest Grid infrastructure to date • over 250 sites • over 80K WNs • over 15 PB of storage

25. User-level tools • DIANE: helps smaller scientific communities using distributed (Grid) resources more efficiently • reduce the application execution time • reduce the manual work overhead by providing fully automatic execution and failure management, • efficiently integrate local and Grid resources • part of EGEE Respect suite • http://cern.ch/diane • Ganga: Job Management Interface • Submission gateway to many distributed systems • Easy job management and application configuration • http://cern.ch/ganga

26. User-level Overlay • User-level overlay • each user uses a (temporary) overlay which is created for the duration the computations (drawing courtesy of ThIS collaboration

27. Master/Worker backbone • Master/Worker processing of tasks • RunMaster executes on a local host • WorkerAgents execute as Grid jobs • TaskScheduler is a software component (python module) which may be arbitrarily customized or replaced • application plugins: • ApplicationWorker • ApplicationManager

28. Flexible architecture • 3 functional parts • Submitter: selection and acquisition of the resources • M/W: scheduling and execution control • Directory Service: late binding of resources • System is easily customized by plugins

29. Examples of QoS Metrics Selected examples of QoS metrics for different applications

30. QoS Metric: predictability of execution • Comparison of G4 Production on LCG: DIANE and direct submission • 6 sites / 173 CPUs / 100 VO-shared, 70 VO-dedicated • 207 tasks, direct: 1 task = 1 job, DIANE workers

31. QoS metric: reliability • Summary of ITU RRC06 runs • 200K jobs in less than 6 hours • worst case reliability: 0.0003 jobs lost run #jobs #task turnaround CPUh #WN comment 1 243K 26K 6.40h 425h 190 lost <10 tasks (3*e-04)‏ 2 237K 23K 6.30h 332h 125 lost 1 task (4*e-05)‏ 3 224K 40K 3.05h 192h 210 OK 4 218K 39K 1.05h 151h 320 OK • ITU RRC-06 (15 May–16 June 2006)‏ • 120countries (~1200delegates) negotiated thenew digital frequency plan • a part of a new international agreement • introduction of digital broadcasting • UHF (470-862 Mhz)‏ • VHF (174-230 Mhz)‏ • preceded by RRC-04 and other international meetings

32. 27% 64% QoS Metric: low latency on the Grid • RRC06 ITU job • 116 LCG workers • 3470 tasks • ~130 CPU h large span of task length not a priori known!

33. QoS Metric: stability of execution

34. Case study: high-throughput Lattice QCD simulation • application-aware scheduler: prioritize tasks based on the simulation parameters • active resource selection via Submitter (WorkerFactory)‏ • dynamically select resources based on their fitness for the application

35. Lattice QCD 2008 @ Grid • Study the behaviour of the critical point of quark-gluon plasma • The scientific results obtained by the LQCD project were published in a paper P. de Forcrand et al.: "The chiral critical point of Nf = 3 QCD at finite density to the order (μ/T)4" and are available at http://arxiv.org/pdf/0808.1096 • Monte-Carlo simulation of discrete space-time lattice • need a lot of CPU • relatively small data (~Gbs)‏

36. LQCD execution history • ongoing since May 2008 • several phases (application and system upgrades, power-cuts, etc...) • routinely production since September 2008 • runs unattended for months • operated by a single, not-a-Grid-expert user • large-scale • ~1000 running jobs at any time • ~700 CPU-years since the May 2008 • ~18 TB of data

37. Routinely LQCD production • 700 CPU years since May 2008 • ~18 TB of data transferred • ~800 simultaneous workers

38. Summary • User-level overlay is a technique enhancing the QoS parameters for scientific applications in the EGEE Grid • Pros & cons: • Existing infrastructure may be used “as is” • Application-specific optimizations (impossible at the system level)‏ • Hard QoS not possible (infrastructure unreliable)‏ • Faire-share implemented by the underlying infrastructure and respected by the overlay (if used appropriately)‏ • Used successful for diverse applications • Overlays are a complementary approach to SLAs • More on tools: http://cern.ch/diane http://cern.ch/ganga