eScience – Grid Computing Graduate Lecture

1. eScience – Grid ComputingGraduate Lecture 9th November 2011 Robin Middleton – PPD/RAL/STFC I am indebted to the EGEE, EGI, LCG and GridPP projects and to colleagues therein for much of the material presented here.

2. eScience Graduate Lecture A high-level look at some aspects of computing for particle physics today • What is eScience, what is the Grid ? • Essential grid components • Grids in HEP • The wider picture • Summary

3. What is eScience ? • …also : e-Infrastructure, cyberinfrastructure, e-Research, … • Includes • grid computing (e.g. WLCG, EGEE, EGI, OSG, TeraGrid, NGS…) • computationally and/or data intensive; highly distributed over wide area • digital curation • digital libraries • collaborative tools (e.g. Access Grid) • …other areas • Most UK Research Councils active in e-Science • BBSRC • NERC (e.g. climate studies, NERC DataGrid - http://ndg.nerc.ac.uk/ ) • ESRC (e.g. NCeSS - http://www.merc.ac.uk/ ) • AHRC (e.g. studies in collaborative performing arts) • EPSRC (e.g. MyGrid - http://www.mygrid.org.uk/ ) • STFC (e.g. GridPP - http://www.gridpp.ac.uk/ )

4. eScience – year ~2000 • Professor Sir John Taylor, former (1999-2003) Director General of the UK Research Councils, defined eScience thus: • science increasingly done through distributed global collaborations enabled by the internet, using very large data collections, terascale computing resources and high performance visualisation’. • Also quotes from Professor Taylor… • ‘e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.’ • ‘e-Science will change the dynamic of the way science is undertaken.’

5. What is Grid Computing ? • Grid Computing • term invented in 1990s as metaphor for making computer power as easy to access as the electric power grid(Foster & Kesselman - "The Grid: Blueprint for a new computing infrastructure“) • combines computing resources from multiple administrative domains • CPU and storage…loosely coupled • serves the needs of one or more virtual organisations (e.g. LHC experiments) • different from • Cloud Computing (e.g. Amazon Elastic Compute Cloud - http://aws.amazon.com/ec2/ ) • Volunteer Computing (SETI@home, LHC@home - http://boinc.berkeley.edu/projects.php )

6. Essential Grid Components • Middleware • Information System • Workload Management; Portals • Data Management • File transfer • File catalogue • Security • Virtual Organisations • Authentication • Authorisation • Accounting

7. Information System • At the heart of the Grid • Hierarchy of BDII (LDAP) servers • GLUE information schema • (http://www.ogf.org/documents/GFD.147.pdf) • LDAP (Lightweight Directory Access Protocol) • tree structure • DN: Distinguished Name o = grid (root of the DIT) c= US c=UK c=Spain st = Chilton or = STFC ou = PPD ou = ESC

8. Workload Management System (WMS) • For example - composed of the following parts: • User Interface (UI) : access point for the user to the WMS • Resource Broker (RB) : the broker of GRID resources, responsible to find the “best” resources where to submit jobs • Job Submission Service (JSS) : provides a reliable submission system • Information Index (BDII) : a server (based on LDAP) which collects information about Grid resources – used by the Resource Broker to rank and select resources • Logging and Bookkeeping services (LB) : store Job Info available for users to query • However, you are much more likely to use a portal to submit work… • Executable = “gridTest”; • StdError = “stderr.log”; • StdOutput = “stdout.log”; • InputSandbox = {“/home/robin/test/gridTest”}; • OutputSandbox = {“stderr.log”, “stdout.log”}; • InputData = “lfn:testbed0-00019”; • DataAccessProtocol = “gridftp”; • Requirements = other.Architecture==“INTEL” && \ other.OpSys==“LINUX” && other.FreeCpus >=4; • Rank = “other.GlueHostBenchmarkSF00”; Example JDL

9. Job Definition & Management • Implemented in Python • Extensible – plug-ins • Used ATLAS, LHCb & non-HEP • http://ganga.web.cern.ch/ganga/index.php Portals - Ganga

10. Data Management • Storage Element (SE) • >1 implementation • all are accessed through SRM (Storage Resource Manager) interface • DPM – Disk Pool Manager (disk only) • secure: authentication via GSI, authorisation via VOMS • full POSIX ACL support with DN (userid) and VOMS groups • disk pool management (direct socket interface) • storage name space (aka. storage file catalog) • DPM can act as a site local replica catalog • SRMv1, SRMv2.1 and SRMv2.2 • gridFTP, rfio • dCache (disk & tape) – developed at DESY • ENSTORE – developed at Fermilab • CASTOR – devloped at CERN • Cern Advanced STORage manager • HSM – Hierarchical Storage Manager • disk cache & tape

11. File Transfer Service • File Transfer Service is a data movement fabric service • multi-VO service, balance usage of site resources according to VO and site policies • uses SRM and gridFTP services of an Storage Element (SE) • Why is it needed ? • For the user, the service it provides is the reliable point to point movement of Storage URLs (SURLs) among Storage Elements • For the site manager, it provides a reliable and manageable way of serving file movement requests from their VOs • For the VO manager, it provides ability to control requests coming from users(re-ordering, prioritization,...)

12. File Catalogue • LFC – LHC File Catalogue - a file location service • Glossary • LFN = Logical File Name; GUID = Global Unique ID; SURL = Storage URL • Provides a mapping from one or more LFN to the physical location of file • Authentication & authorisation is via a grid certificate • Provides very limited metadata – size, checksum • Experiments usually have a metadata catalogue layered above LFC • e.g. AMI – ATLAS Metadata Interface

13. Grid Security • Based around X.509 certificates – Public Key Infrastructure (PKI) • issued by Certificate Authorities • forms a hierarchy of trust • Glossary • CA – Certificate Authority • RA – Registration Authority • VA – Validation Authority • How it Works… • User applies for certificate with public key at a RA • RA confirms user's identity to CA which in turn issues the certificate • User can then digitally sign a contract using the new certificate • User identity is checked by the contracting party with VA • VA receives information about issued certificates by CA

14. Virtual Organisations • Aggregation of groups (& individuals) sharing use of (distributed) resources to a common end under an agreed set of policies • a semi-informal structure orthogonal to normal institutional allegiances • e.g. A HEP Experiment • Grid Policies • Acceptable use; Grid Security; New VO registration; • http://proj-lcg-security.web.cern.ch/proj-lcg-security/security_policy.html • VO specific environment • experiment libraries, databases,… • resource sites declare which VOsit will support

15. Security - The Three As • Authentication • verifying that you are who you say you are • your Grid Certificate is your “passport” • Authorisation • knowing who you are, validating what you are permitted to do • e.g. submit analysis jobs as a member of LHCb • e.g. VO capability to manage production software • Accounting (auditing) • local logging what you have done – your jobs ! • aggregated into grid-wide respository • provides • usage statistics • information source in event of security incident

16. Grids in HEP • LCG; EGEE & EGI Projects • GridPP • The LHC Computing Grid • Tiers 0,1,2 • The LHC OPN • Experiment Computing Models • Typical data access patterns • Monitoring • Resource providers view • VO view • End-user view

17. LCGEGEE->EGI • LCG  LHC Computing Grid • Distributed Production Environment for Physics Data Processing • World’s largest production computing grid • In 2011 : >250,000 CPU cores, 15PB/Yr, 8000 physicist, ~500 institutes • EGEE  Enabling Grids for E-sciencE • Starts from LCG infrastructure • Production Grid in 27 countries • HEP, BioMed, CompChem, • Earth Science, … • EU Support

18. Integrated within the LCG/EGI framework UK Service Operations (LCG/EGI) Tier-1 & Tier-2s HEP Experiments @ LHC, FNAL, SLAC GANGA (LHCb & ATLAS) Working with NGS informing the UK NGI for EGI • Phase 1 : 2001-2004 • Prototype (Tier-1) • Phase 2 : 2004-2008 • “From Prototype to Production” • Production (Tier-1&2) • Phase 3 : 2008-2011 • “From Production to Exploitation” • Reconstruction, Monte Carlo, Analysis • Phase 4 : 2011-2014… • routine operation during LHC running GridPP Tier-1 Farm Usage

19. LCG – The LHC Computing Grid • Worldwide LHC Computing Grid - http://lcg.web.cern.ch/lcg/ • Framework to deliver distributed computing for theLHC experiments • Middleware / Deployment • (Service/Data Challenges) • Security (operations & policy) • Applications (Experiment) Software • Distributed Analysis • Private Optical Network • Experiments  Resources  MoUs • Coverage • Europe  EGI • USA  OSG • Asia  Naregi, Taipei,China… • Other…

20. Lab m Uni x regional group CERN Tier 1 Uni a UK USA Lab a France Tier 1 Tier3 (physics department) Uni n Tier2 ………. Italy Desktop Lab b Germany ………. Lab c  Uni y Uni b physics group   LHC Computing Model The LHC Computing Centre CERN Tier 0

21. LHCOPN – Optical Private Network • Principle means to distribute LHC data • Primarily linking Tier-0 and Tier-1s • Some Tier-1 to Tier-1 Traffic • Runs over leased lines • Some resilience • Mostly based on10 Gigabit technology • Reflects Tierarchitecture

22. LHC Experiment Computing Models • General (ignoring experiment specifics) • Tier-0 (@CERN) • 1st pass reconstruction (including initial calibration) • RAW data storage • Tier-1 • Re-processing; some centrally organised analysis; • Custodial copy of RAW data, some ESD, all AOD, some SIMU • Tier-2 • (chaotic) user analysis; simulation • some AOD (depends on local requirements) • Event sizes determine disk buffers at experiments & Tier-0 • Event datasets • formats (RAW, ESD, AOD, etc); • (adaptive) placement (near analysis); replicas • Data streams – physics specific, debug, diagnostic, express, calibration • CPU & storage requirements • Simulation

23. Typical Data Access Patterns Typical LHC particle physics experiment One year of acquisition and analysis of data Access Rates (aggregate, average) 100 Mbytes/s (2-5 physicists) 500 Mbytes/s (5-10 physicists) 1000 Mbytes/s (~50 physicists) 2000 Mbytes/s (~150 physicists) Raw Data ~1000 Tbytes Reco-V1 ~1000 Tbytes Reco-V2 ~1000 Tbytes ESD-V1.1 ~100 Tbytes ESD-V1.2 ~100 Tbytes ESD-V2.1 ~100 Tbytes ESD-V2.2 ~100 Tbytes AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB AOD ~10 TB

24. Monitoring • A resource provider’s view

25. Monitoring • Virtual Organisation specifics

26. Monitoring - Dashboards • Virtual Organisation view • e.g. ATLAS dashboard

27. Monitoring – Dashboards • For the end user • available through dashboard

28. The wider Picture • What some other communities do with Grids • The ESFRI projects • Virtual Instruments • Digital Curation • Clouds • Volunteer Computing • Virtualisation

29. What are other communities doing with grids ? • Astronomy & Astrophysics • large-scale data acquisition, simulation, data storage/retrieval • Computational Chemistry • use of software packages (incl. commercial) on EGEE • Earth Sciences • Seismology, Atmospheric modeling, Meteorology, Flood forecasting, Pollution • Fusion (build up to ITER) • Ion Kinetic Transport, Massive Ray Tracing, Stellarator Optimization. • Computer Science • collect data on Grid behaviour (Grid Observatory) • High Energy Physics • four LHC experiments, BaBar, D0, CDF, Lattice QCD, Geant4, SixTrack, … • Life Sciences • Medical Imaging, Bioinformatics, Drug discovery • WISDOM – drug discovery for neglected / emergent diseases(malaria, H5N1, …)

30. ESFRI Projects(European Strategy Forum on Research Infrastructures) • Many are starting to look at their e-Science needs • some at a similar scale to the LHC (petascale) • project design study stage • http://cordis.europa.eu/esfri/ Cherenkov Telescope Array

31. Virtual Instruments • Integration of scientific instruments into the Grid • remote operation, monitoring, scheduling, sharing… • GridCC - Grid enabled Remote Instrumentation with Distributed Control and Computation CR: build workflows to monitor & control remote instruments in real-time CE, SE, ES , IS & SS: as in a “normal” grid Monitoring services Instrument Element (IE) - interfaces for remote control & monitoring • CMS run control includes an IE…but notreally exploited (yet) ! • DORII – Deployment Of Remote Instrumentation Infrastructure • Consolidation of GridCC with EGEE, g-Eclipse,Open MPI, VLab • The Liverpool Telescope - robotic • not just remote control, but fully autonomous • scheduler operates on basis of observingdatabase • (http://telescope.livjm.ac.uk/)

32. Digital Curation • Preservation of digital research data for future use • Issues • media; data formats; metadata; data management tools; reading (FORTRAN); ... • digital curation lifecycle - http://www.dcc.ac.uk/digital-curation/what-digital-curation • Digital Curation Centre - http://www.dcc.ac.uk/ • NOT a repository ! • strategic leadership • influence national (international) policy • expert advice for both users and funders • maintains suite of resources and tools • raise levels of awareness and expertise

33. JADE (1978-86) • New results from old data • new & improved theoretical calculations & MC models; optimised observables • better understanding of Standard Model (top, W, Z) • re-do measurements – better precision, better systematics • new measurements, but at (lower) energies not available today • new phenomena – check at lower energies • Challenges • rescue data from (very) old media; resurrect old software; data management; implement modern analysis techniques • but, luminosity files lost – recovered from ASCII printout in an office cleanup • Since 1996 • ~10 publications (as recent as 2009) • ~10 conference contributions • a few PhD Theses (ack S.Bethke)

34. What is HEP doing about it ? • ICFA Study Group on Data Preservation and Long Term Analysis in High Energy Physicshttps://www.dphep.org/ • 5 Workshops so far; intermediate report to ICFA • Available at arxiv:0912.0255 • Initial recommendationsDecember 2009 • “Blueprint for DataPreservation in HighEnergy Physics” to follow

35. (Ack: Bob Jones – former EGEE Project Director) Grids, Clouds, Supercomputers, … • Grids • Collaborative environment • Distributed resources (political/sociological) • Commodity hardware (also supercomputers) • (HEP) data management • Complex interfaces (bug not feature) • Supercomputers • Expensive • Low latency interconnects • Applications peer reviewed • Parallel/coupled applications • Traditional interfaces (login) • Also SC grids (DEISA, Teragrid) • Clouds • Proprietary (implementation) • Economies of scale in management • Commodity hardware • Virtualisation for service provision and encapsulating application environment • Details of physical resources hidden • Simple interfaces (too simple?) • Volunteer computing • Simple mechanism to access millions CPUs • Difficult if (much) data involved • Control of environment  check • Community building – people involved in Science • Potential for huge amounts of real work Bob Jones - October 2009 35

36. Clouds / Volunteer Computing • Clouds are largely commercial • Pay for use • Interfaces from grids exist • absorb peak demands(e.g. before a conference !) • CernVM images exist • Volunteer Computing • LHC@Home • SixTrack – study particle orbitstability in accelerators • Garfield – study behaviour of gas-based detectors

37. Virtualisation • Virtual implementation of a resource – e.g. a hardware platform • a current buzzword, but not new – IBM launched VM/370 in 1972 ! • Hardware virtualisation • one or more virtual machines running an operating system within a host system • e.g. run Linux (guest) in a virtual machine (VM) with Microsoft Windows (host) • independent of hardware platform; migration between (different) platforms • run multiple instances on one box; provides isolation (e.g. against rogue s/w) • Hardware-assisted virtualisation • not all machine instructions are “virtualisable” (e.g. some privileged instructions) • h/w-assist traps such instructions and provides hardware emulation of them • Implementations • Zen, VMware, VirtualBox, Microsoft Virtual PC, … • Interest to HEP ? • the above + opportunity to tailor to experiment needs (e.g. libraries, environment) • CernVM – CERN specific Linux environment - http://cernvm.cern.ch/portal/ • CernVM-FS – network filesystem to access experiment specific software • Security – certificate to assure origin/validity of VM

38. Summary • What is eScience about and what are Grids • Essential components of a Grid • middleware • virtual organisations • Grids in HEP • LHC Computing GRID • A look outside HEP • examples of what others are doing