Building, Monitoring and Maintaining a Grid

1. Building, Monitoring and Maintaining a Grid Jorge Luis Rodriguez University of Florida jorge@phys.ufl.edu July 11-15, 2005

2. Introduction • What we’ve already learned • What are grids, why we want them and who is using them: GSW intro & L1… • Grid Authentication and Authorization: L2 • Harnessing CPU cycles with condor: L3 • Data Management and the Grid: L4 • In this lecture • Fabric level infrastructure: Grid building blocks • The Open Science Grid

3. Grid Building Blocks • Computational Clusters • Storage Devices • Networks • Grid Resources and Layout: • User Interfaces • Computing Elements • Storage Elements • Monitoring Infrastructure…

4. Computer Clusters Cluster Management “frontend” A few Headnodes, gatekeepers and other service nodes I/O Servers typically RAID fileserver The bulk are Worker Nodes Disk Arrays Dell Cluster at UFlorida’s High Performance Center Tape Backup robots

5. Network Switch A Typical Cluster Installation WAN • Computing Cycles • Data Storage • Connectivity Head Node/Frontend Server Pentium III I/O Node + Storage • Cluster Management • OS Deployment • Configuration • Many options • ROCKS (kickstart) • OSCAR (sys imager) • Sysconfig Worker Nodes Pentium III Pentium III Pentium III

6. Internal Networks (LAN) Private, accessible only to servers inside a facility Some sites allow outbound connectivity via Network Address Translation Typical technologies used Ethernet (0.1, 1 & 10 Gbps) HP, Low Latency interconnects Myrinet: 2, 10 Gps Infiniband: max at 120Gps External connectivity Connection to Wide Area Network Typically achieved via same switching fabric as internal interconnects WAN Head Node/Frontend Server Pentium III Worker Nodes Pentium III Network Switch Pentium III Pentium III Networking “one planet one network” Global Crossing I/O Node + Storage

7. The Wide Area Network Ever increasing network capacities are what make grid computing possible, if not inevitable The Global Lambda Integrated Facility for Research and Education (GLIF)

8. Batch scheduling systems Submit many jobs through a head node #!/bin/sh for each i in $list_o_jobscripts do /usr/local/bin/condor_submit $i done Execution done on worker nodes Many different batch systems are deployed on the grid condor (highlighted in lecture 3) pbs, lsf, sge… WAN Head Node/Frontend Server Pentium III Worker Nodes Pentium III Network Switch Pentium III Pentium III Computation on a Clusters I/O Node + Storage • Primary means of controlling CPU usage, enforcing allocation policies and scheduling of jobs on the local computing infrastructure

9. Many hardware technologies deployed from: Single fileserver Linux box with lots of disk: RAID 5… Typically used for work space and temporary space a.k.a. “tactical storage” to Large Scale Mass Storage Systems Large peta-scale disk + tape robots systems Ex: FNAL’s Enstore MSS dCache disk frontend Powderhorn tape backend Typically used as permanent stores a.k.a “strategic storage” Storage Devices StorageTek Powderhorn Tape Silo

10. Typical Hardware Components Servers: Linux, RAID controllers… Disk Array IDE, SCSI, FC RAID levels 5, 0, 50, 1… Local Access Volumes mounted across compute cluster nfs, gpfs, afs… Volume Virtualization dCache pnfs Remote Access gridftp: globus-url-copy SRM interface space reservation request scheduling WAN Head Node/Frontend Server Pentium III Worker Nodes Pentium III Network Switch Pentium III Pentium III Tactical Storage /share/DATA = nfs:/tmp1 /share/TMP = nfs:/tmp2 /tmp1 /tmp2 /share/DATA = nfs:/tmp1 /share/TMP = nfs:/tmp2

11. Layout of Typical Grid Site The Gr i d A Grid Site Computing Fabric User Interface Compute Element Authz server Monitoring Element => + Storage Element + => Grid Middleware Grid Level Services VDT OSG Data Management Services Grid Operations Monitoring Clients Services

12. Grid3 and Open Science Grid

13. The Grid3 Project

14. A total of 35 sites Over 3500 CPUs Operations Center @ iGOC Began operations Oct. 2003 The Grid3 grid

15. Grid3 Metrics CPU usage averaged over a day 03/04-9/04 • CPU usage per VO 03/04 thru 09/04 • Data challenges for ATLAS and CMS CMS’ DC04 ATLAS’ DC2 Simulation Events • Grid3 contribution to MC production for CMS • USMOP = USCMS S&C + Grid3 41.4 x 106 evts, 11/03-3/05

16. The Open Science Grid A consortium of Universities and National Laboratories to building a sustainable grid infrastructure for Science in the U.S. …

17. Grid3  Open Science Grid • Begin with iterative extension to Grid3 • Shared resources, benefiting broad set of disciplines • Realization of the critical need for operations • More formal organization needed because of scale • Build OSG from laboratories, universities, campus grids • Argonne, Fermilab, SLAC, Brookhaven, Berkeley Lab, Jeff. Lab • UW Madison, U Florida, Purdue, Chicago, Caltech, Harvard, etc. • Further develop OSG • Partnerships and contributions from other sciences, universities • Incorporation of advanced networking • Focus on general services, operations, end-to-end performance

18. activity 1 activity 1 activity 1 Activities OSG Organization Technical Groups Advisory Committee Universities,Labs Service Providers Executive Board (8-15 representatives Chair, Officers) Sites Researchers VOs Research Grid Projects OSG Council (Chair, officers from major stakeholders, PIs, Faculty & Lab managers) Enterprise Core OSG Staff (few FTEs, manager)

19. OSG Activities and Tech. Groups TG-Monitoring & Information A Sampling of current OSG TG and Activities Docs TG-Policy TG-Storage OSG Integration Policy MIS Monitoring & Information Systems Privilege OSG deployment IPB Provisioning DRM dCache Operations TG-Support Centers

20. OSG Technical Groups & Activities • Technical Groups address and coordinate • Propose and carry out activities related to their given areas • Liaise & collaborate with other peer projects (U.S. & international) • Participate in relevant standards organizations. • Chairs participate in Blueprint, Grid Integration and Deployment activities • Activities are well-defined, scoped set of tasks contributing to theOSG • Each Activity has deliverables and a plan • … is self-organized and operated • … is overseen & sponsored by one or more Technical Groups

21. Privilege OSG Authentication & Authorization“Authz”

22. Authentication & Authorization • Authentication: Verify that you are who you say you are • OSG users typically use the DOEGrids CA • OSG sites also accept CAs from LCG and other organizations • Authorization: Allow a particular user to use a particular resource • Legacy Grid3 method, gridmap-files • New Privilege method, employed at primarily at US-LHC sites

23. gridmap-file gridmap-file gridmap-file OSG Authentication: Grid3 Style DN mappings edg-mkgridmap user DNs site a client VOMS server @ iGOC iVDGL, GADU… site b client user DNs mapping of user’s grid credentials (DN) to local site group account VOMS server @ LLab LColab, Lexp1 user DNs VOMS server @ OLab Oexp1, Aexp2… site n client VOMS= Virtual Organization Management System DN=Distinguished Name edg= European Data Grid (EU grid project)

24. The Privilege Project Application of a Role Based Access Control model for OSG An advanced authorization mechanism

25. The Privilege Project Provides • Centralized, dynamic mapping of grid users to local OS qualifiers • VOMSes are still used as grid identities DN db • Gone, however are the static grid-mapfiles • Improvement to Access Control policy implementation • The GUMS service • Access rights granted based on user’s • VO membership • User selected role(s) Grid Identity Unix ID Certificate DN <-------> local user VO Role(s) <------> local group(s)

26. 5. HTTPS/SOAP Response: 4. HTTPS/SOAP Request: SAML Query: SAML Statement: Decision=Permit, with obligation local UID=xyz, GID=xyz May user “Markus Lorch” of “VO=USCMS / Role=prod” access this resource? Privilege Project Components User Management (VOMSRS) Server with VDT1.3 based on gt3.2 Client server (UI) A VO service 1. VOMS-proxy-init request with specified role Client tool for role selection: VOMS-proxy-init VOMS Server VOMS Attribute Repository 2. Retrieves VO membership and role attribute 3. Standard globus-job-runrequest with VOMS-extended proxy VO membership synchronization Servers with VDT 1.3 based on gt3.2 Web-Service Container Server with VDT 1.3 based on GT3.2 Server with VDT 1.3 based on GT3.2 GUMS Identity Mapping Service (manages user accounts on resources, incl. dynamic allocation) gridFTP & Gate-keeper Gridmap callout PRIMA module 6. instantiates job-manager An OSG site

27. Authorization in OSG OSG will support multiple modes of operation • Will work with legacy client and/or server combinations • Full legacy server: VOMS+edg-makegridmap • Privilege enabled client requests (VOMS-proxy-init) will be mapped to local user as previously: VOMS extensions ignored • Full Privilege server: All Privilege components enabled • Legacy client requests are supported but user can not be mapped to a different VO or assume a different role under its own VO • Also a Grid3 compatibility mode will be supported • Supports privilege operations but without the globus PRIMA callout and thus can not support the “role” based mapping • The gatekeeper/gridFTP server is operated with legacy Grid3 stack (gtk 2.4 servers…) • Automatically provides reverse maps for Grid3/OSG accounting

28. A. Periodically queries GUMS server B. Response create static grid-mapfile May user “Markus Lorch” of “VO=USCMS / Role=prod” access this resource? The role however, is ignored DN mapping to a local UID=xxx, but no role based assignment Grid3 Compatibility mode User Management (VOMSRS) Server with VDT1.3 based on gt3.2 Client server (UI) A VO service 1. VOMS-proxy-init request Client tool for role selection: VOMS-proxy-init VOMS Server VOMS Attribute Repository 2. Retrieves VO membership and role attribute 3. Standard globus-job-runrequest with VOMS-extended proxy VO membership synchronization Legacy Grid3 Servers Web-Service Container Server with VDT 1.3 based on GT3.2 Server with VDT 1.3 based on GT3.2 to OSG accounting cron job GUMS Identity Mapping Service (manages user accounts on resources, incl. dynamic allocation) gridFTP & Gate-keeper reverse map grid-mapfile 4. instantiates job-manager An OSG site with legacy gatekeeper or gridFTP server

29. MIS Monitoring & Information Systems OSG Grid Monitoring

30. Grid Level Clients User interfaces & APIs Graphical Displays Gridcat MonALISA ACDC job Monitor Metrics DataViewer BDII Site Level infrastructure Scripts & tools collectors Databases Application APIs MonALISA server MonALISA client Metrics DataViewer OSG MIS-CI ACDC job Monitor GidCat MDS: Generic Info. Provider BDII OSG Monitoring & Information System

31. Monitoring Information Consumer API GINI, SOAP, WDSL… https: Web Services GRAM: jobman-mis Site Level Infrastructure • MonALISA server • MIS-Core Infrastructure • MDS GridCat stor_stat ACDC Ganglia MonALISA GIP Collector Discovery Service job_state Monitoring information DataBase Historical information DataBase others… …

32. OSG Grid Level Clients • Tools provide basic information about OSG resources • Resource catalog: official tally of OSG sites • Resource discovery: what services are available, where are they and how do I access it • Metrics Information: Usage of resources over time • Used to asses scheduling priorities • Where and when should I send my jobs? • Where can I put my output? • Used to monitor health and status of the Grid

33. http://osg-cat.grid.iu.edu http://www.ivdgl.org/grid3/gridcat GridCat

34. MonALISA

35. Provisioning OSG Provisioning OSG Software Cache OSG Meta Packager Grid Level Services

36. The OSG Software Cache • Most software comes from the VDT • OSG components include • VDT configuration scripts • Some OSG specific packages too • Pacman is the OSG Meta-packager • This is how we deliver the entire cache to resources

37. What is The VDT ? • A collection of software • Grid software • Virtual data software • Utilities • An easy installation mechanism • Goal: Push a button, everything just works • Two methods: • Pacman: installs and configures it all • RPM: installs some of the software, but no configuration • A support infrastructure • Coordinate bug fixing • Help desk

38. What is in the VDT? Core software User Interface Computing Element Storage Element Authz System Monitoring System Condor Group Condor/Condor-G DAGMan Fault Tolerant Shell ClassAds NeST Globus Alliance (3.2 pre web) Job submission (GRAM) Information service (MDS) Data transfer (GridFTP) Replica Location (RLS) EDG & LCG Make Gridmap Cert. Revocation list updater Glue & Gen. Info. provider VOMS Condor Group Condor/Condor-G DAGMan Fault Tolerant Shell ClassAds NeST Globus Alliance (3.2 pre web) Job submission (GRAM) Information service (MDS) Data transfer (GridFTP) Replica Location (RLS) EDG & LCG Make Gridmap Cert. Revocation list updater Glue & Gen. Info. provider VOMS ISI & UC Chimera & Pegasus NCSA MyProxy GSI OpenSSH UberFTP LBL PyGlobus Netlogger DRM Caltech MonALISA jClarens (WSR) VDT VDT System Profiler Configuration software ISI & UC Chimera & Pegasus NCSA MyProxy GSI OpenSSH UberFTP LBL PyGlobus Netlogger DRM Caltech MonALISA jClarens (WSR) VDT VDT System Profiler Configuration software US LHC GUMS PRIMA Others KX509 (U. Mich.) Java SDK (Sun) Apache HTTP/Tomcat MySQL Optional packages Globus-Core {build} Globus job-manager(s) US LHC GUMS PRIMA Others KX509 (U. Mich.) Java SDK (Sun) Apache HTTP/Tomcat MySQL Optional packages Globus-Core {build} Globus job-manager(s)

39. LIGO UCHEP VDT D-Zero ATLAS iVDGL CMS/DPE NPACI Pacman • Pacman is: • A software environment installer (or Meta-Packager) • A a language for defining software environments • An interpreter that allowing creation, installation, configuration, update, verification and repair such environments • Pacman makes installation of all types of software easy Globus/GPT Nordugrid/RPM LCG/Scram Enables us to easily and coherently combine and manage software from arbitrary sources. ATLAS/CMT NPACI/TeraGrid/tar/make Commercial/tar/make D0/UPS-UPD LIGO/tar/make OpenSource/tar/make CMS DPE/tar/make % pacman –get iVDGL:Grid3 Enables remote experts to define installation config updating for everyone at once.

40. Pacman Installation • Download Pacman • http://physics.bu.edu/~youssef/pacman/ • Install the “package” • cd <install-directory> • pacman -get OSG:OSG_CE_0.2.1 • ls condor/ globus/ post-install/ setup.sh edg/ gpt/ replica/ vdt/ ftsh/ perl/ setup.csh vdt-install.log /monalisa ...

41. Grid Level Services • OSG Grid level Monitoring infrastructure • Mon. & Info. System(s) top level database • Monitoring Authz: mis user • OSG Operations infrastructure • Websites, Ops page, web servers … • Trouble ticket system • OSG top level Replica Location Service

42. Operations OSG Operations

43. Monitoring Grid status Use of Grid monitors and verification routines Report, route and track problems and resolution Trouble ticket system Repository of resource contact information Do this as part of a National distributed system Grid Operations Monitoring and Maintaining the Health of the Grid User support Application support VO issues

44. Operations Model in OSG

45. Ticket Routing in OSG User in VO1 notices problem at RP3, notifies their SC (1). SC-C opens ticket (2) and assigns to SC-F. SC-F gets automatic notice (3) and contacts RP3 (4). Admin at RP3 fixes and replies to SC-F (5). SC-F notes resolution in ticket (6). SC-C gets automatic notice of update to ticket (7). SC-C notifies user of resolution (8). User confirms resolution (9). SC-C closes ticket (10). SC-F gets automatic notice of closure (11). SC-F notifies RP3 of closure (12). 11 2 1 3 7 8 6 10 4 9 5 12 OSG infrastructure SC private infrastructure

46. Integration OSG Integration and Development

47. Path for New Services in OSG OSG Integration Activity Readiness plan Effort Resources Readiness plan adopted VO Application Software Installation Software & packaging OSG Deployment Activity Service deployment OSG Operations-Provisioning Activity Release Candidate Application validation Middleware Interoperability Functionality & Scalability Tests feedback Metrics & Certification Release Description

48. OSG Integration Testbed Layout OSG Integration Testbed VO contributed Service platform Stable Production Release Integration Release Resources enter and leave as necessary applications, test harness, clients

49. OSG Integration Testbed

50. OSG Further Work Managed Storage Grid Scheduling More