Low Level Grid Services (Job Management, Data Management, Monitoring Services)

1. Low Level Grid Services (Job Management, Data Management, Monitoring Services) Ravi K Madduri Argonne National Laboratory University of Chicago

2. Services Overview • Installation • Data Management • GridFTP, RFT, RLS, DAIS • Resource Management • Schedulers, logs, sudo • Information Services • Index service hierarchies, ganglia/hawkeye

3. Full Toolkit Installation • Binaries available for many platforms • Apple • Linux • Debian, Fedora, SuSe, RHEL, Redhat • FreeBSD • HP/UX, Tru64 • AIX • Solaris • Windows (Java code only) • Source code also available • See http://www.globus.org/toolkit/docs/4.0 for installation guide, quickstart, and pre-req documentation

4. Overview of GT4 Data Services • GridFTP • High-performance Data transfer protocol • The Reliable File Transfer Service (RFT) • Data movement services for GT4 • The Replica Location Service (RLS) • Distributed registry that records locations of data copies • The Data Access and Integration Service (DAIS) • Service to access relational and XML databases

5. What is GridFTP? • A secure, robust, fast, efficient, standards based, widely accepted data transfer protocol • A Protocol • Multiple Independent implementation can interoperate • This works. Both the Condor Project at Uwis and Fermi Lab have home grown servers that work with ours. • Lots of people have developed clients independent of the Globus Project. • The Globus Toolkit supplies a reference implementation: • Server • Client tools (globus-url-copy) • Development Libraries

6. Based on XIO Extremely modular to allow integration with a variety of data sources (files, mass stores, etc.) Striping support is provided in 4.0 Has IPV6 support included (EPRT, EPSV) GT4 GridFTP Implementation

7. Configuring GridFTP • Right configuration results in better performance • Add entries to /etc/services and (x)inetd • Configuration options: • Binding to a specific interface/address • Striped backend • TCP tuning parameters

8. What is RFT ? • WS-RF compliant Fault-tolerant, High- performance data transfer service • Soft state. • Notifications/Query • Reliability on top of high performance provided by GridFTP. • Fire and Forget. • Integrated Automatic Failure Recovery. • Network level failures. • System level failures etc. • Essentially a Data transfer scheduler with FIFO as a Queue Policy.

9. IPCReceiver DataChannel DataChannel MasterDSI SlaveDSI Protocol Interpreter SlaveDSI Protocol Interpreter Data Channel MasterDSI IPCReceiver Data Channel IPC Link IPC Link What is RFT (Continued..)? RFT Client SOAP Messages Notifications(Optional) RFT Service GridFTP Server GridFTP Server

10. Control Control Control Control Data Data Data Data Data Transfer Comparison RFT Client SOAP Messages Notifications(Optional) globus-url-copy RFT Service

11. Replica Management in Grids • Data intensive applications produce terabytes or petabytes of data • Hundreds of millions of data objects • Replicate data at multiple locations for reasons of: • Fault tolerance • Avoid single points of failure • Performance • Avoid wide area data transfer latencies • Achieve load balancing

12. A Replica Location Service • A Replica Location Service (RLS) is a distributed registry that records the locations of data copies and allows replica discovery • RLS maintains mappings between logical identifiers and target names • Must perform and scale well: support hundreds of millions of objects, hundreds of clients • E.g., LIGO (Laser Interferometer Gravitational Wave Observatory) Project • RLS servers at 8 sites • Maintain associations between 3 million logical file names & 30 million physical file locations • RLS is one component of a Replica Management system • Other components include consistency services, replica selection services, reliable data transfer, etc.

13. Goals for OGSA-DAI • Aim to deliver application mechanisms that: • Meet the data requirements of Grid applications • Functionality, performance and reliability • Reduce development cost of data centric Grid applications • Provide consistent interfaces to data resources • Acceptable and supportable by database providers • Trustable, imposed demand is acceptable, etc. • Provide a standard framework that satisfies standard requirements • A base for developing higher-level services • Data federation • Distributed query processing • Data mining • Data visualisation

14. Data Management Q & A

15. Grid Monitoring Services • Overview • Index Service • Aggregate the data • Trigger Service • Notify when data changes • Information Providers • Provide the data • WebMDS • Client to visualize data

16. What Is Grid Monitoring? • A way to discover what services and resources are available to use (Discovery) • A way to understand the status/attributes of those services (Monitoring) • A system to warn you when things fail • Sharing of community data between sites using a standard interface for querying and notification.

17. Why Grid Monitoring Hard? • Lack of central control • Different local systems according to local policy • Different interfaces and monitoring requirements • Shared resources • Contention, variability • Communication • Different sites implies different sys admins, users, institutional policies

18. MDS4Monitoring and Discovery System • Grid-level monitoring system used most often for resource selection • Aid user/agent to identify host(s) on which to run an application • Uses standard interfaces to provide publishing of data, discovery, and data access, including subscription/notification • WS-ResourceProperties, WS-BaseNotification, WS-ServiceGroup • Functions as an hourglass to provide a common interface to lower-level monitoring tools

19. Cluster monitors (Ganglia, Hawkeye, Clumon, and Nagios soon) Queueing systems (PBS, LSF, Torque) Services (GRAM, RFT, RLS) Information Users : Schedulers, Portals, etc. WS standard interfaces for subscription, registration, notification GLUE Schema Attributes (cluster info, queue info, FS info)

20. MDS4 Components • Higher level services • Index Service – a way to aggregate data • Trigger Service – a way to be notified of changes • Both built on common aggregator framework • Information providers • Monitoring is a part of every WSRF service • Non-WS services can also be used • Clients • WebMDS • All of the tool are schema-agnostic, but interoperability needs a well-understood common language

21. Sample Deployment

22. WebMDS User Interface • Web-based interface to WSRF resource property information • User-friendly front-end to the Index Service • Uses standard resource property requests to query resource property data • XSLT transforms to format and display them • Customized pages are simply done by using HTML form options and creating your own XSLT transforms • Sample page: • http://mds.globus.org:8080/webmds/webmds?info=indexinfo&xsl=servicegroupxsl

23. WebMDS Service

26. Information Services Q & A

27. GRAM • Overview • Submitting a test job • Resource Specification Language (RSL) • Data Staging • Multi-jobs

28. GRAM Overview • Intended for jobs where arbitrary programs, state-ful monitoring, credential management, and file staging are important • If the application is lightweight, with modest input/output, may be a better candidate for hosting directly as a WSRF service

29. GRAM Prerequisites • A secure container • For staging jobs, access to an RFT service and a GridFTP server • Note that even stderr/stdout are considered staging, so RFT and GridFTP are used in all but the most basic jobs • sudo for running as other accounts • Can be integrated with PBS, LSF, Condor

30. Submitting A Test Job • globusrun-ws –submit –c /bin/true • echo $? • Will run locally. Specify a remote host with –F • globusrun-ws –submit –F host2 –c /bin/true • The return code will be the job’s exit code if supported by the scheduler

31. Data Staging • GRAM allows jobs to stage-in and stage-out data • To perform this task it uses RFT • RFT in turn uses GridFTP servers • Simplest stage-in/stage-out example is stdout/stderr

32. Streaming Results • globusrun-ws –S –s –c /bin/date • -S is short for “-submit” • -s is short for –streaming • The output will be sent back to the terminal, control will not return until the job is done

33. Resource Specification Language • For more complicated jobs, we’ll use RSL to specify the job <job> <executable>/bin/echo</executable> <argument>this is an example_string </argument> <argument>Globus was here</argument> <stdout>${GLOBUS_USER_HOME}/stdout</stdout> <stderr>${GLOBUS_USER_HOME}/stderr</stderr> </job>

34. Submitting Using XML • Create the file containing the RSL • You may validate the RSL ahead of time • globusrun-ws –validate –f rslfile.xml • If the file validates, submit using -submit

35. At Most Once Submission • You may specify a UUID with your job submission • If you’re not sure the submission worked, you may submit the job again with the same UUID • If the job has already been submitted, the new submission will have no effect • If you do not specify a UUID, one will be generated for you

36. Staging Data • GRAM’s RSL allows many fileStageIn/fileStageOut directives • The transfers will be executed by RFT • May specify additional RFT options using the RFTOptions tag • There is no GASS cache staging option anymore

37. Batch Submission • Your client does not have to stay attached to the execution of the job • -batch will disconnect from the job and output an EPR • You may redirect the EPR to a file with –o • Use the EPR file with –monitor or -status • You may also kill the job using -kill

38. Specifying Scheduler Options • RSL lets you specify various scheduler options • what queue to submit to • which project to select for accounting • max CPU and wallclock time to spend • min/max memory required • All defined online under the schema document for GRAM

39. Multijobs • You may specify more than one <job> element in a <multijob> • At that point, you want to specify the <factoryEndpoint> in the RSL rather than the commandline • Will be used by MPICH-G to support MPI jobs

40. Resource Management Q & A

41. For more information • The Globus Toolkit ™ • http://www-unix.globus.org/toolkit/