1 / 17

European DataGrid

European DataGrid. Charles Loomis Seillac June 26, 2002. European DataGrid (EDG). European DataGrid EU-funded, 3-year project (2001-3) Goals: develop grid middleware deploy onto working testbed demonstrate grid technology with working applications Strong application component unique!.

haamid
Download Presentation

European DataGrid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. European DataGrid Charles Loomis Seillac June 26, 2002

  2. European DataGrid (EDG) European DataGrid • EU-funded, 3-year project (2001-3) • Goals: • develop grid middleware • deploy onto working testbed • demonstrate grid technology with working applications • Strong application component unique! 6 Partners; 21 Associates

  3. Computing Grid  Electrical Grid Accessing computing resources as simple as plugging in a toaster! • Fixed resources (CPU) • Mobile resource (Data) • Semi-mobile (Tape) Transparency (I.e. standards) • Simplifies sharing & trading of resources. • Does not imply that all resources are free. • Should be applicable from desktop to supercomputers. Security (Owners retain control!) • System administrators control machine access. • Who can read your data?  Software enforces access control.

  4. Transparency for Physicists Single Login: • Access all available local & remote resources w/o a dedicated account. Consistent User Environment: • Same working environment everywhere--same software tools; same configuration. Brokering: • System finds “optimal” resources for task automatically. • Resource discovery, moving job to data, moving data to job. High-Level Selection of Resources: • Ex. choose storage by size/lifetime/reliability/cost/transfer speed; not disk/tape/shared file system/location. Security: • Does not impede authorized access to resources. • Data and computations are (reasonably) safe from tampering.

  5. Interoperability & Interdependency Standards: • Global Grid Forum (like W3C) • Or from dominant implementation Plethora of Projects: • Sharing of code & expertise. • Competing for people, funding, etc.

  6. Information Systems User Interface Resource Broker Computing Element Storage Element Site X EDG Architecture Global Batch System: • Centralized Architecture. • Heavy infrastructure. • Other choices  Peer-to-peer query submit retrieve publish state broker chooses optimal site for job submit retrieve

  7. Certification Authorities Attest to your identity. Virtual Organizations Maintain membership lists. Used for group authorization. Replica Catalog Location & Directory Info. for data. Hidden Infrastructure Maintained by: ALICE ATLAS BaBar CMS LHCb Biomedical Apps. Earth Observation /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/Email=loomis@lal.in2p3.fr

  8. Production Testbed (1.1.4) For applications to use & stress software in “semi-production” environment. 10 sites (6 countries) Development Testbed (1.2b5) To facilitate testing and integration of new middleware. 4 sites (4 countries) EDG Testbeds

  9. Example Find data file and count events. #!/bin/sh # Get matched physical file name. filename = `./GetPFN.pl $EDG_WL_RB_BROKERINFO` # Analyze this file. count_evts$filename

  10. Example Describe the requirements for the job. # Executable to run and the arguments.Executable    = "ChecksumRC.sh";Arguments     = "none";# Define file names for standard output and error.StdOutput     = "std.out";StdError      = "std.err";# Files to transfer as input and output.InputSandbox  = {"ChecksumRC.sh","GetPFN.pl”,”count_evts”};OutputSandbox = {"std.out","std.err"};# Desired file and protocol.InputData = {"LF:utve_v_1_71008101_12902_0609.lv0"};DataAccessProtocol = {"file"};# My experiment's "Data Catalog"ReplicaCatalog = "ldap://grid-vo.nikhef.nl:10389/rc=EarthObReplicaCatalog,dc=eu-datagrid,dc=org";

  11. Example Create a proxy for authorization  like logging in. > grid-proxy-initYour identity: /C=FR/O=CNRS/OU=LAL/CN=CharlesLoomis/Email=loomis@lal.in2p3.fr Enter GRID pass phrase for this identity:************* Creating proxy .......................................... Done Your proxy is valid until Sun Jun 23 06:13:51 2002

  12. Example Submit job. Broker will automatically find appropriate place. > dg-job-submit ChecksumRC.jdlConnecting to host testbed011.cern.ch, port 7771Logging to host testbed011.cern.ch, port 15830************************************************************************************                                     JOB SUBMIT OUTCOME  The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is:https://testbed011.cern.ch:7846/137.138.181.253/162422289427581?testbed011.cern.ch:7771************************************************************************************

  13. Example Get the status of the submitted job. >dg-job-status <job id>… dg_JobId                =    https://testbed011.cern.ch:7846/137.138.181.253/… Status              =    OutputReady Last Update Time (UTC)  =    Sat Jun 22 16:29:30 2002 Job Destination         =    gppce05.gridpp.rl.ac.uk:2119/jobmanager-pbs-S Status Reason           =    terminated Job Owner               =    /C=FR/O=CNRS/OU=LAL/CN=Charles Loomis/… Status Enter Time (UTC) =    Sat Jun 22 16:29:30 2002

  14. Example Pull the output back to home machine. >dg-job-get-output <job id>…  Output sandbox files for the job: - https://testbed011.cern.ch:7846/137.138.181.253/… have been successfully retrieved and stored in the directory:/tmp/162422289427581 >cat /tmp/162422289427581/std.outFile: /flatfiles/eo/utve_v_1_71008101_12902_0609.lv0Contains: 1034 events

  15. Future Developments EDG Plans: • more advanced data management (I.e. real “Storage Element”) • job splitting, checkpointing; interactive jobs • more robust, consistent security model OGSA (Open Grid Services Architecture) • Proposal by Globus/IBM based on web-services • Adds concepts like notification and soft-state registration. • Standardizes the protocol and interface descriptions for Grid services. Cross-fertilization of Projects • LAL is ideally situated for this. (EDG, LCG, XtremWeb, CGP2P)

  16. Conclusions EDG has interesting (to a physicist) software & working testbed: • Current release: • Basic features available. • Somewhat painful: reliability, low-level mucking around • Fall release: • Expect big change in functionality, especially data management. • Concerted effort on reliability should make testbed less frustrating. • About the time to get involved… Long-term success: • When you work routinely in a “Grid” environment on your desktop. • Fully automatic: resource discovery, optimization, and task dispatch. • Not there yet, but going in right direction.

  17. Getting Involved & Further Surfing Talk to grid contact: • ATLAS (Laura Perini) • ALICE (Piergiorgio Cerello) • CMS (Andrea Sciaba) • LHCb (Eric van Herwijnin) • others (see me) Register: • “User Registration” @ WP6 website Warnings: • Dev. project  Frequent changes. • No guarantees on availability. Schedule: • EDG 1.2 (shortly) • EDG 2.0 (fall 2002)

More Related