1 / 27

Grid Components

Grid Components. Dave Berry Research Manager National e-Science Centre (UK). EGEE is funded by the European Union under contract IST-2003-508833. Outline. Introduction Curation Grid components now Web Service Grids Architecture. Wearing many hats…. DCC. EGEE. OGSA-DAI. OGSA.

Leo
Download Presentation

Grid Components

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Components Dave Berry Research ManagerNational e-Science Centre (UK) EGEE is funded by the European Union under contract IST-2003-508833 DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 1

  2. Outline • Introduction • Curation • Grid components now • Web Service Grids • Architecture DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 3

  3. Wearing many hats… DCC EGEE OGSA-DAI OGSA DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 4

  4. Trusted Repositories of Knowledge The Maori entrusted their knowledge to people, trained to be the repositories, who could: • receive information with the utmost accuracy • store information with integrity beyond doubt • retrieve the information without amendment • apply appropriate judgement in the use of the information • pass on the information appropriately. Whatarangi Winiata, (2002), Repositories of Röpü Tuku Iho: A Contribution to the Survival of Mäori as a People, Wellington: Library & Information Association of New Zealand Aotearoa Annual Conference, 17-20 November 200 Special thanks to Professors Derek Law & Seamus Ross DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 5

  5. Digital Curation Centre communities of practice: users curation organisations eg DPC community support & outreach Collaborative Associates Network of Data Organisations service definition & delivery management & admin support research collaborators research development co-ordination testbeds& tools Industry standards bodies DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 6

  6. Curation is distribution over time • Migration & Refreshment • software & media • while interest (& funding) is active • Emulation & Encapsulation • re-creating the IT environment • Digital Archaeology & Rescue • urgent action to save key datasets • Representation Information • as well as bits • formats and more • e.g. OAIS (Open Archival Information Systems), Dublin Core, … DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 7

  7. Researchers perform their activities regardless geographical location, interact with colleagues, share and access data Scientific instruments and experiments provide huge amount of data Grid computing is distribution over space The Grid: networked data processing centres and ”middleware” software as the “glue” of resources. DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 8

  8. The essence of Grid computing “Grid computing is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations” (I.Foster) • A Virtual Organisation is: • People from different institutions working to solve a common goal • Sharing distributed processing and data resources • Grid infrastructure enables virtual organisations • Requirements include security, autonomy, scalability, heterogeneity, discovery, naming, reliability, usability, efficiency, policy, management, … • Standards, standards, standards DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 9

  9. The EGEE Consortium Total of 70 full partners covering entire EU and beyond Total budget: ~32 M€ DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 10

  10. 9 12 18 Technology Growth Optical Fibre(bits per second) Doubling Time(months) Gilder’s Law(32X in 4 yrs) Data Storage(bits per sq. inch) Storage Law (16X in 4yrs) Performance per Dollar Spent Chip capacity(# transistors) Moore’s Law(5X in 4yrs) 0 1 2 3 4 5 Number of Years Triumph of Light – Scientific American. George Stix, January 2001 DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 11

  11. Database Growth PDB Content Growth EMBL Content Growth DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 12

  12. CERN: Data intensive science • The Large Hadron Collider (LHC) • The most powerful instrument ever built to investigate elementary particles physics • Data Challenge: • 10Petabytes/year of data • 20 million CDs each year • Simulation, reconstruction, analysis: • LHC data handling requires computing power equivalent to ~100,000 of today's fastest PC processors Mont Blanc (4810 m) Downtown Geneva DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 13

  13. Main EGEE Grid Services • Authentication & Authorisation • Job submission service • Resource Broker • Replica Management • EDG-Replica-Manager • Mass storage system support • Logging & Bookkeeping • Monitoring DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 14

  14. Replica Catalogue Input “sandbox” DataSets info Information Service UI JDL Output “sandbox” grid-proxy-init Resource Broker SE & CE info Output “sandbox” Expanded JDL Job Submit Event Author. &Authen. Job Query Input “sandbox” + Broker Info Publish Job Status Storage Element Globus RSL Job Submission Service Job Status Computing Element Logging & Book-keeping Job Status The lifecycle of an EGEE job DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 15

  15. Globus Toolkit 2: Key components • Grid Security Infrastructure (GSI) • X.509 authentication with delegates and single sign-on • Grid Resource Allocation Mgmt (GRAM) • Remote allocation, reservation, monitoring, control of compute resources • GridFTP protocol (FTP extensions) • High-performance data access & transport • Grid Resource Information Service (GRIS) +Monitoring and Discovery Service (MDS) • Access to structure & state information • XIO • TCP, UDP, IP multicast, and file I/O DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 16

  16. Using Web Services for Grids • Standard interface definition language • Foundation for better engineering • Standard invocation mechanism • Foundation for interoperability • Other channels can be used for performance • Good commercial tooling • Reliability and performance • Service-Oriented Architecture • Valuable scalability and durability properties DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 17

  17. Cataloging Provisioning Work in Progress: The Open Grid Services Architecture VO Mgmt Integration Policy Mgmt Access Context Services Information Services Data Services Trouble- shooting Event Mgmt Discovery Logging Execution Mgmt Services Infrastructure Services Application Mgmt Workflow Mgmt Workload Mgmt Execution Planning Job Mgmt WSRF WSN WSDM Naming Self Mgmt Services Resource Mgmt Services Reservation Configuration Deployment Provisioning Security Services Heterogeneity Mgmt Authentication Optimization Authorization Service Level Attainment Integrity QoS Mgmt Boundary Traversal DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 18

  18. 1a. Request to Registry for sources of data about “x” OGSA-DAI: Data Access & Integration Services SOAP/HTTP service creation API interactions Registry 1b. Registry responds with Factory handle 2a. Request to Factory for access to database Factory Client 2c. Factory returns handle of GDS to client 2b. Factory creates GridDataService to manage access 3a. Client queries GDS with XPath, SQL, etc XML / Relational database Grid Data Service 3c. Results of query returned to client as XML 3b. GDS interacts with database DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 19

  19. SOAP/HTTP service creation API interactions ProblemSolving Environment SemanticMeta data Application Code Future DAI Services 1a. Request to Registry for sources of data about “x” & “y” Data Registry 1b. Registry responds with Factory handle 2a. Request to Factory for access and integration from resources Sx and Sy Data Access & Integrationmaster 2c. Factory returns handle of GDS to client 3b. Client 2b. Factory creates tells GridDataServices network analyst Client 3a. Client submits sequence of scripts each has a set of queries XMLdatabase GDTS GDS GDTS to GDS with XPath, SQL, etc 1 Analyst GDS Relationaldatabase 3c. Sequences of result sets returned to analyst as formatted binary described in GDTS GDS GDS GDTS 2 3 a standard XML notation DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 20

  20. Web Service Grid Implementations • Globus Toolkit 4 • Web Service versions of the Globus components • OGSA-DAI • GLite • Web Services for EGEE • OMII • Open Middleware Infrastructure Institute (UK) • OGSA-DAI • Others DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 21

  21. GLite: Web Services for EGEE • Focus is upon interfaces that can be composed into useful services DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 22

  22. Storage Functionality • Storage Resource Manager interface • File copying and replication • File management and control • Using SRM standards [GSM WG](with possible evolution) • Posix-like file I/O • Direct file access • Open, read, write Control SRM interface POSIXAPI File I/O rfio dcap chirp aio User dCache NeST Castor Disk DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 23

  23. Data management (file replication) • Scheduled data transfers (like jobs) • SRM based storage • Reliable file transfer • Current model: dotted line = query full line = request single box = service per sitedouble box = service per VO DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 24

  24. Many open issues, including: • Naming system • (WSRF, Handle, ARK, LSID, …) • OASIS committee to look at this • Metadata standards • CRG extending CIM to describe relational databases • Semantic Web • etc. • Architectural details • Work still in progress DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 25

  25. Summary • Data Grids face the same problems as digital achives • Web Services seem to be a good platform for Grids • Standards are essential for interoperability DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 26

  26. My fifth hat… the e-Science Institute • Research visitors • 1 week to 1 year • Event programme • Workshops • Conferences • Training • Summer schools DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 27

  27. Questions? • http://www.nesc.ac.uk • http://public.eu-egee.org/ • http://www.dcc.ac.uk/ • http://www.ogsadai.org.uk • https://forge.gridforum.org/projects/ogsa-wg/ DELAMAN Access Workshop: Grid Components – November 30th, 2004 - 28

More Related