1 / 41

Grid Infrastructure & Related Projects

Grid Infrastructure & Related Projects. Eddie Aronovich Tel-Aviv University, School of CS Eddie.Aronovich@cs.tau.ac.il. Acknowledgements. Presentation is based on slides from: Roberto Barbera, University of Catania and INFN (EGEE Tutorial Roma, 02.11.2005)

margie
Download Presentation

Grid Infrastructure & Related Projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Infrastructure & Related Projects Eddie Aronovich Tel-Aviv University, School of CS Eddie.Aronovich@cs.tau.ac.il

  2. Acknowledgements • Presentation is based on slides from: • Roberto Barbera, University of Catania and INFN (EGEE Tutorial Roma, 02.11.2005) • Mike Mineter, Concepts of grid computing • Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (Naregi Symposium 2005 – Tokyo) • Fabrizio Gagliardi, EGEE Project Director, CERN, Geneva, Switzerland (APAC, 27 September 2005) • Guy Warner, NeSC Training Team (An Induction to EGEE for GOSC and the NGS NeSC, 8th December 2004 ) • http://www.beingrid.eu Eddie Aronovich – CS_IAP, Apr 2007

  3. What is a grid ? • EGEE project • Connectivity with other projects • How a job is submitted • A few more words Eddie Aronovich – CS_IAP, Apr 2007

  4. When the Grid can do ? • Coordinates Distributed Resources • Open for connectivity (using standard, open, general-purpose protocols) • Deliver non-trivial qualities of service Eddie Aronovich – CS_IAP, Apr 2007

  5. INTERNET A grid • Grid middleware runs on each shared resource • Data storage • (Usually) batch jobs on pools of processors • Users join VO’s • Virtual organisation negotiates with sites to agree access to resources • Distributed services (both people and middleware) enable the grid Eddie Aronovich – CS_IAP, Apr 2007

  6. What is a grid ? • EGEE project • How a job is submitted • A few more words Eddie Aronovich – CS_IAP, Apr 2007

  7. EGEE project in 1K words https://goc.grid-support.ac.uk/gridsite/monitoring/ Eddie Aronovich – CS_IAP, Apr 2007

  8. 30+ Grid Projects @ EGEE’06 Eddie Aronovich – CS_IAP, Apr 2007

  9. 20 + Businesses @ EGEE06 Capitalising on e-Science to make e-Business Eddie Aronovich – CS_IAP, Apr 2007

  10. EU GRID Related EU projects ISSeG Eddie Aronovich – CS_IAP, Apr 2007

  11. EGEE as partner • Ongoing collaborations • with non EU partners in EGEE: US, Israel, Russia, Korea, Taiwan… • with other European infrastructure projects: • GÉANT • DEISA • SEE-GRID • with other European grid projects • DILIGENT • GRIDCC • with non-European projects: • OSG: OpenScienceGrid (USA) • NAREGI (Japan) • EGEE as incubator • >10 related projects have been created Eddie Aronovich – CS_IAP, Apr 2007

  12. Collaborating e-Infrastructures TWGRID Potential for linking ~80 countries Eddie Aronovich – CS_IAP, Apr 2007

  13. e-Infrastructure • implementation blocks From a talk by Mario Campolargo, Brussels, 30 May 2005 Eddie Aronovich – CS_IAP, Apr 2007

  14. GRID . INFRASTRUCTURE KNOWLEDGE . INFRASTRUCTURE GÉANT . INFRASTRUCTURE Support for digital repositories 3 layered model to support access to heterogeneousinformation and connect resources throughcommon shared services Grids for digital repositories: • Sharing of resources • Secure Access Control • Data management • Execution of computationally demanding applications (e.g. multi-media content) Eddie Aronovich – CS_IAP, Apr 2007

  15. What is a grid ? • EGEE project • Connectivity with other projects • How a job is submitted • A few more words Eddie Aronovich – CS_IAP, Apr 2007

  16. Input “sandbox” DataSets info Output “sandbox” SE & CE info Job Submit Event Job Query Publish Job Status Storage Element Jobs life (major components) Replica Catalogue “User interface” Information Service Resource Broker Author. &Authen. Input “sandbox” + Broker Info Output “sandbox” Logging & Book-keeping Computing Element Job Status Eddie Aronovich – CS_IAP, Apr 2007

  17. UI RB/WMS node Replica Location Server Network Server Workload Manager Inform. Service Job Contr. Characts. & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  18. Job Status UI RB node submitted Replica Location Server Network Server Workload Manager Inform. Service UI: allows users to access the functionalities of the WMS (via command line, GUI, C++ and Java APIs) Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  19. edg-job-submit myjob.jdl Myjob.jdl JobType = “Normal”; Executable = "$(CMS)/exe/sum.exe"; InputSandbox = {"/home/user/WP1testC","/home/file*”, "/home/user/DATA/*"}; OutputSandbox = {“sim.err”, “test.out”, “sim.log"}; Requirements = other. GlueHostOperatingSystemName == “linux" && other. GlueHostOperatingSystemRelease == "Red Hat 7.3“ && other.GlueCEPolicyMaxCPUTime > 10000; Rank = other.GlueCEStateFreeCPUs; Job Status UI RB node submitted Replica Location Server Network Server Workload Manager Inform. Service Job Contr. - CondorG CE characts & status SE characts & status Job Description Language (JDL) to specify job characteristics and requirements Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  20. submitted waiting UI NS: network daemon responsible for accepting incoming requests RB node Job Status Replica Location Server Network Server Job Input Sandbox files Workload Manager Inform. Service RB storage Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  21. submitted waiting UI RB node Job Status Replica Location Server Network Server Job Workload manager Inform. Service RB storage WM: acts to satisfy the request Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  22. Job Status submitted waiting UI RB node Replica Location Server Network Server Match- Maker/ Broker Workload Manager Inform. Service RB storage Where must this job be executed ? Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  23. submitted waiting UI RB node Job Status Matchmaker: responsible to find the “best” CE for a job Replica Location Server Network Server Match- Maker/ Broker Workload Manager Inform. Service RB storage Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  24. Where are (which SEs) the needed data ? submitted waiting UI RB node Job Status Replica Location Server Network Server Match- Maker/ Broker Workload Manager Inform. Service RB storage What is the status of the Grid ? Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  25. submitted waiting UI RB node Job Status Replica Location Server Network Server Match- Maker/ Broker Workload Manager Inform. Service RB storage CE choice Job Contr. - CondorG CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  26. submitted waiting UI RB node Job Status Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Adapter Job Contr. - CondorG Job Adapter: responsible for the final “touches” to the job before performing submission (e.g. creation of wrapper script, PFN, etc.) CE characts & status SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  27. submitted waiting UI ready RB node Job Status Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Job Contr. CE characts & status Job Controller: responsible for the actual job management operations (done via CondorG) SE characts & status Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  28. submitted waiting UI ready scheduled RB node Job Status Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Contr. - CondorG CE characts & status SE characts & status Job Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  29. submitted waiting UI ready scheduled running Job RB node Job Status Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Contr. - CondorG Input Sandbox files “Grid enabled” data transfers/ accesses Storage Element Computing Element Eddie Aronovich – CS_IAP, Apr 2007

  30. submitted waiting UI ready scheduled running done RB node Job Status Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Contr. - CondorG Output Sandbox files Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  31. submitted waiting UI ready scheduled running done RB node Job Status edg-job-get-output <dg-job-id> Replica Location Server Network Server Workload Manager Inform. Service RB storage Job Contr. - CondorG Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  32. UI RB node Job Status submitted Replica Location Server Network Server waiting RB storage ready Workload Manager Output Sandbox files Inform. Service scheduled Job Contr. - CondorG running done cleared Computing Element Storage Element Eddie Aronovich – CS_IAP, Apr 2007

  33. UI RB node edg-job-status <dg-job-id> edg-job-get-logging-info <dg-job-id> Network Server LB: receives and stores job events; processes corresponding job status Workload Manager Job status Logging & Bookkeeping Job Contr. - CondorG Log Monitor Log of job events LM: parses CondorG log file (where CondorG logs info about jobs) and notifies LB Computing Element Eddie Aronovich – CS_IAP, Apr 2007

  34. No Cross- Domain Trust Certification Certification Authority Authority Policy Policy Authority Authority Sub-Domain B1 Sub-Domain A1 Domain A Domain B Task Federation Service GSI Server X Server Y Virtual Organization Domain What about security ? Eddie Aronovich – CS_IAP, Apr 2007

  35. What is a grid ? • EGEE project • Connectivity with other projects • How a job is submitted • A few more words Eddie Aronovich – CS_IAP, Apr 2007

  36. Used CPU time this year Eddie Aronovich – CS_IAP, Apr 2007

  37. Over the Chasm Eddie Aronovich – CS_IAP, Apr 2007

  38. Applications in EGEE • Astrophysics - MAGIC, Planck • Computational Chemistry • Earth Sciences - Earth Observation, Solid Earth Physics, Hydrology, Climate • Financial Simulation - E-GRID • Fusion • Geophysics - EGEODE • High Energy Physics - LHC (ALICE, ATLAS, CMS, LHCb) BaBar, CDF, DØ, ZEUS • Life Sciences - Bioinformatics (Drug Discovery, GPS@, Xmipp_MLrefine, etc.) • Medical imaging (GATE, CDSS, gPTM3D, SiMRI 3D, etc.) • Multimedia press for more details Eddie Aronovich – CS_IAP, Apr 2007

  39. Steps to become grid production • Contact IAGneeded contribution: • 30CPUs or cores • 1TB of storage attached to the processosrs • 4 computers for operational activities • Plan your site • Get a certificates • Install the site • Certify the site…. Eddie Aronovich – CS_IAP, Apr 2007

  40. Questions ?

  41. Contacts • Israeli Academic Grid (IAG) http://iag.iucc.ac.il/ • EGEE Website http://www.eu-egee.org • Grid Demos https://glite-demo.ct.infn.it/ Eddie Aronovich – CS_IAP, Apr 2007

More Related