1 / 42

European and German Grid Computing Projects

European and German Grid Computing Projects. Marcel Kunze Forschungszentrum Karlsruhe Marcel.Kunze@hik.fzk.de DESY Seminar December 2002. Grid Computing Potential. 1980 2005.

quanda
Download Presentation

European and German Grid Computing Projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. European and German Grid Computing Projects Marcel Kunze Forschungszentrum Karlsruhe Marcel.Kunze@hik.fzk.de DESY Seminar December 2002

  2. Grid Computing Potential 1980 2005 Marcel Kunze, DESY Seminar

  3. What is Grid Computing? Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations Marcel Kunze, DESY Seminar

  4. Grid Computing: Diverse Areas of Application • Computational GridHigh Performance Computing (HPC)Perspective: Parallelization of programs, as fast as possible • Data GridHigh Throughput Computing (HTC)Perspective : Parallelization of data, as much as possible • Gaming GridCommunication between playersPerspective: LAN + WAN-Party, as entertaining as possible Marcel Kunze, DESY Seminar

  5. Grid Computing: Commercial Aspects • Huge potential of Grid Computing in the field of information technology: • Cost reduction through more efficient/diverse use of system resources • „Resource on demand“Added flexibility, e.g. dynamically satisfy increased computing demand in bank accounting sector at the end of an accounting period • „Business continuity“ Proliferation of mission-critical IT-Services to ease disaster recovery • Possibility of system consolidation: Reduction of system complexity by standardization of services Marcel Kunze, DESY Seminar

  6. Grid Computing: Scientific Aspects • Enables formation of „critical mass“ of resources and expertise to allow for interdisciplinary projects in so-called “Virtual Organizations” (VO) • Tele-Collaboration • Instrument Sharing • Computing Resource Sharing • Data Sharing • Potential fields of application • Bio-Informatics and Medicine • Astronomy • Earth Sciences • High Energy Physics (HEP) Marcel Kunze, DESY Seminar

  7. Data mining on genomic databases (exponential growth) Indexing of medical databases (Tb/hospital/year) Collaborative framework for large scale experiments (e.g. epidemiological studies) Parallel processing for Databases analysis Complex 3D modelling Biomedical Applications Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  8. Astronomy: Past and Future of the Universe New phase of astronomy, storing, searching and analysing Petabytes of data: • Virtual Observatories – GAVO, NVO, AVO, AstroGrid • Store all wavelengths, need distributed joins • NVO 500 TB/yr from 2004 • Grid Computing might help to • Master the data streams • Federate databases with different schema • Maintain meta data (information) and provenance data (history) Crab Nebula viewed At four different wavelengths: X-ray, optical, infrared, radio. Marcel Kunze, DESY Seminar

  9. Earth Observation • ESA missions: • about 100 Gbytes of data per day (ERS 1/2) • 500 Gbytes, for the next ENVISAT mission (launched March 1st) • EO requirements for the Grid: • enhance the ability to access high level products • allow reprocessing of large historical archives • improve Earth science complex applications (data fusion, data mining, modelling …) Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  10. What is Grid Computing? Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations Marcel Kunze, DESY Seminar

  11. Elements of the Problem • Resource sharing • Computers, storage, sensors, networks, … • Heterogeneity of device, mechanism, policy • Sharing conditional: negotiation, payment, … • Coordinated problem solving • Integration of distributed resources • Compound quality of service requirements • Dynamic, multi-institutional virtual orgs • Dynamic overlays on classic org structures • Map to underlying control mechanisms Marcel Kunze, DESY Seminar

  12. The Grid World: Current Status • Dozens of major Grid projects in scientific & technical computing/research & education • Deployment, application, technology • Considerable consensus on key concepts and technologies • Open source Globus Toolkit™ a de facto standard for major protocols & services • Far from complete or perfect, but out there, evolving rapidly, and large tool/user base • Global Grid Forum a significant force • Industrial interest emerging rapidly Marcel Kunze, DESY Seminar

  13. “Web Services” • Increasingly popular standards-based framework for accessing network applications • W3C standardization; Microsoft, IBM, Sun, others • WSDL: Web Services Description Language • Interface Definition Language for Web services • SOAP: Simple Object Access Protocol • XML-based RPC protocol; common WSDL target • WS-Inspection • Conventions for locating service descriptions • UDDI: Universal Description, Discovery & Integration • Directory for Web services Marcel Kunze, DESY Seminar

  14. Open Grid Services Architecture(OGSA) • Platform independent industrial and scientific standard to construct Grid computing components • Service orientation to virtualize resources • From Web services: • Standard interface definition mechanisms: multiple protocol bindings, multiple implementations, local/remote transparency • Building on Globus Toolkit: • Grid service: semantics for service interactions • Management of transient instances (& state) • Factory, Registry, Discovery, other services • Reliable and secure transport • Multiple hosting targets: J2EE, .NET, “C”, … Marcel Kunze, DESY Seminar

  15. The Grid Service • A (potentially transient) Web service with specified interfaces & behaviors, including • Creation (Factory) • Global naming (GSH) & references (GSR) • Lifetime management • Registration & Discovery • Authorization • Notification • Concurrency • Manageability Marcel Kunze, DESY Seminar

  16. Grid Projects Marcel Kunze, DESY Seminar

  17. GriPhyN PPDG iVDGL GRID Projects World Wide • EU • EDG (EU-IST) – R&D EU GRID project [ www.edg.org ] • CrossGRID QoS – Real Time apps. [ www.crossgrid.org ] • DataTAG GLUE (EU-USA) [ www.datatag.org ] • LCG The LHC Computing GRID – Deployment [ cern.ch/lcg ] • The new 16,2 B Euro EU VI Framework Prog. GEANT based GRID projects • USA • GriPhyN iVDGL-VDTv1 PPDG ( NSF, DoE ) [ www.griphyn.org ] [ www.idvgl.org ] [ www.ppdg.org ] • Asia • ApGrid Pragma (USA-Asia) [ www.apgrid.org ] Marcel Kunze, DESY Seminar

  18. IST Grid Project Space EGSO CROSSGRID GRIA EUROGRID DATAGRID GRIP GRIDLAB DAMIEN DATATAG Industry / business Science -Links with European National efforts - Links with US projects (GriPhyN, PPDG, iVDGL,…) Applications Middleware & Tools Underlying Infrastructures A. Baxevanidis Marcel Kunze, DESY Seminar

  19. The EU DataGrid Project • 21 Partners • 9.8 M Euros EU funding over 3 years • 90% for middleware and applications (Particle Physics, Earth Obs. and Biomedical) • Three year phased developments & demos (2001-2003) • Spin-off: • DataTAG (2002-2003) • CrossGrid (2002-2004) • GridStart (2002-2004) Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  20. DataGrid Partners • Main Partners • CERN – International (Switzerland/France) • CNRS - France • ESA/ESRIN – International (Italy) • INFN – Italy • NIKHEF – The Netherlands • PPARC - UK • Industrial Partners • Datamat (Italy) • IBM-UK (UK) • CS-SI (France) • Research and Academic Institutes • CESNET (Czech Republic) • Commissariat à l'énergie atomique (CEA) – France • Computer and Automation Research Institute,  Hungarian Academy of Sciences (MTA SZTAKI) • Consiglio Nazionale delle Ricerche (Italy) • Helsinki Institute of Physics – Finland • Institut de Fisica d'Altes Energies (IFAE) - Spain • Istituto Trentino di Cultura (IRST) – Italy • Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany • Royal Netherlands Meteorological Institute (KNMI) • Ruprecht-Karls-Universität Heidelberg - Germany • Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands • Swedish Research Council - Sweden Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  21. EU DataGrid Project Objectives • Use Grid technology to develop a sustainable computing model for effective share of computing resources and data for large scientific communities • Specific project objectives: • Middleware for fabric & Grid management • Large scale testbeds • Production quality demonstrations • Key products: • Resource broker (Compute elements, storage elements) • Replica manager • Virtual Organization manager Marcel Kunze, DESY Seminar

  22. CrossGrid Using the same security certs. Testbed sites install EDG software Extending it for needs of intensive interactive applications Participating in the EDG testing activities Representatives in each projects architecture & management groups DataTAG (EDT) EDT is deploying EDG sw to investigate inter-operability with US projects (iVDGL, GriPhyN, PPDG) Results feedback into EDG software releases (e.g. GLUE compatible information providers/consumers) NorduGrid Using the same security certs. Involved in EDG architecture work Good ideas for gatekeeper and MDS configuration Helped develop GDMP and GSI extensions for Replica Catalog Involved in GLUE schema work Security policy Middleware testing Working in WP8 (HEP applications) iVDGL/GriPhyN/PPDG US members in EDG architecture group Looking for common packaging and toolkit usage solutions GriPhyN PPDG iVDGL EDG: Interaction with Sister Projects Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  23. CrossGrid: An EDG Partner • New category of Grid enabled applications • Computing and data intensive • Distributed • Interactive, near real time response (a person in a loop) • Layered • New programming tools • Grid more user friendly, secure and efficient • Interoperability with other Grids • Implementation of standards Marcel Kunze, DESY Seminar

  24. CrossGrid Layered Architecture Weather Forecast application Biomedical Application Flood Application HEP Interactive Distributed Data Access Application HEP Data Mining on Grid Application HEP High LevelTrigger Applications And Supporting Tools Portal MPI Verification Performance Analysis Metrics and Benchmarks Applications Development Support MPICH-G Globus Replica Manager Interactive Distributed Data Access Grid Visualisation Kernel Data Mining on Grid Distributed Data Collection Roaming Access Grid Common Services Datagrid Job Manager DataGrid Replica Manager Grid Resource Management User Interaction Service Grid Monitoring GRAM Replica Catalog GSI Globus-IO MDS GridFTP GASS Resource Manager Local Resources Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Secondary Storage CPU Optimization of Data Access Scientific Instruments (Medical Scaners, Satelites, Radars) Detector Local High Level Trigger VR systems (Caves, immerse desks) Visualization tools Tertiary Storage Marcel Kunze, DESY Seminar

  25. Interactive Treatment Planning The vascular geometry can be modified using a library of models Draw interactively Computational geometry Marcel Kunze, DESY Seminar

  26. Biomedical ApplicationSample pulse flow simulation Marcel Kunze, DESY Seminar

  27. GriPhyN PPDG iVDGL Benefit of Standardization Through links with sister projects, there is the potential for a uniform global scientific applications grid Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  28. Common WorldGrid Demo at IST2002 VO centric Ganglia monitor • WorldGrid Demo together with US and EU partners at IST2002 • Seamless interoperation of EDG with US Middleware (via GLUE) Marcel Kunze, DESY Seminar

  29. WorldGrid Lessons (+) • Advantages of the Grid: • Possibility to execute tasks and move files over a distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords) • Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system • Possibility to do worldwide production in a perfectly coordinated way, using identical software (RPMs), scripts and databases • GLUE interfacing works to make the middleware worlds talk to each other. Better solution: Agree on open standards environment like OGSA ! Marcel Kunze, DESY Seminar

  30. WorldGrid Lessons (-) • EDG stability very much dependent on the quality of the basic building blocks (Globus, Condor, etc) • Globus support needs to be strengthened (formal contract, European support center, internal Globus support team) • Understand impact of OGSA and industrial involvement • EDG Toolkit might need to be refactored Marcel Kunze, DESY Seminar

  31. EDG Future Plans • Concentrate on production quality and real applications • Educate new users and disseminate results • Complete the program of work till end of 2003 • Port EDG to other platforms than Linux (Solaris) • Port EDG to Globus ToolKit version 3 (OGSA) • Make plans to conserve momentum and assets in the EU FP6 Marcel Kunze, DESY Seminar

  32. A Proposal to the EU • EDG has already demonstrated the viability of Grid technology • EU had a fast start in supporting 18 Grid projects in the last two years! • RN Geant offers an excellent basis for a large European Grid infrastructure • The EU FP6 program should encourage and support the deployment and production quality operation of a large international Grid infrastructure open to research and industry in Europe Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  33. Enabling Grids and e-Science in Europe • EGEE: Integrated Infrastructure Initiative to support European Research Area • Vision: to create and deploy Grid technologies to enable the widespread uptake of e-Science applications throughout the European Research Area • Four key objectives: • integrating Grid technological developments from across Europe; • establishing a Europe-wide Grid infrastructure for science and industry with a focus on heterogeneity and interoperability; • enabling the creation of e-Science applications from across the scientific and industrial spectrum; • ensuring the timely delivery of the project’s programme of work, guided by the needs of academic and industrial partners. Start by integration of the national Grid initiatives Fabrizio Gagliardi Marcel Kunze, DESY Seminar

  34. GRID Computing Center GridKa GRID Computing Center: Infrastructure and Services Competence Center: Applications and e-Science Marcel Kunze, DESY Seminar

  35. Partners in the HGF24.000 employees, 2.5 Billion Euro yearly budget Grid Computing perfectly well matches the HGF vision of “Konzertierte Forschung” • ALFRED-WEGENER-INSTITUT FÜR POLAR- UND MEERESFORSCHUNG AWI • DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY • DEUTSCHES KREBSFORSCHUNGSZENTRUM DKFZ • DEUTSCHES ZENTRUM FÜR LUFT- UND RAUMFAHRT DLR • FORSCHUNGSZENTRUM JÜLICH FZJ • FORSCHUNGSZENTRUM KARLSRUHE FZK • GESELLSCHAFT FÜR BIOTECHNOLOGISCHE FORSCHUNG GBF • GEOFORSCHUNGSZENTRUM POTSDAM GFZ • GKSS-FORSCHUNGSZENTRUM GEESTHACHT GKSS • GSF-FORSCHUNGSZENTRUM FÜR UMWELT UND GESUNDHEIT GSF • GESELLSCHAFT FÜR SCHWERIONENFORSCHUNG GSI • HAHN-MEITNER-INSTITUT BERLIN HMI • MAX-PLANCK-INSTITUT FÜR PLASMAPHYSIK IPP • MAX-DELBRÜCK-CENTRUM FÜR MOLEKULARE MEDIZIN MDC • UFZ-UMWELTFORSCHUNGSZENTRUM LEIPZIG-HALLE UFZ Marcel Kunze, DESY Seminar

  36. Rostock Kiel Hamburg Global Upstream Oldenburg Braunschweig Hannover Berlin Magdeburg Bielefeld Essen Göttingen Leipzig St. Augustin Dresden Marburg Ilmenau Aachen Würzburg Frankfurt Erlangen Heidelberg Karlsruhe Regensburg Kaiserslautern Stuttgart Garching Augsburg A German Grid InitiativeD-GRID • Initially driven by the HGF centers and the DFN • Open to accept further partners in academia and industry • Aim at a coordination of Grid activities • Deployment of a new generation networking infrastructure (towards “TeraGrids”) • Promotion of open standards for interfaces and protocols (GGF) 10 Gbit/s 2,4 Gbit/s 2,4 Gbit/s 622 Mbit/s Marcel Kunze, DESY Seminar

  37. Scientific Program • Installation of a coordination office for Grid Computing • Foster the enhanced application of IT in the field of scientific and engineering disciplines • Management of the relationship between the German Grid competence centers and representation of their common interests in the Global Grid Forum • Training and education in the field of Grid Computing and e-Science • Deployment of national Grid resources and development of a program for academic and industrial projects Marcel Kunze, DESY Seminar

  38. Technical Program • Goal: Integration of all kind of resourcese.g. sensors, computing infrastructure, people,… • Deployment of a state-of-the-art production backbone (2003: 10 GBPS, 2006: 100 GBPS) • Discussion of safety and trust relationship • Management of German Grid certificates • Selection of middleware components and installation procedures (Globus, UNICORE) • Infrastructure: Find qualified manpower to offer and support the additional services • Definition of cost model (Grid-Accounting, credit points) • Collaboration with the German HPC centers => D-GRID Kickoff Workshop in January 2003 (Wissenschaftszentrum Bonn) Marcel Kunze, DESY Seminar

  39. Where do we go from here?? Marcel Kunze, DESY Seminar

  40. e-Science (enhanced Science)Konzertierte Forschung • e-Science is about more than networks, GRIDs, High Performance Computing, cluster computing ... • e-Science: „e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.“ (Dr. John Taylor, director of UK research council) • Foster the transition of data Grid to semantic Grid • Federation of resources • Federation of distinguished data sources • Knowledge retrieval and collaboratories Marcel Kunze, DESY Seminar

  41. Semantic Grid: Three Layer Grid Abstraction e-Science Knowledge Grid Control Information Grid Data to Knowledge Computation/Data Grid Automation Marcel Kunze, DESY Seminar

  42. Grid-enabled Applications Prototype Grid Infrastructures Gèant: World Class Networking The Future • Where do we need to get to ? • Applications to support an “e-society” (“Cyber-Infrastructure”) • An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”) • A powerful and flexible network infrastructure • Where do we need to invest ? • Applications targeted at realistic problems in “e-science” • Prototypes of Grid infrastructures • Maintain and improve the GEANT network Marcel Kunze, DESY Seminar

More Related