1 / 26

Innovative services and tools for NorduGrid (NGIn) Nordunet 2008, 11.04.08

Innovative services and tools for NorduGrid (NGIn) Nordunet 2008, 11.04.08. Jon Kerr Nilsen e-mail: j.k.nilsen[at]fys.uio.no University of Oslo, Department of Physics PhD student, NGIn. The NorduGrid Collaboration. A research collaboration that: Develops and supports ARC middleware

draco
Download Presentation

Innovative services and tools for NorduGrid (NGIn) Nordunet 2008, 11.04.08

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Innovative services and tools for NorduGrid (NGIn)Nordunet 2008, 11.04.08 Jon Kerr Nilsen e-mail: j.k.nilsen[at]fys.uio.no University of Oslo, Department of Physics PhD student, NGIn

  2. The NorduGrid Collaboration • A research collaboration that: • Develops and supports ARC middleware • Provides middleware to research groups and national Grids • Coordinates the various inputs to the KnowARC code Nordic Grid Neighbourhood www.nordugrid.org

  3. ARC middleware deployed on 50+ sites in 15+ countries all around the world http://www.nordugrid.org/monitor www.nordugrid.org

  4. Large Hadron Collider Reduced to few hundred ”good” events/s 40 million events/s per each of 4 experiments ~15 petabytes of data per year www.nordugrid.org

  5. NOS-N DK S SF N Nordic Data Grid Facility Nordic Data Grid Facility • NDGF pilot project started 2003 • Goal: Create a Nordic Grid infrastructure • Builds on history and competence of Nordic Grid collaboration • NorduGrid/ARC middleware chosen as basis • Funded (2 M.EUR/year) by National Research Councils of the Nordic countries • Coordinates and hosts major Grid projects (e.g. the Nordic LHC Tier-1) • Operates Nordic storage facility for major projects • Develops grid middleware and services www.nordugrid.org

  6. EU KnowARC project:10 partners from 7 countries NorduGrid members and research teams in medicine, bioinformatics, physics, engineering, automotive industry apps, IT Univ. of Oslo Norway Univ. of Lund Sweden Univ. of Copenhagen Denmark Univ. of Uppsala Sweden NIIF Hungary Geneva Univ. Hospitals Switzerland Univ. of Kosice Slovakia Univ. of Lübeck Germany science+computing ag Germany SUN Microsystems Hungary www.nordugrid.org

  7. Popular Grid middleware for know-how sharing Existing ARCmiddleware Industry, academia application developers Experiencedmiddlewaredevelopers Standards, Interoperability Goals of KnowARC The mission of KnowARC is • to create a novel, powerful Next Generation Grid middleware • extend and re-design ARC • ARC in major Linux distributions • New platforms: Windows, Solaris, Mac OS-X • to promote Grid standardization and interoperability • interoperability with other Grid solutions (EGEE and OGF compliant grids) • possibilities to include (or to be included in) other infrastructures • to prove usage in Health care, Industry and Science • Bioinformatics: Autoimmune Diseases • Medical Informatics: Lung Diseases • Automotive industry www.nordugrid.org

  8. Next Generation ARC • Developed using SOA and existing and comming standards • Web Services, SOAP, GLUE2.0 ... • Non-intrusive (no need to change excisting computing resources) • Client-Server communication mainly through Web Services www.nordugrid.org

  9. Nordic Grid Neighbourhood RUSSIA: St. Petersburg State University, Dep. of Physics Petrodvorets Telecommunication Center Dubna? ESTONIA: National Institute of Chemical Physics and Biophysics (NICPB) Estonian Educational and Research Network (EENet) Tartu University, Institute of Technology FINLAND: University of Helsinki, Helsinki Institute of Physics LITHUANIA: Vilnius University, Institute of Materials Science and Applied Research LATVIA ICELAND DENMARK: University of Copenhagen (NBI, DIKU) Southern Danish University, Dep. of Computer Science Aalborg University, Dep. of Computer Science SWEDEN: Lund University, Department of Physics Linköping University, National Supercomputer Center Umeå University, Department of Computing Science and HPC2N Uppsala University, Department of Radiation Sciences Royal Institute of Technology, NADA/PDC, KTH Stockholm University and KTH, Department of Physics NORWAY: University of Oslo, Physics institute University of Bergen, Department of Physics University of Tromsø, Department of Computer Science NTNU, Trondheim, Department of Computer and Information Science NTNU, Trondheim, IT division (ITEA) Project: … , NO27, Nabo 05-06, Nordic Grid Neighbourhood Project no.: 172724/V11 We have read and approved your final academic and financial project report for the funding period 2005-2006. We are pleased that most of the planned goals of the network have been reached through, i.a. workshops, conferences and tutorials. We are impressed by the number of partners and participants in the various activities. ... 9countries, 24 institutions

  10. NGIn for Research and Education • Innovative services and tools for NorduGrid. PI: Farid Ould-Saada, University of Oslo. Nordunet3-funded Internet Research Program • 4 year grant • Oslo, Lund, Uppsala, Copenhagen, Helsinki, Reykjavik • Mainly: • Data Management: storage, indexing and movement • Education and expert training in Grid computing • 5 PhD grants starting January 2007 • Grid school • call for collaboration with Baltic and Russia … ?? http://www.nordugrid.org/ngin http://www.nordunet3.org/ www.nordugrid.org

  11. NGIn management • Project leader • Prof. Farid Ould-Saada, Univ. of Oslo, Norway • Technical coordinator • Prof. Alex Read, Univ. of Oslo, Norway • Steering Group • Prof. Farid Ould-Saada, Univ. of Oslo, Norway • Prof. Paula Eerola, Univ. of Lund, Sweden • Prof. Tord Ekelöf, Univ. of Uppsala • Dr. Miika Tuisku, Helsinki Institute of Physics, Finland • Dr. Anders Wäänänen, Niels Bohr Institute, Denmark • Participating institutions • Physics Department, Univ. of Oslo, Norway • Physics Department, Univ. of Lund, Sweden • Physics Department, Univ. of Uppsala, Sweden • Department of Radiation Sciences, Univ. of Uppsala, Sweden • Niels Bohr Institute, Univ. of Copenhagen and Univ. of Aalborg, Denmark • Helsinki Institute of Physics (HIP), Univ. of Helsinki, Finland • CSC, Finnish IT Centre for Science, Helsinki, Finland • Univ. of Iceland, Reykjavik, Iceland www.nordugrid.org

  12. NGIn - training new Grid experts • From the original application: ”The goal of this project is twofold: To extend the excisting middleware and to train new Grid experts...” • NGIn obtained 30% of funding applied for • The primary goal of the NGIn project is now to train new Grid experts • Training programme includes: • Grid PhD students • A Nordic Grid school (first held in Copenhagen in September 2007) • The NGIn activities will go in line with the other NorduGrid related projects • Budget 2008: www.nordugrid.org

  13. NGIn PhD students • Four PhD students hired, fifth student soon to be hired • In chronological order: • PhD 1, Jon Kerr Nilsen, Univ. of Oslo, Norway • PhD 2, Mika Silander, Helsinki Institute of Physics, Finland • PhD 3, Salman Toor, Univ. of Uppsala, Sweden • PhD 4, Phillippe Gros, Univ. of Lund, Sweden • PhD 5, to be announced soon, Univ. of Oslo, Norway • The students are funded 50% by Nordunet3, 50% by local sources • All the PhD projects involves data management www.nordugrid.org

  14. Jon Kerr Nilsen, Oslo Thesis title: Data Management in Grid Middleware • Address limitations and implement solutions in data management and data handling in ARC middleware • Use the solutions in ATLAS data analysis • Work in collaboration with both NGIn and the Scientific Computing Group at Univ. of Oslo • Current status • Got familiar with ARC and data management (and Grid acronyms) • Survey on data management in conjunction with Grid technology presented at NorduGrid2007 • Set up a test cluster with dCache and ARC Storage Element (in collaboration with Salman Toor) www.nordugrid.org

  15. Jon Kerr Nilsen, Oslo • Near future • Storage in conjunction with virtualization • Virtualization provides possibilities to run Linux applications in Windows • Challenge to get data out of virtual machine and store them securely and efficiently • Client side of the next generation ARC storage solution • Will work with KnowARC in implementing client-side tools for accessing data in the next generation ARC • Look at data management of local versus external dCache instances in current ARC • Is there any gain in prefering dCache instances in the same network as the computing resource? www.nordugrid.org

  16. Problem: delegation is too coarse-grained in current middleware, e.g. Users delegate all rights (Impersonation) Limited proxy certificates (exclude shell access) Delegating the least set of rights is desirable Work so far on Analysing ARC components for types of operations and objects for rights modeling Analysing SAML for extending it to support fine-grained rights expressions Evaluation of existing solutions with applicable features, e.g. SecPAL Mika Silander, Helsinki Thesis title: Fine-grained rights delegation in Grids www.nordugrid.org

  17. Potential solution candidates SAML + XACML Pros: other Grid mw committed to these standards Cons: complexity in policy definition and evaluation, fine-grained rights not modeled Security Policy Assertion Language (SecPAL) Pros: broad set of security functionality including rights restrictions, policies expressed with easily understandable syntax Cons: restrictive license and implementation languages Mika Silander, Helsinki www.nordugrid.org

  18. Near future plans • Near future plans • Explore possibility of combining pros of the two former for ARC: • Easily understandable syntax similar to SecPAL, but, • Implementation relying on C/C++ and adhering to pure SAML & XACML • Ensure this is compatible with the security architecture of ARC v1.x being developed • Evaluate SAML + XACML solution of gLite • Applicable, parts reusable? www.nordugrid.org

  19. Salman Toor, Uppsala Thesis title: Data indexing and data movement in grid environments • Topic: Management of scientific data in Grid systems • Making a Grid Storage Resource Manager • A layer on top of various Storage Managers (dCache, DPM, Castor, GPFS...) • A complete cost based estimation system to find the best SM amongst the SMs registered www.nordugrid.org

  20. Salman Toor, Uppsala • Progress • Deployed dCache test site in Uppsala (with a pool-node in Oslo) • Started study of SRM specifications in conjunction with different Storage Managers • Deployed the full Globus toolkit and implemented test services and clients • Next step • write a service that • gather the information (owner, permissions etc) of data stored in dCache • replicate the data in a different Storage Manager www.nordugrid.org

  21. Philippe Gros, Lund • Physics oriented project • become an expert user of AliEn • more Grid oriented project • work on the AliEn-ARC integration Current status: • AliEn expertise • still a basic user, which is already a first step(submitting jobs on the Grid)‏ • AliEn-ARC integration • project formulated recently • studying the current state, existing simple interface Thesis title: Analysis and management of high energy physics data with NorduGrid

  22. NGIn Grid School • The NGIn training programme includes: • Grid PhD students • A Nordic Grid school • The NorduGrid 2007 Grid school was held at Niels Bohr Institute in Copenhagen 26-27 September 2007 • Organized by CSC, Finland • More than 35 present for the introductory lectures on first day • Tutorials and demos had 15 participants • Tutorials in topics from High Energy Physics to bio-informatics and medical imaging • From beginner to specialized user in two days! • Several participants spotted on the Grid later on! www.nordugrid.org

  23. NGIn School program Day 1 (General Grid Introduction)‏ Intro to Grid Intro to ARC First steps with ARC (tutorials)‏ Day 2 (Specialized tutorials)‏ HEP distributed analysis ARC in bioinformatics, ARC in medical imaging ARC-gLite interoperability Grid Job Manager Dynamic Runtime Environments Day 3 (ARC Development)‏ Sys-admin and developer training

  24. Next Grid School Next Grid School planned to be held during NorduGrid2008 in Budapest, Fall 2008 Similar program as in the first school See you there!

  25. Summary • NorduGrid is a collaboration developing and deploying the ARC middleware • NDGF provides infrastructure and deploys and contributes to the current ARC • KnowARC develops the next generation ARC • NGIn educates and trains new Grid experts, both through PhD projects and Grid schools • The PhD projects contributes to both current ARC and the next generation ARC • Next Grid school in Budapest, Fall 2008 www.nordugrid.org

  26. Positions available in NorduGrid! www.nordugrid.org

More Related