1 / 28

The Open Science Grid OSG

The Open Science Grid OSG. Ruth Pordes Fermilab. What is OSG?. A Consortium of people working together to Interface Farms and Storage to a Grid and Researchers using these resources by adapting their applications to run on the Grid and Software developers providing middleware and

mufutau-kane
Download Presentation

The Open Science Grid OSG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Open Science GridOSG Ruth Pordes Fermilab

  2. What is OSG? A Consortium of people working together to Interface Farms and Storage to a Grid and Researchers using these resources by adapting their applications to run on the Grid and Software developers providing middleware and A project that provides the Operations, Support, Training and Help to make it effective.

  3. Who is OSG ? • Large global physics collaborations: US ATLAS, US CMS, LIGO, CDF, D0, STAR • Research collaborations such as Mariachi, GROW, • Grid technology groups: Condor, Globus, SRM, NMI • Many DOE Labs and DOE/NSF sponsored University IT facilities and • Partnerships with TeraGrid and EGEE, and Campus Grids such as TACC, GLOW etc.

  4. OSG Consortium

  5. When is OSG ? • Grown from of grass-roots collaboration of GriPhyN, iVDGL and PPDG participants in 2003. • 5 years of funding starting ~9/2006 from DOE SciDAC-II and NSF MPS and OCI • Deliver to US LHC and LIGO scales in 2008 and 2009: • Need to routinely distribute data at 1-5 Gbps over 10-50 sites. • Need to routinely exceed 10,000 running jobs per client • Need to reach 99% success rate for 10,000 jobs per day submission under heavy load • Active engagement effort centered at RENCI to include new sciences.

  6. The OSG Map Aug-2006

  7. OSG’s world is flat - a Grid of Grids - from Local to Global Global Science Community Systems e.g. CMS , D0 National CyberInfrastructures for Science Local Campus And Regional Grids e.g. OSG - TeraGrid e.g. FermiGrid, NWIC

  8. From the Local (Campus) Grids to the Wide Area Grid Within an organization a Local Grid provides for simplicity in sharing and efficiency in purchasing and administration. However, researchers collaborate outside the bounds of a single campus. With a uniform environment acting globally is just an extension of acting locally. Open Science Grid focuses on interoperation of the local and the wide area.

  9. A resource can be accessed by a user via the campus, community or national grid. • A user can access a resource with a campus, community or national grid identity.

  10. Example Uses • High Energy Physics • ATLAS - >15 Million proton collision events simulated at 10 minutes each • CMS - >70 Million events simulated, reconstructed and analyzed • Biology - GADU • Populates databases from search and analysis of similarities and differences among thousands of publicly available genome and protein sequences and metabolic pathways. • Gravitational Wave Physics - LIGO Data grid • Eases grid tools to ensure that 9 computing sites have a copy of the interesting data, and researchers at 36 LSC institutions use the LDG to find the data they need." • Math research • Education • Grid Summer Workshop teaches students to run jobs on OSG and TeraGrid.

  11. Running (and monitored) “OSG jobs” in 06/06.

  12. Example GADU run in 04/06

  13. Integration Testing of the System • Multi-site Integration Grid tests new OSG Releases and Configurations. • Software Readiness and Validations occur before deployment on the Integration Grid. Integration Grid Sites

  14. Taiwan UK Italy Purdue Wisconsin UCSD Caltech Florida CMS - US part of a Global Community Grid OSG EGEE CERN USA France Germany UNL MIT Data & jobs moving locally, regionally & globally within CMS grid. Transparently across grid boundaries from campus to global.

  15. How do People and Organizations Participate? • VO Registers with with Operations Center • Signs VO Agreement • User registers with VO • User added to VOMS of one or more VOs. • VO responsible for users to sign AUP. • VO responsible for VOMS service support. • Sites Register with the Operations Center • Signs the Service Agreement. • Agree on which VOs to support (striving for default admit) VOs and Sites provide Support Center Contact and join Operations groups

  16. Community Documentation..

  17. OSG Principles • Systems are Virtual Organization scoped -- groups of users working together with a shared, common environment. • Sites maintain control, authority and management of use of their Grid accessible resources. • Opportunistically available resources are beneficial. Priorities are governed by Policies (role based within a VO). • The distributed system is heterogeneous. Information must be provided to allow applications to know which resources they can successfully use.

  18. NSF Middleware Initiative (NMI): Condor, Globus, Myproxy VO software stacks that Rely on the Virtual Data Toolkit … LHCServices & Framework BioServices &Framework OSG VO Framework LIGOData Grid Applications OSG Release Cache: VDT + Configuration, Validation, VO management Virtual Data Toolkit (VDT) Common Services NMI + VOMS, CEMon (common EGEE components), MonaLisa, Clarens, AuthZ Infrastructure

  19. What is the VDT? • A collection of software • Grid software (Condor, Globus and lots more) • Virtual Data System (Origin of the name “VDT”) • Utilities • An easy installation • Goal: Push a button, everything just works • Two methods: • Pacman: installs and configures it all • RPM: installs some of the software, no configuration • A support infrastructure

  20. Who uses the VDT? • Open Science Grid • LIGO Data Grid • LCG • LHC Computing Grid, from CERN • EGEE • Enabling Grids for E-Science

  21. What software is in the VDT? • Job Management • Condor (including Condor-G & Condor-C) • Globus GRAM • Data Management • GridFTP (data transfer) • RLS (replication location) • DRM (storage management) • Globus RFT • Information Services • Globus MDS • GLUE schema & providers • Security • VOMS (VO membership) • GUMS (local authorization) • mkgridmap (local authorization) • MyProxy (proxy management) • GSI SSH • CA CRL updater • Monitoring • MonaLISA • gLite CEMon • Accounting • OSG Gratia

  22. What software is in the VDT? • Client tools • Virtual Data System • SRM clients (V1 and V2) • UberFTP (GridFTP client) • Developer Tools • PyGlobus • PyGridWare • Testing • NMI Build & Test • VDT Tests • Support • Apache • Tomcat • MySQL (with MyODBC) • Non-standard Perl modules • Wget • Squid • Logrotate • Configuration Scripts • And More!

  23. Due diligence to Security • Risk assessment, planning, Service auditing and checking • Incident response, Awareness and Training, Configuration management, • User access Authentication and Revocation, Auditing and analysis.End to end trust in quality of code executed on remote CPU -signatures? • Identity and Authorization: Extended X509 Certificates • OSG is a founding member of the US TAGPMA. • DOEGrids provides script utilities for bulk requests of Host certs, CRL checking etc. • VOMS extended attributes and infrastructure for Role Based Access Controls.

  24. Operations Model Real support organizations often play multiple roles Lines represent communication paths and, in our model, agreements. We have not progressed very far with agreements yet. Gray shading indicates that OSG Operations composed of effort from all the support centers

  25. The OSG VO • A VO for individual researchers and users. • Managed by the OSG itself. • Learn how to use the Grid!

  26. In Summary… A production grid is the product of a complex interplay of many forces: • Resource providers • Users • Software providers • Hardware trends • Commercial offerings • Funding agencies • Culture of all parties involved • …

  27. Where do you learn more? • www.opensciencegrid.org • http://vdt.cs.wisc.edu • http://osg.ivdgl.org/twiki/bin/view

More Related