OSG Baseline Services

1. OSG Baseline Services • In my talk I’d like to discuss two questions: • What capabilities are we aiming for in 2005/6? • How do we introduce new services into the OSG? R. GARDNER OSG Baseline Services

2. Guidance for Capabilities - take a wish list of the present • Principles and paths to deployment are guided by essential needs of the participating VO’s • Example list • Ability to store, serve, catalog, manage and discover collaboration-wide datasets on a very large scale • Ability to access opportunistically non-dedicated resources • Ability to host VO managed services and agents on gatekeeper hosts R. GARDNER OSG Baseline Services

3. …with hard lessons from the past • Protect grid services • that are vulnerable in a multi-VO environment • Managed data transfer services required • since data staging is most likely point of failure • Policy-based authorization infrastructure • to distinguish user roles within a VO • Delay binding jobs to resources to last possible moment • to optimize utilization • Robustness and reliability required • to keep operating costs low R. GARDNER OSG Baseline Services

4. …and performance targets for the (near!) future. • Submission of collections of O(1000) jobs should happen within a few seconds. It is expected that even a typical data analysis task will translate into submission of O(1000) jobs. • WMS needs to be able to keep all available resources busy. • Overall reliability must be such that task completion is generally guaranteed within less than 3 retries even for large tasks. For late 2007 this implies >95% success rate per job. R. GARDNER OSG Baseline Services

5. Example: ATLAS Production System DDMS Requirements for VO and Core Services WMS R. GARDNER OSG Baseline Services

6. ..introduces challenging distributed data management issues • A number of catalogs in play - standardize on an interface • Require traffic shaping and load balancing for I/O (SRM-dCache deployed on US Tier2 Centers) • IO & space resources utilized based on VO policies • Site-level catalogs for managed Tier2 storage elements • Reliable transfer agents used also by WMS R. GARDNER OSG Baseline Services

7. Managing Persistent Services • VO-owned and dedicated sites will allow running of “persistent” VO specific services and agents for cataloging, space management, replication, web caching, and WMS related services • Non-dedicated sites can be managed with VO services running at another site… • Or dynamically deploying VO services & agents R. GARDNER OSG Baseline Services

8. Resources Resources Shared OSG Resources Resources GRAM GIP G A T E K E E P E R S SRM GRIDFTP VO Services, Agents, Proxy Caches, Catalogs Guest VO Services, Agents, Proxy Caches Portal Partitions & Edge Services • Strategy for multiple VOs: partition resources into VO-managed and shared • Provision for hosting persistent “Guest” VO services & agents R. GARDNER OSG Baseline Services

9. Edge Services • Need to support VO specific agents and services on “leased” gateways • Typical: • Need a local MySQL database • Access to port 80 (or generally, a port range) • Will give I/O requirements in advance • Need this for a time >> job execution time R. GARDNER OSG Baseline Services

10. Example Edge Service: Scalable Remote Data Access • ATLAS reconstruction and analysis jobs require access to remote database servers at CERN, and elsewhere. • Presents additional traffic on network, long latencies for remote sites & bottleneck at central DB. • Suggest use of local mechanisms, such was web proxy caches, to minimize this impact R. GARDNER OSG Baseline Services

11. Summary of OSG 0.4 targets (end 2005) • GT4 gridftp already deployed; move to GT4 gram • Managed computing elements for multi-VO • Edge services framework providing capabilities, VO-managed late binding of job-to-resources • Job sandbox inspection, globally • Policy and trust infrastructure • Data location services and transfer agents dynamically deployed via edge services • Site catalogs, providing VO space management R. GARDNER OSG Baseline Services

12. New Services in OSG - how? • Requirements and schedule are determined with the OSG deployment activity • Architectural coherence will maintained through participation with the blueprint group • Integrate middleware services from technology providers targeted for the OSG • Provide testbed for evaluation and testing of new services and applications • Test and exercise installation and distribution methods • Provide feedback to service providers and VO application developers • Prepare release candidates for provisioning. R. GARDNER OSG Baseline Services

13. Service Readiness and Integration Plans • Service proponents come to the integration testbed with an appropriately scoped functionality and integration plan • Purpose, scope • Service Description • Packaging Description • Dependencies: resources and services needed • Test use cases identified • Testing tools – clients, harness; metrics for success clearly defined • Effort to contribute to the OSG-IVC and schedule • Links to appropriate documentation, WSDL, etc R. GARDNER OSG Baseline Services

14. Path for New Services in OSG OSG Integration Activity Readiness plan Effort Resources Readiness plan adopted VO Application Software Installation Software & packaging OSG Deployment Activity Service deployment OSG Operations-Provisioning Activity Release Candidate Application validation Middleware Interoperability Functionality & Scalability Tests feedback Metrics & Certification Release Description R. GARDNER OSG Baseline Services

15. OSG Integration Testbed Layout OSG Integration Testbed VO contributed Service platform Stable Production Release Integration Release Resources enter and leave as necessary applications, test harness, clients R. GARDNER OSG Baseline Services

16. Deployed ITB R. GARDNER OSG Baseline Services

17. Validation: CMS-MOP and ATLAS-Capone R. GARDNER OSG Baseline Services

18. Validation Jobs on ITB 0.1.5 R. GARDNER OSG Baseline Services

19. Validation: GT4 GridFTP http://osg.ivdgl.org/twiki/bin/view/Integration/GridFTP R. GARDNER OSG Baseline Services

20. Conclusions • OSG driven by VO requirements, core capabilities, and principles for guidance • Capabilities for next major release to introduce flexibility for maturing middleware and VO boxes for delegated responsibility, via Edge Services • Paths for new services, validation and release process within a contributed, consortium model is working reasonably well so far • Process is leading to reliable, core computing substrate, with VO flexibility R. GARDNER OSG Baseline Services