WP2: Data Management

1. WP2: Data Management Gavin McCance University of Glasgow GridPP2 Workshop, UCL

2. EDG Summary ReportWP2: Data Management

3. The team • Based at Glasgow University • William Bell • David Cameron • Tony Doyle • Gavin McCance • Paul Millar • Caitriana Nicholson • We formed close collaboration with WP2 of European DataGrid • Became an integral part of the WP2 team • All our products were delivered jointly with EDG

4. What we set out to do? • Initial work-plan was in two key areas: • Grid Optimisation • Simulating the Grid from the data-centric viewpoint • Applying the results from the simulations to develop algorithms for optimisation of replicas • Development of optimisation software for replica management • Metadata management • Development of secure metadata services for the experiments

5. What we actually did… • Grid Optimisation • Developed a data-centric Grid simulation [OptorSim] (fully documented final release available now) • Compared of a variety of replica optimisation techniques and algorithms (many publications) • Worked on developing and refining these algorithms • Exploitation of some of the early work, we developed the Replica Optimisation Service (ROS) component of the Replica Management System • provides file access cost estimation to the user and the WP1 resource broker.

6. What we actually did… • Metadata Management • Early production of the Spitfire browser service allowing remote access to RDBMS via a web browser • Early adoption of GSI security into Spitfire to allow secure use of the service using Grid certificates • Initial adoption of web services into EDG with the creation of the web services API version of Spitfire, allowing programmatic access to remote RDMBS • The web services technology built for Spitfire was subsequently factored out and used in all other WP2 services • The security built into Spitfire was subsequently factored out and led to the creation of the WP2 Java security modules

7. What we actually did… • Replica Management Services • As focus of EDG changed to “stability, stability, stability” and production, we changed with it • Helped factor-out the web services and security frameworks from Spitfire • Moved into the core replica management services • Replica Manager • Replica Location Service • Replica Metadata Catalogue • Replica Optimisation Service • Provided continued support for these products to EDG and LCG • Led the thorough performance testing of our products for WP2’s final EU deliverable

8. What we actually did… • WP2 Integration and Testing • Significant integration expertise developed • Paul Millar is the WP2 integration team member • Glasgow ran the WP2 CVS repository • We created the testing framework for WP2 software • WP2 Testbed support • Provided support for the WP2 mini-testbed • Machines managed at ScotGrid and at CERN • WP2 Management • Gavin McCance is WP2 deputy • Major contributions to WP2 deliverables

9. What we actually did… • Standards • Contribution of GGF Data Area documents • Publications • Work was fully reported: 15 or so papers published…

10. EDG Effort Delivered • Substantial UK effort delivered to EDG : ~150% of contracted

11. Exploitation • Exploitation of the technology to deliver the next solutions to the experiments and to enable their own solutions • Stateful web services • Security • Service oriented • Focus on manageability and robustness – easily deployable • On the metadata side, emphasis upon Grid enabling the experiments own metadata solutions

12. WP2: What we learned • Software development process… • sometimes learned the hard way…! • Much feedback from our users on the software • Very useful for the design process in the next projects • The amount of effort required to make production quality services out of our software • Take all this forward into GridPP2!