HPDC12 Seattle Structured Data and the Grid Access and Integration Prof. Malcolm Atkinson Director

1. HPDC12 Seattle Structured Data and the GridAccess and Integration Prof. Malcolm Atkinson Director www.nesc.ac.uk 23rd June 2003

2. Outline • What is e-Science? • Structured Data at its Foundation • Key Uses of Distributed Data Resources • Data-intensive Challenges • Data Access & Integration • DAIS-WG • OGSA-DAI: Progress and Dreams • Unanswered Architectural Questions

3. computers software Grid instruments Shared data archives colleagues Foundation for e-Science • e-Science methodologies will rapidly transformscience, engineering, medicine and business • Driven by exponential growth (×1000/decade) • Enabling and requiring a whole-system approach sensor nets

4. Theory:Models & Simulations→ Shared Data Experiment:Advanced Data Collection→Shared Data Requires Much Engineering, Much Innovation Computing Science:Systems, Notations &Formal Foundation → Process & Trust Changes Culture, New Mores, New Behaviours Three-way Alliance Multi-national, Multi-discipline, Computer-enabled Consortia, Cultures & Societies New Opportunities, New Results, New Rewards

5. Curated& SharedDatabases Carries knowledge Carries knowledge SimulationCommunities Discoveries Database-mediated Communication ExperimentationCommunities Data Analysis &TheoryCommunities Data knowledge

6. in-flight data global network eg SITA ground station airline DS&S Engine Health Center internet, e-mail, pager data centre maintenance centre global in-flight engine diagnostics 100,000 engines 2-5 Gbytes/flight 5 flights/day = 2.5 petabytes/day Distributed Aircraft Maintenance Environment: Universities of Leeds, Oxford, Sheffield &York

7. Database Growth PDB Content Growth Bases 39,856,567,747

8. Distributed Structured Data • Key to Integration of Scientific Methods • Key to Large-scale Collaboration • Growing Number of Growing Data Resources • Independently managed • Geographically distributed • Key to Discovery • Extracting nuggets from multiple sources • Combing them using sophisticated models • Analysis on scales required by statistics • Repeated Processes and Decisions!

9. RAM time to move 15 minutes 1Gb WAN move time 10 hours ($1000) Disk Cost 7 disks = $5000 (SCSI) Disk Power 100 Watts Disk Weight 5.6 Kg Disk Footprint Inside machine RAM time to move 2 months 1Gb WAN move time 14 months ($1 million) Disk Cost 6800 Disks + 490 units + 32 racks = $7 million Disk Power 100 Kilowatts Disk Weight 33 Tonnes Disk Footprint 60 m2 Tera → Peta Bytes Now make it secure & reliable! May 2003 Approximately Correct See also Distributed Computing Economics Jim Gray, Microsoft Research, MSR-TR-2003-24

10. Mohammed & Mountains • Petabytes of Data cannot be moved • It stays where it is produced or curated • Hospitals, observatories, European Bioinformatics Institute, … • Distributed collaborating communities • Expertise in curation, simulation & analysis •  Can’t collocated data in a few places • Distributed & diverse data collections • Discovery depends on insights •  Unpredictable sophisticated application code • Tested by combining data from many sources

11. DynamicallyMove computation to the data • Assumption: code size << data size • Code needs to be well behaved • Develop the database philosophy for this? • Queries are dynamically re-organised & bound • Develop the storage architecture for this? • Compute closer to disk? • System on a Chip using free space in the on-disk controller • Data Cutter a step in this direction • Develop the sensor & simulation architectures for this? • Safe hosting of arbitrary computation • Proof-carrying code for data and compute intensive tasks + robust hosting environments • Provision combined storage & compute resources • Decomposition of applications • To ship behaviour-bounded sub-computations to data • Co-scheduling & co-optimisation • Data & Code (movement), Code execution • Recovery and compensation Dave Patterson Seattle SIGMOD 98

12. Data Access & Integration

13. First steps towards a generic framework forintegrating data access and computation Using the grid to take specific classes of computation nearer to the data Kit of parts for building tailored access and integration applications

14. DAIS-WG • Specification of Grid Data Services • Chairs • Norman Paton, Manchester University • Dave Pearson, Oracle • Current Spec. Draft Authors Mario Antonioletti Malcolm Atkinson Neil P Chue Hong Amy Krause Susan Malaika Gavin McCance Simon Laws James Magowan Norman W Paton Greg Riccardi

15. DBMS DB ResultSet Conceptual ModelExternal Universe External data resource manager External data resource External data set

16. DBMS DB ResultSet Conceptual ModelDAI Service Classes Data resource manager Data resource Data activity session Data request Data set

17. $5 million, 20 months, started February 2002 Additional 24 months, starts October 2003 OGSA-DAI Partners IBM USA EPCC& NeSC Glasgow Newcastle Belfast Manchester Daresbury Lab Cambridge Oxford Oracle Hinxton RAL Cardiff London IBM Hursley Southampton

18. Job Submission Brokering Workflow Structured Data Integration Registry Banking Authorisation Data Transport Resource Usage Transformation Structured Data Access Structured Data Relational XML Semi-structured - Infrastructure Architecture Data Intensive X-ology Researchers Data Intensive Applications for X-ology Research Simulation, Analysis & Integration Technology for X-ology Generic Virtual Data Access and Integration Layer OGSA OGSI: Interface to Grid Infrastructure Compute, Data & Storage Resources Distributed Virtual Integration Architecture

19. 1a. Request to Registry for sources of data about “x” SOAP/HTTP service creation API interactions Registry 1b. Registry responds with Factory handle 2a. Request to Factory for access to database Factory Client 2c. Factory returns handle of GDS to client 2b. Factory creates GridDataService to manage access 3a. Client queries GDS with XPath, SQL, etc XML / Relational database Grid Data Service 3c. Results of query returned to client as XML 3b. GDS interacts with database Data Access & Integration Services

20. SOAP/HTTP service creation API interactions ProblemSolving Environment SemanticMeta data “scientific” Application coding scientific insights Application Code Future DAI Services 1a. Request to Registry for sources of data about “x” & Data “y” Registry 1b. Registry responds with Factory handle 2a. Request to Factory for access and integration from resources Sx and Sy Data Access & Integrationmaster 2c. Factory returns handle of GDS to client 3b. Client 2b. Factory creates tells GridDataServices network analyst Client 3a. Client submits sequence of scripts each has a set of queries GDTS to GDS with XPath, SQL, etc 1 XML Analyst GDS GDTS database GDS 2 S x GDS S 3c. Sequences of result sets returned to y Relational analyst as formatted binary described in GDTS GDS GDS 2 3 a standard XML notation 1 database GDS GDTS

21. What Architecture will Enable Data & Computation Integration? • Common Conceptual Models • Common Planning & Optimisation • Common Enactment of Workflows • Common Debugging • … • What Fundamental CS is needed? • Trustworthy code & Trustworthy evaluators • Decomposition and Recomposition of Applications • … • Is there an evolutionary path?

22. Comments & Questions Please www.ogsadai.org.uk www.nesc.ac.uk

23. Opportunities Global Production of Published Data Volume Diversity Combination  Analysis  Discovery Challenges Data Huggers Meagre metadata Ease of Use Optimised integration Dependability Scientific Data • Opportunities • Specialised Indexing • New Data Organisation • New Algorithms • Varied Replication • Shared Annotation • Intensive Data & Computation • Challenges • Fundamental Principles • Approximate Matching • Multi-scale optimisation • Autonomous Change • Legacy structures • Scale and Longevity • Privacy and Mobility