Grid DAIS: Database Access and Integration Services

1. Grid DAIS:Database Access and Integration Services Greg Riccardi Florida State University riccardi@cs.fsu.edu

2. Overview of Presentation • Goals of DAIS • Conceptual model of Grid database access • Examples of client-service interactions • Discovery and creation of services • Asynchronous query processing and datasets • Updating from datasets • Representing Sky Query in DAIS • Other topics/issues

3. Goals of DAIS • The group seeks to promote standards for the development of grid database services, focusing principally on providing consistent access to existing, autonomously managed databases. • Provide service-based access to existing data management systems. • Accommodate several widely used data management paradigms (e.g., relational, object, XML) within a consistent framework. • Provide sufficient information about itself to allow the service to be used given the specification of the service and the metadata provided by the service. • Peacefully coexist with other Web and Grid Service standards. • Be orthogonal to Grid authentication and authorization mechanisms. • Support higher-level information-integration and federation services.

4. Desirable Properties of DAIS Systems • OGSI/A compliant • Letter and Spirit • Plugability/Extensibility • Different kinds of data resources • Many access mechanisms • Evolvable • Easy to understand and apply • Existing standards/designs • Tooling • GridServices and WebServices applicable • Supports current technology • Access AND integration • Integration of different models at the data level • Implementable • Integrateable into customer scenarios • Technology independent

5. The Model – External Artifacts External data resource manager External data resource External data set DBMS DB Resultset External = external to the OGSI compliant grid

6. The Model – Logical Artifacts data resource manager DBMS DB data resource data activity session data request data set Resultset

7. Data Resource Manager • External data resource manager (edrm) • A data management system such as a relational database management system or a file system • Data resource manager (drm) • A grid service that represents the external data resource manager • Binds to an existing edrm • Supports management operations such as start and stop • Mainly out of scope of DAIS. A place holder for interaction with other working groups

8. Data Resources • External Data Resource (edr) • A data construct managed by the external data resource manager, for example, a database or a directory structure. • An external data resource manager may manage many external data resources • Data Resource (dr) • A grid service that represents an external data resource • Represents the point of contact to the data structures managed by the edrm. • Exposes meta-data about the structure of the edr • Defines the the target for queries across the edr • Can act as a notification source for notifications associated with the edr • Is bound to existing or newly created edr • Has similarities with a data set. More of which later.

9. Data Sets • External Data Set (eds) • Data logically separated from an external data resource manager • Could be a snapshot (query) of a relational database or data generated by some process prior to being inserted into a database • Will be typed and identifiable • Data Set (ds) • A service wrapper for the eds • Exposes meta data about the type, description, format of the eds • Immutable • Exposes simple data access operations depending on the type of data. • getAllData, createIterator, getTuple, getFile, getByte, etc. • Can be moved while maintaining its handle and data identity • Can be copied or replicated while maintaining its data identity • Can be delivered to a data manager for persistence • Query and update could be supported

10. Putting It Together Logical Artifact = Service External world edrm create edr eds bind/ create create bind bind/ create drm create dr create das ds data request DAIS world locate requester access data

11. Exploiting The Logical Artifacts: Data Sets edr edr GSH GSH move copy dr ds ds ds dr create reference launch launch reference create reference create das das move service copy service query insert/update target details target details Analyst1

12. Client-Server Interaction Patterns Update/Insert Retrieve Pipeline 1. 4. Q + U Q 7. Q1 G = P A G A G S S + R S1 I A U/R Q2 + D U 5. 2. Q + D I P S2 G = C A Q + D G S A G R S C Q1 + D G = P 8. 6. 3. Q P U S1 A A U/R G Q Q2 S A G D C S R S2 G = C

13. Examples of client-service interactions • Discovery and creation of services

14. Examples of client-service interactions • Asynchronous query processing and datasets

15. Examples of client-service interactions • Updating from datasets

16. Example of performance estimation

17. SkyQuery Cross Match Query

18. Cross Match Estimation in DAIS

19. Cross Match in DAIS

20. Other topics and issues for DAIS • Data provenance management • Transaction management • Fault tolerance • Security, logging, auditing • Supporting many concurrent users • Establishing the identity and provenance of datasets • Creating pipelines and other workflows • Querying streams of data