1 / 33

Introduction to OGSA-DAI

Introduction to OGSA-DAI. The OGSA-DAI Team info@ogsadai.org.uk. The OGSA-DAI Project. A generic framework for integrating data access and computation Uniform interface to relational, XML, flat file data resources Using the grid to take specific classes of computation nearer to the data

leyna
Download Presentation

Introduction to OGSA-DAI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to OGSA-DAI The OGSA-DAI Team info@ogsadai.org.uk http://www.ogsadai.org.uk

  2. The OGSA-DAI Project • A generic framework for integrating data access and computation • Uniform interface to relational, XML, flat file data resources • Using the grid to take specific classes of computation nearer to the data • Kit of parts for building tailored access and integration applications • Investigations to inform DAIS-WG • One reference implementation for DAIS • Releases publicly available NOW http://www.ogsadai.org.uk

  3. Powered by …. Project Partners Funded by the Grid Core Programme http://www.ogsadai.org.uk

  4. Programme Management Board Chair Principal Investigators Research Team Technical Review Board Chair Project Manager EPCC Team Project Membership Malcolm Norman Paul Kostas Neil Charaka Mike Mario Simon Neil Patrick Ally Andy Dave Brian Amy Charaka IBM Development Team IBM Dissemination Team http://www.ogsadai.org.uk

  5. Project Status • Current release 4.0 • Globus Toolkit 3.2 compliant • Platform and language independent • Java 1.4 • Document model • Work concentrated on data access • Wraps data resources without hiding underlying data model • Provide base for higher-level services • Distributed Query Processing (DQP) • Data federation services http://www.ogsadai.org.uk

  6. Supported Data Resources http://www.ogsadai.org.uk

  7. Service Registry Discover Publish Bind Service Consumer Service Provider Web Service Architecture http://www.ogsadai.org.uk

  8. DAISGR Discover Publish Bind Service Consumer GDSF GDS OGSA-DAI Service Architecture http://www.ogsadai.org.uk

  9. DAISGR GDS GDSF creates locates represents accesses Data Resource OGSA-DAI Services • OGSA-DAI uses three main service types • DAISGR (registry) for discovery • GDSF (factory) to represent a data resource • GDS (data service) to access a data resource • This will change http://www.ogsadai.org.uk

  10. GDSF and GDS • Grid Data Service Factory (GDSF) • Represents a data resource • Persistent service • Currently static (no dynamic GDSFs) • Cannot instantiate new services to represent other/new databases • Exposes capabilities and metadata • May register with a DAISGR • Grid Data Service (GDS) • Created by a GDSF • Generally transient service • Required to access data resource • Holds the client session http://www.ogsadai.org.uk

  11. DAISGR • DAI Service Group Registry (DAISGR) • Persistent service • Based on OGSI ServiceGroups • GDSFs may register with DAISGR • Clients access DAISGR to discover • Resources • Services (may need specific capabilities) • Support a given portType or activity http://www.ogsadai.org.uk

  12. Location Registry DAISGR findServiceData registerService Factory GDSF Analyst findServiceData • Data resource publication through registry • Data location hidden by factory • Data resource meta data available through Service Data Elements http://www.ogsadai.org.uk

  13. OGSI Container OGSI Container GDSF DAISGR Interaction Model: Start up 1. Start OGSI containers with persistent services. 2. Here GDSF represents Frog database. Frogs Database http://www.ogsadai.org.uk

  14. OGSI Container OGSI Container GDSF DAISGR Interaction Model: Registration 3. GDSF registers with DAISGR. Frogs: GSH Frogs Database http://www.ogsadai.org.uk

  15. OGSI Container OGSI Container GDSF DAISGR Interaction Model: Discovery 4. Client wants to know about frogs. Can: (i) Query the GDSF directly if known or (ii) Identify suitable GDSF through DAISGR. Frogs: GSH FindService: Frogs GSH: GDSF Mmmmm… Frogs? Frogs Database http://www.ogsadai.org.uk

  16. OGSI Container OGSI Container GDSF DAISGR Interaction Model: Service Creation 5. Having identified a suitable GDSF client asks a GDS to be created. Frogs: GSH CreateService Frogs Database GDS GSH: GDS http://www.ogsadai.org.uk

  17. OGSI Container OGSI Container GDSF DAISGR Interaction Model: Perform 6. Client interacts with GDS by sending Perform documents. 7. GDS responds with a Response document. 8. Client may terminate GDS when finished or let it die naturally. Frogs: GSH Frogs Database Perform Document GDS Response Document http://www.ogsadai.org.uk

  18. Interaction Model: Summary • Only described an access use case • Client not concerned with connection mechanism • Similar framework could accommodate service-service interactions • Discovery aspect is important • Probably requires a human • Needs adequate definition of metadata • Definitions of ontologies and vocabularies - not something that OGSA-DAI is doing … http://www.ogsadai.org.uk

  19. Deliver data back to the client. Container Container Client GDT Deliver data to a third party. GDT Deliver data another GDS. GDS Data Resource Data Resource Data Resource More Complex Behaviour GDS And there's a lot more that you can do … http://www.ogsadai.org.uk

  20. Q+U Data Flow G G G G A A A A Q Call Response Q1 G1=P S+R S S1 Q+D A U/R I U Actors - OGSI process - Non-OGSI process A - Analyst C - Consumer G - GDS P - Producer G A P P S I Q2+D C R G2=C Q+D S2 Q1+D S G1=P Q U A G S1 S Q U/R A I D Q2 C R S G2=C S2 Usage Patterns Retrieve Update/Insert Pipeline Data Q - Query D - Delivery S - Status R - Result U - Update I - Data id http://www.ogsadai.org.uk

  21. Project Using OGSA-DAI http://www.ogsadai.org.uk

  22. Projects Using OGSA-DAI Bridges (http://www.brc.dcs.gla.ac.uk/projects/bridges/) N2Grid (http://www.cs.univie.ac.at/institute/index.html?project-80=80) BioSimGrid (http://www.biosimgrid.org/) AstroGrid (http://www.astrogrid.org/) BioGrid (http://www.biogrid.jp/) GEON (http://www.geongrid.org/) OGSA-DAI (http://www.ogsadai.org.uk) eDiaMoND (http://www.ediamond.ox.ac.uk/) OGSA-WebDB (http://www.gtrc.aist.go.jp/dbgrid/) GeneGrid (http://www.qub.ac.uk/escience/projects.php#genegrid) FirstDig (http://www.epcc.ed.ac.uk/~firstdig/) myGrid (http://www.mygrid.org.uk/) INWA (http://www.epcc.ed.ac.uk/) ODD-Genes (http://www.epcc.ed.ac.uk/oddgenes/) IU RGRBench (http://www.cs.indiana.edu/~plale/projects/RGR/OGSA-DAI.html) http://www.ogsadai.org.uk

  23. Project classification • ODD-Genes • AstroGrid • Bridges • BioSimGrid Physical Sciences • BioGrid • GEON • eDiamond • myGrid Biological Sciences • GeneGrid OGSA-DAI • N2Grid • MCS • OGSA Web-DB • GridMiner • IU RGBench • FirstDig Computer Sciences • INWA Commercial Applications http://www.ogsadai.org.uk

  24. Points to Note • Feedback from users largely positive • Good suggestions • Fair criticisms • How OGSA-DAI is being used • Where it succeeds and where it fails • Helping us to capture requirements • Hope to allow user contributions • Plan to establish a policy/framework for this • Engage more with User Community • Meetings scheduled for this year • OGSA-DAI mini-workshop at AHM 2004 • OGSA-DAI tutorials at various meetings/locations http://www.ogsadai.org.uk

  25. e-Digital MammOgraphy National Database • Mammogram - X-ray of the breast • Built prototype of a national database of mammographic images • In support of the UK Breast screening programme • Employed Grid technologies to facilitate process Thanks to eDiaMonND project and the Digital Database for Screening Mammography for this image. http://www.ogsadai.org.uk

  26. Breast screening in the UK began in 1988 • Women aged 50-64 screened every 3 Years • Women aged 50-70 from 2004 • 1 View/Breast → 2 views by 2003 • UK has • Over 90 Breast screening units throughout the UK • Each one deals with about 45000 women on average p.a. • Each centre sees 5000-20000 images/year • In 2001-02 → 2002-03 • Screened: 1.4M → 1.5M • Recalled for Assessment : 77911 → 79441 • Cancers detected : 10003 → 10467 • Lives per year Saved: 300 → 1250 (by 2010) • Distributed team of doctors perform the analysis http://www.ogsadai.org.uk

  27. CHU KCL UED UCL Training Application Data Load Training App Data Load Training App Data Load Training App Data Load Training App Core API Training API Core & Training API Core & Training API Core & Training API Core & Training API Training Services Core Services Core Services Core Services Core Services Content Manager Content Manager Content Manager Content Manager DB2 DB2 DB2 DB2 OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI DB2 Federation Files Database http://www.ogsadai.org.uk

  28. eDiaMoND Findings: • OGSA-DAI provides a flexible framework • Dynamically configure the system through discovery • Activities can operate with different levels of granularity • Federation can be introduced at various levels • Good documentation on how to extend the framework • Extended Activities to access IBM DB2 Content Manager • Changes between versions broke some things • Low level XML issues http://www.ogsadai.org.uk

  29. FirstDIG • Data mining with the First Transport Group, UK • Example: “When buses are more than 10 minutes late there is an 82% chance that revenue drops by at least 10%” • "The results of this exercise will revolutionise the way we do things in the bus industry.“, Darren Unwin, Divisional Manager, First South Yorkshire. OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI Client Application Data Mining Application http://www.ogsadai.org.uk

  30. INWA • Innovation Node: Western Australia • Informing Business & Regional Policy: Grid-enabled fusion of global data and local knowledge • Project • Run from Nov 2003 - Aug 2004 • Involved 10 partners (6 UK + 4 Australia) • Aim • Data mine commercially sensitive data • Security an absolute MUST • Employ Grid technologies • Need access to data and computational resources • Demonstrator using: • OGSA-DAI • Incorporate data resources • Sun DCG's TOG (Transfer-queue Over Globus) • Handle job submission to analyse micro array data http://www.ogsadai.org.uk

  31. TOG EPCC,UK user@australia OGSA-DAI OGSA-DAI Bank data UK Property Grid Engine Grid Engine TOG Curtin,Australia Bank Bank Telco Telco user@edinburgh OGSA-DAI OGSA-DAI Telco data Australian property Data Browser Data Browser INWA http://www.ogsadai.org.uk

  32. INWA: Lessons Learned • Performing Data Integration: • TimeZone date problems • Security issues: • Bugs in • JavaCoG in GT3 • OGSA-DAI could not switch security for Grid data transfers • TOG had no security option • All of these have been fixed • Middleware not mature enough for commercial deployment http://www.ogsadai.org.uk

  33. Why OGSA-DAI? • Why use OGSA-DAI over JDBC? • Can embed additional functionality at the service end • Transformations, compressions • Third party delivery • The extensible activity framework • Avoiding unnecessary data movement • Common interface to heterogeneous data resources • Relational, XML databases, and files • Usefulness of the Registry for service discovery • Dynamic service binding process • Provision of good meta-data is necessary • Language independence at the client end • Do not need to use Java • Platform independence • Do not have to worry about connection technology, drivers, etc http://www.ogsadai.org.uk

More Related