OntoGrid Semantic Grid Tutorial Manchester, February 2007

1. Semantic Grid.Introduction www.ontogrid.eu Oscar Corcho University of Manchester OntoGrid Semantic Grid Tutorial Manchester, February 2007

2. Outline • Background • The Grid and its characteristics • Open Grid Services Architecture-OGSA • Grid Standardization Activities • Semantic Grid • OntoGrid and Semantic-OGSA (S-OGSA) • The S-OGSA model • S-OGSA capabilities and mechanisms • Lifetime specification • S-OGSA scenarios of use • Conclusions • Agenda Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

3. What is a Grid? R2AD License Webserver Printer Database A grid is a system consisting of • Distributed but connected resources and • Software and/or hardware that provides and manages logically seamless access to those resources to meet desired objectives • Infrastructure that will enable “coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations” Handheld Server Supercomputer Workstation Cluster Data Center Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Adapted from Hiro Kishimoto (GGF17 opening keynote)

4. Virtual Organizations • Dynamic confederations organized around common goals • Diverse membership & capabilities • People, compute resources, data resources, etc. • Diverse geographic distribution • Sharing is well-controlled • Minimumknowledge about physical characteristics of resources • Construction of higher level capabilities via composition of existing ones similar to SOA ° Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 From http://www.globus.org

5. Grid & Related Paradigms • Cluster • Tightly coupled • Homogeneous • Cooperative working • Distributed Computing • Loosely coupled • Heterogeneous • Single Administration • Grid Computing • Large scale • Cross-organizational • Geographical distribution • Distributed Management • Utility Computing • Computing “services” • No knowledge of provider • Enabled by grid technology Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Source: Hiro Kishimoto (GGF17 opening keynote)

6. Open Grid Service Architecture - OGSA queryProperties Total capacity Get/ set Properties create Used space destroy Available space rewind stop WS-RF WS-Management WS-I+ WS-GAF Service Resource • Cross cutting requirements • Interoperable • VO level • Optimized • Reliable • Certain QoS Guarantee • Scalable • Available • Extensible • Characteristics • Service Orientation • Management operations • Resource Representation/ State • Lifetime Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

7. The Open Grid Services Architecture • An open, service-oriented architecture (SOA) • Resources as first-class entities • Dynamic service/resource creation and destruction • Built on a Web services infrastructure • Resource virtualization at the core • Build grids from small number of standards-based components • Replaceable, coarse-grained • e.g. brokers • Customizable • Support for dynamic, domain-specific content… • …within the same standardized framework Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Hiro Kishimoto: Keynote GGF17

8. Grid Reality • Requires experts to install, configure and maintain • Not near the ambitious OGSA landscape in terms of cross-cut requirements • Heavy use of XML “Virtual Homogeneity” OGSA - Vision Realizations . . . . . . . . Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

9. “To realise the Next Generation Grid requires semantically rich information representation, the exploitation of knowledge, and co-ordination and orchestration that is aware of context and task” David Snelling, NextGRID, Fujitsu, OGF Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Source: Carole Goble

10. Semantic Grid. Motivation (II) Malcolm Atkinson UK eScience envoy Organisations that manage large datasets have to find agreements on what terms mean Data versus metadata: we need bindings between the data and the data structure Amarnath Gupta San Diego Supercomputing Centre Well-typed workflows can be annotated with semantic types Kepler can use keyword-based or ontology-based search Data, metadata an ontology (NSF report) Provenance in Taverna is stored in RDF and OWL Workflow reuse Stuart Owen myGrid Making this change in the code would change the [implicit] semantics of this Globus service Lisa Childers Globus Toolkit Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

11. Its called metadata. Or vocabularies. Or glossaries. It’s the state properties of a resource. Its in information services. And registries and catalogues. And configuration files. And policy definitions. And service level agreements. And file names. And file headers. And directory naming conventions And code libraries. And type systems. And schemas. And applications. And data formats. And best practice. And documentation. And workflows. And notification events And monitoring logs And embedded in XML tags … And even ontologies! And protocols. And decision procedures. Don’t we have Semantics in the Grid already? Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

12. Embedding and implicit meaning is the enemy of shareability and reuse in an open and decoupled and collaborative environment. Machine processable descriptions are machine actionable descriptions

13. In summary SWRL OWL RDF(S) RDF XML Expressive models Inference Model fusion Controlled vocabularies Data fusion Integration Integration Extensible metadata schemas that you don’t have to nail down Annotation Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Source: Carole Goble

14. Metadata Matters • Particularly for the following activities: • Provenance • Systems Configuration • Policy representation and reconciliation • Resource discovery • Using: • Annotation • Open, flexible and extensible self describing schemas • “Let’s describe my data set, or the output format of this tool, that changes all the time” • Lightweight, referable, non-flat schemas • Decoupled, interoperable systems, which resist to syntactic changes • Global naming schemes • Data integration • Resource models • Policy models • Reasoning • Complex problems Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

15. The Semantic Grid “The Semantic Grid is an extension of the current Grid in which information and services are given well-defined and explicitly represented meaning, so that it can be shared and used by humans and machines, better enabling computers and people to work in cooperation” D. De Roure, et. al Semantics in and on the Grid • Web Sites • www.semanticgrid.org • Setting up the www.semanticgridcafe.org • GGF Semantic Grid Research Group (SEM-RG) • Mailing List: sem-grd@gridforum.org Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

16. Semantic Grid trajectory CombeChem Demonstration Phase Efforts Systematic Investigation Phase Specific experiments Part of the Architecture Dagstuhl Seminar Grid Resource Ontology Semantic Grid workshops Pioneering Phase Ad-hoc experiments, early pioneers SRB GGF Semantic Grid Research Group Many workshops Implicit Semantics OGSA generation Implicit Semantics 1st generation Time Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

17. From the pioneering phase to the systematic investigation phase • In the pioneering phase... • Ontologies and their associated technologies are not completely integrated in the Grid applications • They are used as in Semantic Web applications • But there are distinctive features of Grid applications • Distribution of resources • Scale • Resource management and state • ... (non exhaustive and non compulsory list) • In the systematic investigation phase • We have to take these features into account • And incorporate semantics as another Grid resource • Our proposal is: S-OGSA Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

18. Decision making Knowledge Discovery Lots Ontology building Workflow discovery and design VO mgt Configuration Semantics Resource discovery & brokering Information linking Flexible & extensible metadata schemas Provenance Not much General language annotation Not much Lots Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Grid

19. Outline • Background • The Grid and its characteristics • Open Grid Services Architecture-OGSA • Grid Standardization Activities • Semantic Grid • OntoGrid and Semantic-OGSA (S-OGSA) • The S-OGSA model • S-OGSA capabilities and mechanisms • Lifetime specification • S-OGSA scenarios of use • Conclusions • Agenda Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

20. EU-STREP Project OntoGrid • Middleware for the Semantic Grid • Metadata Storage & Querying • Ontology Access • Annotation • Data and provenance • Services • Business Process Monitoring • Negotiation • Coordination • SEMANTIC OGSA • Capabilites & Behaviors for Semantic Grids • Principled way of realization • Applications • Insurance Settlement • Satellite Image Quality Analysis Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

21. S-OGSA Design Principles • Conceptual: reference architecture that can be applied to any grounding (WSRF, WS-Man, WS-I+, etc.) • Parsimony: Architecture as lightweight as possible: minimise the impact on tooling, not dictate content • Extensibility: Extensible and customisable as opposed to complete and generic architecture • Diversity : Mixed ecosystem of Grid and Semantic Grid services. Semantics Ignorant, Semantics aware but incapable, Semantics aware and capable • Uniformity: Everything is OGSA compliant. Our services are Grid services, knowledge and Metadata are Grid Resources. • Multiform-Multiplicity: Any resource can have multiple descriptions and any description can be in different formalisms • Enlightenment: Straightforward migration path Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

22. S-OGSA • Semantic-OGSA (S-OGSA) is... • Our proposed Semantic Grid reference architecture • Alow-impact extension of OGSA • Mixed ecosystem of Grid and Semantic Grid services • Services ignorant of semantics • Services aware of semantics but unable to process them • Services aware of semantics and able to process (part of) them • Everything is OGSA compliant • Defined by • Information model • New entities • Capabilites • New functionalities • Mechanisms • How it is delivered Model provide/ consume expose Capabilities Mechanisms use Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

23. S-OGSA Model. Semantic Bindings Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

24. S-OGSA Model Example METADATAas Semantic Annotations Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

25. From OGSA to the S-OGSA Semantic Provisioning Services Semantic binding Knowledge Metadata Ontology Annotation Reasoning Application 1 Application N Security Optimization Data OGSA Execution Management Semantic-OGSA Semantic Provisioning Services Resource management Information Management Infrastructure Services Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

26. S-OGSA Model and Capabilities. The complete picture Is-a WebMDS Annotation Service Metadata Service Ontology Service OGSA-DAI Grid Service Semantic BindingProvisioning Service Knowledge Service Is-a Reasoning Service Is-a CAS Is-a Is-a Is-a Knowledge Entity Semantic ProvisioningService Grid Entity 1..m 1..m SAMLfile uses Ontology Is-a Is-a Semantic aware Grid Service Knowledge Resource Grid Resource DFDL file Rule set 1..m 1..m produce consume JSDL file 0..m 0..m Semantic Binding Model 0..m 0..m Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms Is-a Knowledge Semantic Grid Grid

27. OntoKit: An implementation of S-OGSA Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

28. OntoKit: An implementation of S-OGSA Semantically Aware OntologyRole-basedAuthZ Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

29. S-OGSA Patterns. Semantic-ignorant service Ontology Service Metadata Service Refers to Access/Query Metadata Properties Lifetime Metadata Seeking Client Resource props Resource Others…. Service Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

30. S-OGSA Patterns. Semantic Aware but Incapable Service Ontology Service Metadata Service Access/Query Semantic Bindings Refers to 2 Properties Lifetime 1 Metadata Seeking Client Get Semantic Binding Pointers Resource properties Resource Service Others… Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

31. S-OGSA Patterns. Semantic Aware and Capable Service Ontology Service Metadata Service Farm out request 1.1 Properties Lifetime 1 Metadata Seeking Client Access/Query Semantic Bindings Semantics Resource Service Others… Semantic aware interface Model Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007 Capabilities Mechanisms

32. S-OGSA Grounding. Grid Ontology and S-OGSA Ontology • Grid Ontology • Common set of ontologies to describe Grid entities (resources and services) • Based on work from UniGrids • Effort to be continued by OntoGrid • Available in OntoGrid’s CVS Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

33. S-OGSA Metadata Access/Management Protocols Semantic Binding Service Suite create SB Factory create WS-Addressing: epr SB Semantic Binding SB WS-RP: Get/Set/Query Properties query Client SB WS-Notif: Subscribe / Notify RDF Inspect-props . . . WS-RL: Destroy , SetTerminationTime WS-RL ++: archive Query w/o Inference, UpdateContent query Query( over unified view) Metadata Query Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

34. Semantic Binding Service. Lifetime Specification • What happens if... • ...any or all of the Grid entities it refers to disappears? • Instrument and planning files for satellites do not disappear • Insurance contracts, cars, repair companies, etc., may disappear • ...the Knowledge entities disappear or evolve? • Ontologies may change • ... a SB is no longer available (its content is not useful any more)? • Damage claims: add witness reports, improve info about location, create new hypothesis... • When do/shouldSBs become invalid? How often should this be checked? • What is the status of the content of a SB (e.g., content checked, stable, unchecked, etc.)? • Is a SB always active or can it be archived after a period of time? • Satellite data that is not used after some time Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

35. Semantic Binding Service. WS-SBResourceLifetime Stable Stale GE changed KE changed Archived Deleted • Lifetime specification based on WS-ResourceLifetime • Extension with • Resource properties (state) • Updates • Archive • Notifications Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

36. WS-SBResourceLifetime vs WS-ResourceLifetime • Basic Operations • createSemanticBinding (Factory) • addGridEntityReference/removeGridEntityReference • addKnowledgeEntityReference/removeKnowledgeEntityReference • getContent • updateSBContent • query • queryWithInference • WS-SBResourceLifetime • archive • setUpdateTime • WS-ResourceLifetime • setTerminationTime • destroy Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

37. Notifications Client Client Client WS-Notif. [state] subscribe Query-RP [state] Semantic Binding Service subscribe subscribe WS-Notif [lastModificationTime] Knowledge Entity Grid Entity Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

38. Outline • Background • The Grid and its characteristics • Open Grid Services Architecture-OGSA • Grid Standardization Activities • Semantic Grid • OntoGrid and Semantic-OGSA (S-OGSA) • The S-OGSA model • S-OGSA capabilities and mechanisms • Lifetime specification • S-OGSA scenarios of use • Conclusions • Agenda Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

39. Satellite Use Case: Technical issues Space Segment Ground Segment SATELLITE FILES: DMOP files Product files Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

40. Satellite Use Case: Technical issues • Comparison between planning and product generation: DMOP_File#(n+1) StartTime DMOP#(n+1)_ File (StopTime) DMOP_File#n(StartTime) DMOP_File#n(StopTime) Instr#1 planning ... DMOP_er (ORBIT_NUMBER, ELAPSED_TIME) Instr#n (RA_2) planning DURATION ... Instr#n(RA_2) Product Generation ... ... PRODUCT_data_gap PRODUCT_FILE Start_time (SENSING_START) PRODUCT_FILE Stop_time (SENSING_STOP) RA2_CAL_1P Start_time (SENSING_START) RA2_CAL_1P Stop_time (SENSING_STOP) Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

41. Satellite Use Case: Deimos Integrated Prototype Satellite File 2 WebDAV client e.g. MS Windows Explorer WebDAV HTTP PUT Annotate file 3 1 Copy satellite file Convert time to canonical representation 5 4 UTC2Seconds Soaplab Obtain ontology 6 Type metadata Convert time to canonical representation Store 7 Input criteria 1 2 QUARC-SG client JSP 3 WS-DAIOnt-RDF(S) Metadata Service Query SatelliteDomain Ontology Metadata for annotation Metadata generation process Metadata querying process Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

42. Satellite Use Case: Technical issues • Namefile (Product): RA2_MW__1PNPDK20060201_120535_000000062044_00424_20518_0349.N1" Corresponds to: Satellite files: PRODUCT filename Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

43. Satellite Use Case: Technical issues Satellite files: Annotated PRODUCT FILE [...] <rdf:Description rdf:about="http://protege.stanford.edu/kb#10605"> <rdf:type rdf:resource="http://protege.stanford.edu/kb#Product_file"/> <NS0:file_name>&quot;RA2_MW__1PNPDK20060201_120535_000044792044_00424_20518_0334.N1&quot;</NS0:file_name> <NS0:sensing_start rdf:datatype="http://www.w3.org/2001/XMLSchema#int">192110735</NS0:sensing_start> <NS0:sensing_stop rdf:datatype="http://www.w3.org/2001/XMLSchema#int">192115215</NS0:sensing_stop> <NS0:specific_product_header rdf:resource="http://protege.stanford.edu/kb#10608"/> </rdf:Description> <rdf:Description rdf:about='http://protege.stanford.edu/kb#10649'> <NS0:file_name>&quot;RA2_MW__1PNPDK20060202_160340_000058672044_00441_20535_0344.N1&quot;</NS0:file_name> <NS0:sensing_start rdf:datatype='http://www.w3.org/2001/XMLSchema#int'>192211420</NS0:sensing_start> <NS0:sensing_stop rdf:datatype='http://www.w3.org/2001/XMLSchema#int'>192217287</NS0:sensing_stop> <NS0:specific_product_header rdf:resource='http://protege.stanford.edu/kb#10652'/> </rdf:Description> [...] Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

44. Satellite Use Case: Technical issues RECORD ID RECORD parameters RECORD parameters corresponding to other RECORD structure. Satellite files: • DMOP (PLANNING) FILES FILE ; DMOP (generated by FOS Mission Planning System) RECORD fhr FILENAME="DMOP_SOF__VFOS20060124_103709_00000000_00001215_20060131_014048_20060202_035846.N1" DESTINATION="PDCC" PHASE_START=2 CYCLE_START=44 REL_START_ORBIT=404 ABS_START_ORBIT=20498 ENDRECORD fhr ................................ RECORD dmop_er RECORD dmop_er_gen_part RECORD gen_event_params EVENT_TYPE=RA2_MEA EVENT_ID="RA2_MEA_00000000002063" NB_EVENT_PR1=1 NB_EVENT_PR3=0 ORBIT_NUMBER=20521 ELAPSED_TIME=623635 DURATION=41627862 ENDRECORD gen_event_params ENDRECORD dmop_er ENDLIST all_dmop_er ENDFILE Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

45. Satellite Use Case: Technical issues Satellite Ontology (General view) Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

46. Satellite Use Case: Technical issues • Timeline Planning-Product Generation: Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

47. Insurance Grid • Business values: • Value (cost reduction, billing) • Time to market / speed of implementation • Ahead of competitors • Fit within (human and technical) organization • Innovation drive • Solve existing problems: • Making processes more efficient with a new approach • (more) Reliable / Accepted • Proven / Cheaper • -> CarRepairGid • Solve problems that could not be solved before: • Lack of trust/ Unfamiliar • Politics • Technical / organizational limitations • -> CarFraudGrid Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

48. Business Case 1: Car Repair Business Case • Context: • Repair damaged cars • Negotiation between insurance and repair company • Speed, Price, Quality • Method of repair, Selection of material,Paint, Coalition • Now: • negotiation by hand • long term (yearly) • Challenge: • Automated negotiation • short term (every claim) • Include SLA Metadata for data integration Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

49. S-OGSA Scenario. Insurance settlement Repair CO. 2 (Nego. Srvc.Contractor) Repair CO. 3 (Nego. Srvc.Contractor) Repair CO. 1 (Nego. Srvc.Contractor) Negotiation client 2 Cfp Job 4 Refuse 1 Do Negotiation calculatePrice 3 Job + Contractor List accept 5 Cfp Job 2 Retrieve public Job desc. Negotitation Service (Manager) propose Offer 4 calculatePrice 3 Reject 5 Job Cfp 2 WS-DAIOnt RDF RDF Offer RDF propose calculatePrice 4 3 InsurranceCo DB WS-DAIOnt Atlas RDF RDF RDF RDF RDF Legacydatabases Car Parts Legacydatabases Motor Vahicles Car Repair DB Car Repair DB Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007

50. Business Case 2:CarFraudGrid • Situation: • A lot of tricks to get money from insurance companies • Now: • Ad hoc manual techniques • Only pattern search on local or national scale • Most tricks found on accident • Challenge: • Automated fraud detection Metadata for data integration Metadata for reasoning Oscar Corcho. OntoGrid Semantic Grid tutorial. February 2007