The Generalized Ontology Evaluation Framework & the I-Choose Use case

1. The Generalized Ontology Evaluation Framework & the I-Choose Use case Ontology Summit Hackathon – April 6th, 2013

2. Outline • GOEF- Evaluation Methodology • I-Choose & Sustainability Certification • I-Choose: Use Case & Ontology

3. Evaluating ontologies • Ontologies are composed of intrinsic and extrinsic qualities. • Intrinsic qualities can be decontextualized and evaluated for syntactical accuracy.

4. Evaluating ontologies • But how to evaluate ontologies according to its particular context, or use case that evaluates both intrinsic and extrinsic? • We propose a: Generalized Ontology Evaluation Framework (GOEF)

5. GOEF Approach The use case contains important contextual information needed for the evaluation. This must be extracted out from the use case, i.e. and separate what is being evaluated from the evaluation metrics, then apply the evaluations. Two stages: • (1) Recast use case into its contextual components: • (a) Functional objective • (b) Standards requirements (Design) specification • (c) Semantic components required to meet the above • (2) Evaluate components using objective metrics • Create new or use existing evaluation methods at the appropriate levels to meet 1c, 1b and 1a. 5

6. Three Levels of Evaluation 6

7. Function Level • Represents the top level of the use case. • i.e. the function of the intended use (for search, for integration, for certification) • Additionally, the primary characteristics that define the classification of the domain of the ontology (organism, aircraft, assessment instrument.). 7

8. Standard Level • Represents the quality or standard that has to be met by the application (e.g. for legal, interoperability, function, compliance, etc.) • Further specifies the domain characteristics. 8

9. Component Level Identifies ontology fragments needed to fulfill the function and meet the standard. (e.g. wings, engine, wheels). In complex systems, these may be composite; many “standard” components reused (nuts and bolts used in doors and wheels)). 9

10. Formalizing Use Cases • Issue1: Methodology for formalizing use cases still needed. 10

11. I-Choose Vision “To create interoperable data networks across communities of producers, supply chain operators, and third party certifiers, capable of producing trusted information packages to support end consumers to make purchasing decision based on their values”

12. Rationale for I-Choose $9.34 $9.58 $8.00 $2.49 (All prices normalized to US$ per pound)

13. Use-Case 1 – Ellen goes to the supermarket Let us imagine the following scenario. A consumer, Ellen, goes to the supermarket and is looking for a particular coffee. She is concerned with the production process of the coffee and notices that some coffees have some types of certifications, such as organic or fair trade. She finds a particular coffee which is certified by Fairtrade International, a third-party certifier. She is interested in retrieving particular information about the certification process of this certifier. She is interested in the criteria that were used in the evaluation of that coffee, and specifically if “child labor” was used in that production process. If the certifier which gave the coffee the particular certification inspected for child labor then that information would be recorded (somehow). The goal of the I-Choose ontology is to enable a structured way to record this information, and allow easy exchange and retrieval of it.

14. Certification Criteria – Information to be represented in ontology

15. I-Choose Ontology

16. GOEF – 1st stage How to evaluate this ontology? ** GOEF ** (1) Recast components into its parts

17. Potential recast of use case components Function: Enable retrieval of specific compliance criteria from a specific inspection process of a particular product. Design objective: Initial system: Satisfy consensus user criteria pre-determined by survey research [we will initially use synthetic criteria] Semantic components: Compliance Criteria a) Children Below 15 b) Under 18 dangerous work c) Preventive measures ensure safety Certification Body a) Flo-Cert Standard Setter Organization a) FairTrade International b) USDA Organic c) ISO 65 Product a) Coffee 17

18. GOEF – 2nd stage ** GOEF ** (2) Evaluate components using objective metrics Identify evaluation metrics needed (consistency, coverage, etc.) some exist and can be plugged in, others may need to be developed.

19. Evaluation Metrics Issue2: Objective metrics need also to be formalized. 19

20. Potential evaluation metrics Correctness: • General logical/syntactical validation • Are the right terms used (compliance criteria vs. code of conduct vs. standards) • Match information provided in the ontology to general accepted definition provided by ILO (e.g. age of 15 is accepted standard) Completeness: • All child work criteria, and necessary characteristics included • Ability of ontology to distinguish compliant vs. non-compliant criteria Utility: • Consumer/Consumer Advocate Questions Satisfied 20

21. Use Case Management in GOEF Add a New Use Case Or, Select a Pre Registered Use Case About “Child Labor Compliance Criteria 3.3.7 An ontology representing the FLO-CERT compliance criteria. Flo-Cert Small Farmers Organization (SPO) Function [Details] Standard [Details] Components [Details]

22. GOEF & SADI Service Issue3: Could (or how would) GOEF be incorporated into SADI Service OR Could GOEF be more formalized by using the approach of SADI Service?

23. BACK-UP slides from here on…

24. Description of I-Choose: • I-Choose is transnational project in CTG funded by NSF Interop program and CONACYT in Mexico • To build data architecture to support ethical consumption • Focus on sustainable coffee products produce in Mexico and consume and distribute in Canada and in the US. • Team of researchers with mixed skills and expertise • Components of I-Choose System: • A set of data standards to share information across sustainable supply-chain • A governance system

25. Is the information in the packaging trustworthy? What are the criteria? How do I know this is not a “Green Washing”? Which certification should I trust? • There are currently 432 eco-labels worldwide • The label available in 246 countries • (http://www.ecolabelindex.com/)

26. Motivating Example: i-ChooseUse Case 26

27. Envision Application of I-Choose

28. Overview of Sustainability Certification Schemes A certification system generally consists of a standard setter and a certification body. • E.g. Fairtrade International (FLO) : creates standards and manages the labeling initiative • E.g. Flo-cert : internal certification body but independent to FLO; interprets the standard into verifiable control points called compliance criteria

29. Certification Mark/Label • An applicant wishing to receive the Fairtrade (FLO) certification will be evaluated against a list of compliance criteria by an inspector/auditor appointed by Flo-cert • The result from the audit/inspection will be used for the ultimate certification decision which, in this case, will grant the particular Fairtrade International certification mark/label to be attached to the product

30. Overview ofCertification System Figure 1. General Overview of Certification System. Ontology work to be focused on data obtained in compliance evaluation

31. What is an elephant, exactly? 31

32. Existing Semantic Web Design Methodology

33. Data Provenance

34. Data Provenance Data provenance enables exposure of relations between consumer products and their supply chains to consumer advocates. From an encoding perspective, there are two requirements for provenance records: • Lineage for the consumer product • Semantics for individual members of the lineage

35. Hackathon Clinics Information • HC-02. The General Ontology Evaluation Framework (GOEF) & the I-Choose Use Case primary project contact: JoanneLuciano  (jluciano@cs.rpi.edu) James Michaelis (michaelis@cs.rpi.edu) Nicolau dePaula (ndepaula@albany.edu) • Activities will be in session:   Offline Meeting: • Sat 2013.04.06 - 11:00am – 05:00pm EDT • Location: Winslow 2nd Floor, RPI   Open webcast segment: • Sat 2013.04.06 - 11:00am – 12:45pm EDT • Project chat-room : http://webconf.soaphub.org/conf/room/hc-02

36. Acknowledgement:

37. Presenters: Nicolau dePaula is a PhD student in Information Science at the University at Albany. His interests are in e-government, knowledge management, and information policy. James Michaelis is a research assistant in RPI's Tetherless World Constellation Lab, working under Professors Deborah McGuinness and James Hendler.  His research is centered on the design of explanation interfaces for intelligence gathering systems. Additionally, he has conducted prior work on techniques for logging processing activities within Semantic Web enabled systems. Djoko Sigit Sayogo is PhD student at Rockefeller College of Public Administration and Policy and graduate assistant at the Center for Technology in Government, the University at Albany, SUNY. He is currently involved in the I-Choose research project at the Center for Technology in Government. His research interests include e-government, collaborative network, and data sharing. Joanne S. Luciano - Research Associate Professor, TWC,Rensselaer Polytechnic Institute