Ontology Assessment – Proposed Framework and Methodology

1. Ontology Assessment – Proposed Framework and Methodology

2. Goals & Objectives • Framework against which any so-called ontology can be neutrally assessed and characterized • Assessment for the purpose of • informing users about an ontology • Providing developers with methodology for comparison and improvement • Additional purpose…. • Setting minimum standards for what is/is not an ontology • Defining thresholds for formal and informal ontologies

3. Biological Classification Scheme AMS Classification Scheme NASA Thesaurus Library of Congress Subject Headings Dublin Core Metadata Scheme Organizational Chart ISO Country List Metadata repository scheme Master Data Repository Content architecture models (OO models) SCORM XML Schema for Directory Records Classification Scheme Social Network Representation Folksomony Domain Knowledge Map Visual representation of concept clusters Financial ratios Economic indicators Mathematical formula XML structured electronic journal issue WordNet Which one is an ontology? Why/not?

4. Proposed Ontology Assessment Methodology • Requires that we suspend our current terminology of things that we call ontologies and adopt a neutral mental model – convince others to do the same • Suggest that a factor analysis methodology can be applied to ontologies, just as it is applicated to knowledge economies • Factor analysis involves … • defining the essential dimensions of an ontology • defining those factors which characterize each dimension • quantifying the factors • analyzing the factors for any given application (factor analysis) or comparison • visually representing the analysis for a single ‘ontology’ and/or for comparisons of ‘ontologies’ • Let me explain how factor analysis might be used

5. Factor Analysis Methodology Methodology is currently used to calculate and visually display factors which Contribute to the development or knowledge economies.

6. Dimensionality of Ontologies • Dimensionality proposed in the framework includes: structure, expressiveness, representational granularity, intended use, automated reasoning, descriptive/prescriptive, and design methodology • Are these theoretical or practical dimensions? Do they work at an analytical level? • Other possible dimensions …. • Concepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, • Relationships – nature, type, extent, specification of relationships, logic associated with relationships • Context – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain, • Governance – control and management of the concepts, relationships and context exercised by the developer or current user

7. Representation of Ontological Assessments Another Dimensionality Framework Relationships Concepts Context Governance Methodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies.

8. Representation of Ontological Assessments Dimensionality Suggested in the Framework Paper Structure Expressiveness Intended Use Represetational Granularity Use of Automated Reasoning Descriptive vs. Prescriptive Critical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?

9. Representation of Ontological Assessments Sample assessment of a folksonomy Relationships Context Concepts Governance Methodology could be used to generate an ontological factor index for ‘ontological things’, and to rank and compare ontologies.

10. Representation of Ontological Assessments Sample assessment of a medical disease classification scheme Relationships Context Concepts Governance

11. Representation of Ontological Assessments Sample assessment of an institutional records classification scheme Relationships Context Concepts Governance

12. Defining and Quantifying Factors • For each component an orthogonal, independent set of factors must be defined • Factors must be independent of any particular pre-existing ontology (neutral) • Each factor must have a quantifiable method of representation that lends itself to ‘scoring’, analysis and comparison • Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in their coverage of values/conditions • To illustrate the idea, selected examples are presented in following slides (please note – these were ‘brainstormed’ with about 1 hour of thinking and sleep… NEEDS IMPROVEMENT AND DEFINITELY NOT RIGOROUS)

13. Concept types Data/numbers Calculation/ratios Words Grammatical fragment Logical statement Rule expression Engineering equations Degree of ambiguity Context sensitivity/insensitivity of definition Representational form Usable encoding method Availability of representational specifications (Strings vs. syntax) Degree of conceptualization/ specification Theoretical to commital What else…? Selected Examples of Concept Factors

14. Simple expressive form of relationships Grammatical Mathematical Logical Relationship behavior Membership dependence Representation or instance Equivalence Causal dependence Derivational dependence Degree of Relationship Validation/Rigor Fully Subjective Grammatical validation Mathematical validation Logical rigor/validation What else? Selected Examples of Relationship Factors

15. Knowledge Context Formal vs. informal knowledge domain Application Context System vs. human application/ consumption Managed/standardized application vs. home grown Functional context Search Mathematical or statistical analysis Logical inference Classification Dynamic clustering Metadata representation Concept indexing What else…? Selected Examples of Context Factors

16. Standards Availability Published formal vs. guidelines vs. ad hoc concepts Published formal vs. guidelines vs. ad hoc relationships Prescriptive vs. Descriptive Governance Enforcement of standards Design Guidelines Top-down (model) vs. Bottom-up (empirical) Extensibility Degree to which others can add to or extend either the concepts or the relationships Currency Degree to which the concepts and/or relationships represent our current view or knowledge of the context What else…? Selected Examples of Governance Factors

17. Factor Analysis • Factor analysis could be conducted: • At the component level on that subset of factors • At the ontology level, across all factors • Developers or users could determine what the optimal dimensionality was for their particular use • Summit members and the Ontology community could identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontology • Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development and assessment