Creating , Maintaining , and Integrating Understandable Knowledge Bases Richard Fikes Deborah McGuinness Sheila McIlrai

1. Creating, Maintaining,and Integrating Understandable Knowledge Bases Richard Fikes Deborah McGuinness Sheila McIlraith Jessica Jenkins Steve Wilder Kengo Ishii Gleb Frank Yulin Li Honglei Zeng Knowledge Systems Laboratory Stanford University www.ksl.stanford.edu 1/24/01

2. Forming Understandable Knowledge • Knowledge formation requiresknowledge evolution • KBs require multiple developmental steps to become useful • KSL is building tools to support KB evolution • KB diagnostics • Bugs, missing knowledge, heuristic warnings, architectural advice • KB explanation • Customized to individual users and tasks • KB merging • Consistency checking using a hybrid reasoner (JTP) • KB modularization • To produce reusable KB building blocks • Knowledge formation requires expressive KR languages • KSL is extending current representation formalisms • Defaults, KB partitioning, perspectives, …

3. Chimaera • A Knowledge Evolution Environment • Tools for KB diagnosis and merging • Available as a Web service • www-ksl-svc.stanford.edu • Usable from a Web browser • Online user manual, tutorial, and demonstration movie • Performs KB diagnostics in batch mode • Uploads and analyzes user’s KB • Accepts KBs in MELD, KIF, OKBC, RDF, XML, DAML, … • Provides results as HTML pages linked to frames and axioms • Provides user selectable set of diagnostic tests • Analyzes both the structure and content of a KB • Uses reasoners to analyze content • Currently runs 28 diagnostic tests

4. Recent Publications and Presentations • AAAI 2000 • Deborah McGuinness, Richard Fikes, James Rice, and Steve Wilder; “The Chimaera Ontology Environment”; Intelligent Systems Track, Seventeenth National Conference on Artificial Intelligence; Austin, Texas; July 30 - August 3, 2000. • ICCS 2000 • Deborah McGuinness; “Conceptual Modeling for Distributed Ontology Environments”; Eighth International Conference on Conceptual Structures: Logical, Linguistic, and Computational Issues; Darmstadt, Germany; August 14-18, 2000. • Invited Talks • Deborah McGuinness; “Ontology Environments”; • Autumn School for Cognitive Science – Freiburg, Germany, Sept., 2000 • Free University of Amsterdam (Vrij) – Sept. 2000 • Sun Microsystems – Palo Alto, CA – Dec. 2000 • National Center for Atmospheric Research – Boulder, CO, coming Feb 2001

5. Classification of Diagnostic Results • Errors • Logical inconsistencies E.g., contradictory type constraints • Content structure errors E.g., terms used but not defined • Anomalies • Missing information E.g., type constraints • Redundancies E.g., redundant superclass and type links • Extraneous structure or content E.g., terms defined but not used • Summaries E.g., counts of term references • Suggestions E.g., use consistent naming conventions

6. Diagnose Both Frames and Axioms • Examples of frame-oriented diagnostics • Local constraint contradicts inherited constraint (error) • Object an instance of disjoint classes (error) • Cyclic subclass links (anomaly) • Class with a single subclass (anomaly) • Object an instance of a non-leaf class (anomaly) • Class contains no local information (anomaly) • Include disjointness statements about class siblings (suggestion) • Examples of axiom-oriented diagnostics • Quantified variable not in the body of the axiom (anomaly) • Variable in implication antecedent but not in consequent (anomaly) • Illegal number of arguments for implication or negation (error) • Conjunction or disjunction with only one argument (anomaly) • Suggest breaking up exceptionally long axioms (suggestion)

7. Next Steps: Repair Dialogues • Provide interactive follow-up to diagnostics • Identify KB content on which diagnosis result is based • Suggest repairs or repair strategies • Guide user through repair procedure • Examples • Class is a direct subclass of “THING” • Provide direct subclasses of THING as candidate superclasses • Step down through the class hierarchy • Class has redundant superclass links • Present the redundant links • Suggest removal of link(s) to most general classes • Type, cardinality, or bounds conflict • Present conflicting constraints • Suggest changing local conflicting constraint(s) • Missing information • Initiate acquisition dialogues for missing information

8. Next Steps: Acquisition Dialogues • Chimaera notes missing information about “parent” of “Person” • User requests that Chimaera initiate an acquisition dialogue • Chimaera responds by asking questions: • “How many parents must a person have?” • n • At least n • At most n • Any number • Don’t know • “What kind of an object is a parent of a person?” • The parent is a … • Assume the SME responds: • “The parent is a person and a cat” • Chimaera might respond: • “A person cannot also be a cat. Is a parent of a person always a person?”

9. Next Steps: “Background” Analysis • Reasoning tests that may take substantial time • Performed in background • Results incrementally posted on Web page • Result summaries sent to user via e-mail when ready • Example tests • Redundant axioms that are inferred by the KB (anomaly) • Inconsistent axioms whose negations are inferred by the KB (error) • Determine which relations in KB are primitive and non-primitive (summary) • Show relations on which each non-primitive relation depend • Determine classes that are disjoint (suggest adding results to KB) • Derive subclass and instance of links (suggest adding links to KB) I.e., classification and recognition • Suggest reordering of an implication’s antecedents based on number of inferable instances of each antecedent (suggestion)

10. Contributions To SRI Team • Defaults • Delivered design document, June 2000 • Providing design support for defaults in the Summer 2001 KB • Partition-Based Logical Reasoning • Techniques for answering queries from large KBs • Working jointly with Eyal Amir in John McCarthy’s group • Knowledge Base Diagnostics • SRI has provided sample RKF KBs • KSL has diagnosed the sample KBs and obtained feedback from SRI • KSL to provide a KB diagnosis service on an ongoing basis • SRI team to provide evaluation feedback on the diagnosis service • KSL to develop strategies for KB repair dialogues • KSL to develop strategies for incremental diagnosis

11. KB Diagnosis Component Evaluation • Evaluation methodology • Obtain structured feedback from KB developers • “Check Box” feedback on individual diagnosis results • Follow-up questions on a sampling of diagnosis results • Summary assessments of overall value of diagnosis • Record and analyze repair dialogue use • Sample “Check Box” questions for KB developer • Does this diagnostic provide information you did not know? [yes no] • Does this diagnostic provide information you need to know? [1 2 3 4 5] • Are you going to change the KB in response to this diagnostic? [yes perhaps no] • How difficult would it be to obtain this information in some other way? [1 2 3 4 5] • Is the diagnostic understandable? [yes mostly marginally no] • Was running these diagnostics worth the time and effort? [1 2 3 4 5]

12. KB Diagnosis Component Evaluation • Sample follow-up questions • Check box question: Is the diagnostic understandable? [yes mostly marginally no] • Follow-up question: What would have made the diagnostic more understandable? • Check box question: Does this diagnostic provide information you need to know? [1 2 3 4 5] • Follow-up question: What property or capability of the KB did this diagnostic enable you to improve? • Sample summary assessment questions • What diagnostic information about this KB would you like to have that is not provided by Chimaera? • What would make running these diagnostics more worthwhile?

13. Summary • KSL is building the Chimaera tool suite to support KB evolution • Our current focus is on diagnosing KBs • Providing a KB diagnosis Web service • Finding errors and anomalies in both structure and content • Providing advice to KB authors • Using reasoners to provide sophisticated diagnostics • Developing KB repair and acquisition dialogues • We are providing support to the SRI team in multiple ways • Defaults • Partition-Based Logical Reasoning • KB Diagnostics • We will do a component evaluation experiment • To evaluate Chimaera’s KB diagnostics • Based on structured feedback from KB developers and repair dialogue use