Knowledge Base Diagnostics Richard Fikes (Stanford KSL) Adam Pease (Teknowledge)

1. Knowledge Base Diagnostics Richard Fikes (Stanford KSL) Adam Pease (Teknowledge) Mala Mehrotra (Pragati Synergetic Research Inc.) Yolanda Gil (USC ISI) Deborah McGuinness (Stanford KSL) 10/18/01

2. Knowledge Evolution Tools • KB development requires knowledge evolution Debugging, refining, structuring, modularizing, … • Power tools are needed to support KB evolution • KB diagnosis • Bugs, omissions, heuristic warnings, architectural advice • KB partitioning • To enable effective reasoning • To produce reusable KB building blocks • KB merging • To enable interoperation of KBs with overlapping content • KSL is developing knowledge evolution tools

3. Chimaera • A Knowledge Evolution Tool Environment • Tools for KB diagnosis and merging • Available as a Web service or an OKBC client • www.ksl.stanford.edu/software/chimaera • Usable from a Web browser • Online user manual, tutorial, and demonstration movie • Performs KB diagnostics in batch mode • Uploads and analyzes user’s KB • Accepts KBs in OKBC, KIF, MELD, RDF, DAML, … • Provides results as HTML pages linked to frames and axioms • Provides user selectable set of diagnostic tests • Analyzes both the structure and content of a KB • Uses reasoners to analyze content

4. Classification of Diagnostic Results • Errors • Logical inconsistencies E.g., contradictory type constraints • Content structure errors E.g., terms used but not defined • Anomalies • Missing information E.g., type constraints • Redundancies E.g., redundant superclass and type links • Extraneous structure or content E.g., terms defined but not used • Summaries E.g., counts of term references • Suggestions E.g., use consistent naming conventions

5. “Background” Reasoning Analysis • Reasoning diagnostics that may take substantial time • Performed in background • Results incrementally posted on Web page • Completion notification sent to user via e-mail • Example reasoning diagnostics • Redundant axioms that are inferred by the KB (anomaly) • Inconsistent axioms whose negations are inferred by the KB (error) • Determine which relations in KB are primitive and non-primitive (summary) • Show relations on which each non-primitive relation depend • Determine classes that are disjoint (suggest adding results to KB) • Derive subclass and instance links (suggest adding links to KB) I.e., classification and recognition • Suggest reordering of an implication’s antecedents based on number of inferable instances of each antecedent (suggestion)

6. Integration Into SHAKEN • Chimaera is a KB diagnostics tool in the SHAKEN system • Used to diagnose both pump priming and SME KBs • OKBC was used to do the integration • Chimaera is an OKBC client • Interacts with any OKBC server using the OKBC API • The Chimaera Web service uses Ontolingua as its OKBC server • SRI added an OKBC wrapper to the KM system • Enabled KM to be an OKBC server usable by OKBC clients • Enabled Chimaera’s diagnostics to run directly on KM KBs

7. Chimaera Useful To SRI Team “Overall, we found that Chimaera was quite useful. It found 2 concepts (Indole and Imidazole) that were corrupted, several occurrences of redundant superclasses, and several incorrect domain and range constraints (due to our poor representation of "Information"). … We're currently fixing the bugs it revealed. It would be helpful if we could run Chimera on the component library frequently.” – Bruce Porter

8. Next Steps: SME-Oriented Support • Provide interactive repair oriented follow-up to diagnostics • Identify KB content on which diagnosis result is based • Suggest repairs or repair strategies • Guide user through repair procedure • Examples • Class is a direct subclass of “THING” • Provide direct subclasses of THING as candidate superclasses • Step down through the class hierarchy • Class has redundant superclass links • Suggest removal of link(s) to most general classes • Type, cardinality, or bounds conflict • Suggest changing local conflicting constraint(s) • Missing information • Initiate acquisition dialogues for missing information

9. Next Steps: Architectural Analysis • Summarize architectural features of a KB • Percentage of • Relations that are functions • Axioms that are propositional, first order, higher order • Axioms that are not horn clauses • Distribution of • Axioms by type (using the HPKB, RKF types) • Axiom lengths by number of literals • Functions by number of arguments • Relations by number of arguments • Direct subclasses per class • Direct subproperties per property • Restrictions per object • Property values per object

10. Next Steps: Partitioning and Beyond • Integration of KB partitioning tools into Chimaera • Provide automatic KB partitioning to enhance usability • Automatic running of test cases E.g., queries and expected answers • Support regression testing of evolving KB • Provide result summaries from failed tests • Help with typographical errors • Spelling correction for undefined names E.g., classes, slots, relations, functions, constants • Spelling correction for anomalously occurring variables • Suggest is the same as another variable in the sentence

11. Summary • KSL is developing Chimaera to support KB evolution • Chimaera was integrated into the SHAKEN Y1 system Using OKBC(!) • Incrementally adding diagnostics E.g., “background” diagnostics that use sophisticated reasoning • Next steps • KB partitioning tools • Repair dialogues for SMEs • KB architectural analysis • Regression testing

12. Role of Diagnostics in Systems • KE support • SME support • Increase productivity (“lightly trained”) • Step in managing KB development • Focus attention (e.g., redundant links) • Evaluation support • Diagnose KBs produced during evaluation • Batch mode • Foreground • Background • Changes in “patterns” in the KB between versions

13. Sharing Diagnostics Information • Diagnostic specifications • Logical specifications • English specifications • Test cases • Diagnostic classifications • Learnings • Tricks of the trade • Sharing facilitators: • Working group • Mailing list • Findings data • Author, group, or team specific • Repair strategies • Alignments during collaborative development

14. Developer Needs and Desires • Reasoner-specific diagnostics • Highly informative diagnostic results • Reporting architectural bias in a KB • Binary versus higher order relations • First order versus higher order axioms • Weakly versus strongly higher order • Disjunctions or conjunctions • Existential versus universal quantifiers • Frames to axioms ratios • Horn clauses • Axiom lengths • Functions • Confusion of existential and universal quantifiers • Type restrictions too general • Misspelling of variables

15. Developer Needs and Desires • Domain-specific tests • Semantic tests • Maintainability measures • Recognizing typographical errors • Spell check undefined or unused terms • Redefining (e.g., breaking up) a predicate • Large scale modification techniques • Prioritizing diagnostics

16. Integration Issues • Architecture • Use hosted services (like KSL) • Integrate special code • Take specifications from library • API • Interaction Mode - Batch versus Interactive/Repair • Translation issues • One major use of diagnostics is also in testing translators • Certain translations need to be done to do better analysis • Output integration

17. Evaluation • Record types and numbers of errors • Comparing KBs produced by SMEs versus KEs • Record use of repair strategies • Evaluate during testing • Feedback from SMEs about diagnostics

18. Classification of Diagnostic Results • Errors • Logical inconsistencies • Content structure errors • (See Randy Davis thesis) • Anomalies • Missing information • Missing portions of descriptions • Redundancies • Extraneous structure or content • Summaries • Architectural biases • Suggestions • Stylistic suggestions • Static versus operational tests • Use of expertise about KR paradigms

19. Diagnostic Issues/Goals • Role of Diagnostics in Systems • KE support, SME support • Evaluators of KBs • How to Share Diagnostics • Working Group? • Logical specification, English descriptions, tests, … • Know the Main Contributors • Possible Diagnostics • What do users want? • What can tool builders provide? • Integration Issues • Developer Needs/Desires • Evaluation

20. The Role of KB Diagnostics • KE support • SME support • Increase productivity (“lightly trained”) • Mgmt of kb • Inference dependent quality improvement • Focus attention (ex. Redundant links) • Evaluation support • Abstract patterns – average fanout of specialization, statistics of number of uses of a predicate – big picture view • Version comparison • Regression testing

21. Diagnostic Sharing • Diagnostic specifications • Logical specifications • English specifications • Test cases • Diagnostic classifications • Taxonomy of errors – bottlenecks, • Quantification • Alignments across systems – inconsistencies among smes • Repair strategies • How informative a system is (core dump vs. useful explanation) • Learnings • Tricks of the trade • Sharing Facilitators: • Working Group • Mailing list

22. Sharing facilities • Working group • Mailing list • Posting of papers • Utilize Teknowledge

23. biases • Binary vs. higher arity • First order vs higher order • Weakly vs strongly higher order • Universal over existential • Disjunction vs. conjunction • Frame-ism • Horn clauses • Lisp style • Relations -> functions • Depth vs. breadth in hierarchy • …. Maybe report in summarizations.. • At least document biases

24. Organizations/People • Cycorp – many special purpose - Kahlert • ISI – Why Not? – Chalupsky – KANAL – Gil - expect - Gil • Pragati – Clustering - Mehrotra • Stanford FRG/KSL – Partitioning – McCarthy, Amir, McIlraith • Stanford KSL – Chimaera - Fikes, McGuinness

25. Diagnostics • Errors – provable logical inconsistencies • Anomalies – redundancies, cycles,… • Summaries – word counts, … • Suggestions – naming conventions • Incompletenesses – explicit salient assertions or statistics • Stylistics - length of rule, … bad factoring, Randy davis – errors – incompleteness, inconsistent • Get this - Top ten list of things people do wrong in cyc - goolsbey Perspectives/units: Frame-like content vs. axioms vs. problem solving technology vs. learning to correct components

26. style • Static • Reasoner • Simulation / execution • Using examples • Summarization/improvements/critiquer

27. Integration Issues • Architecuture • Use hosted services (like KSL) • Integrate special code • Take specifications from library • API • Interaction Mode – Batch vs. Interactive/Repair • Translation issues • one major use of diagnostics is also in testing translators • Certain translations need to be done to do better analysis • Background ontologies – meld starter ontology • Output integration

28. Developer Needs/Desires Missing existentials Too high a type specification Variable name mismatch Semantic requests: Wrong semantic paradigm? Typos Spell check Large scale modification tools and their integration example removal/ fixing top level priotizing Diagnostics to minimize cost, ease maintenance

29. Evaluation • Record types of errors • Fine granularity • Kb differences across sme vs. ke developed ontologies across team • Record use of repair strategies… • Evaluate during testing… • Feedback from smes on features, usefulness, etc. • Attempt to keep extremely complete audit trails for future analysis • Important to be careful with diagnostic reporting

30. Action Items • Working Group • Diagnostics repository • Web site • Follow up briefing • Mailing list

31. Chimaera • A Knowledge Evolution Environment • Tools for KB diagnosis and merging • Available as a Web service • www-ksl-svc.stanford.edu www.ksl.stanford.edu/software/chimaera • Usable from a Web browser • Online user manual, tutorial, and demonstration movie • Provides user selectable set of diagnostic tests • Performs kb diagnostics in batch mode • Uploads and analyzes user’s KB • Accepts KBs in MELD, KIF, OKBC, DAML, RDF, XML, … • Provides results as HTML pages linked to frames and axioms • Analyzes both the structure and content of a KB • Uses hybrid reasoners to analyze content • Currently runs 28 diagnostic tests

32. Collection/Specification • Logical Specification of diagnostic • English Specification • Example kb that triggers diagnostic output

33. Classification of Diagnostic Results II • Axiom Analysis • Axiom Syntax Problems E.g., no consequent to a implications • Axiom Redundancy E.g., 1. A =>B 2. A=>C 3. C =>B means 1 is redundant • Axiom Variable Usage E.g., Variable used in antecedent but not in consequent • Axiom Consistency E.g., A => not A • Axiom Tautology E.g., consequent repeats (portion of) antecedent