Building KB’s by Assembling Components: An early evaluation of the approach

1. Building KB’s by Assembling Components: An early evaluation of the approach Bruce Porter (University of Texas) Peter Clark (Boeing) and Colleagues

2. Claims A component is a set of axioms that: • describe a consensus view of a common concept • can be frequently reused with little modification … • … to combine with other components … • … to represent domain knowledge well.

3. Claims A component is a set of axioms that: • describe a consensus view of a common concept • can be frequently reused with little modification … • … to combine with other components … • … to represent domain knowledge well. versus an idiosyncratic view

4. Claims A component is a set of axioms that: • describe a consensus view of a common concept • can be frequently reused with little modification … • … to combine with other components … • … to represent domain knowledge well. versus infrequently reused without significant modification

5. Claims A component is a set of axioms that: • describe a consensus view of a common concept • can be frequently reused with little modification … • … to combine with other components … • … to represent domain knowledge well. versus difficult or impossible to combine (“round holes and square pegs)

6. Claims A component is a set of axioms that: • describe a consensus view of a common concept • can be frequently reused with little modification … • … to combine with other components … • … to represent domain knowledge well. versus representations are incomplete or inconsistent We’ll evaluate each of these claims in turn …

7. The Context for our Evaluation: The Component Approach in Practice • We are building a component library while assembling a microbiology KB for the TKCP • For each microbiology topic, we will: • Identify the core concepts used in textbook accounts • Build components for these concepts • Use these components to represent the topic in the KB

8. Do the components capture a consensus view? • Internal evaluation: • Have multiple KE’s (team members) independently encode each component. • Measure the agreement among the representations • External evaluation: • Translate components from KM into English, yielding “dictionary definitions” • Have n subjects review the definitions then revise them if they deem appropriate. • Have an independent subject extract the “consensus view” from the n definitions. • Measure the agreement between the consensus view and the original ones.

9. Can the components befrequently reused with little modification? • Count the amount of reuse of each component. • Count the number of modifications made for each instance of reuse, and weight it by its severity.

10. Do the componentscombine together well? • For each microbiology topic: • Write a high-level design for its representation which shows: • The components that comprise it • Their instantiations and inter-relationships • Implement the design • Measure and study the times when implementation details force changes to the design

11. Do the componentsrepresent domain knowledge well? • For each topic represented in the KB: • Measure how much of its representation was provided by components; the balance was coded specifically for this topic. • The TKCP will measure the quality of the overall representation, in terms of consistency, completeness, support for inference, and so on. • Analyze the KB’s successes and failures at the TKCP to assign responsibility to the components (versus other aspects of the KB system).

12. Summary • We have identified the “strong claims” of the component approach • We plan to evaluate the claims early – while building a microbiology KB • The data we collect will be invaluable to study questions we’ve only begun to formulate