Update on RKF progress October, 2000

1. Update on RKF progressOctober, 2000 Ken Forbus Qualitative Reasoning Group Northwestern University

2. Overview • Analogical Reasoning • Reasoning Engines • Domain Theories • Sketching

3. Base SME CVmatch SME CVmatch Target SME CVmatch SME CVmatch Our analogical processing tools Structure-Mapping Engineprovides analogical matching Inputs = propositional descriptions, w/ incremental updates Output = one or two mappings Mappings = correspondences + structural evaluation + candidate inferences Operates in polynomial time, by exploiting graph labels & greedy algorithms MAC/FAC providessimilarity-based retrieval Probe Memory Pool Output = memory item + SME results No hand-indexing of cases required Cheap, fast, non-structural

4. Generalizations New Stimulus … SME Exemplars How SEQL Works SEQL refines knowledge by progressive alignment of examples New: The GEL algorithm 1. Compare against each generalization Gi. If close enough, assimilate input into Gi by replacing Gi with the overlap of Gi and input and halt. 2. Compare input against each exemplar Ei. If similar enough, create new generalization from overlap of Ei and input, halt. If nothing similar enough, add input to set of exemplars

5. Case Mapper: An Analogy GUI • Goal: Provide civilized interface for entering knowledge via analogy • Should be useful platform for experimenting with dialogue moves • Current state • Basic functionality showing signs of life • AI-expert friendly • Next steps • Improved pidgin • Interface to inference machinery for candidate inference evaluation • Explore using dialogue management, simple NLP for interaction

6. Initial results of Matching

7. Exploring the candidate inferences

8. Integrating into the E2E system • Strategy: Provide analogy server • KQML for communication • Strategies for analogical reasoning coded in next-generation reasoner • Advantages • Neutral with respect to uniprocessor/distributed operation • Enables us to tune our strategies more easily • Drawbacks • Sockets as bottleneck • Need to keep KB in synch • Alternative strategy: Assimilation

9. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Domain Theory Environment (DTE)

10. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Domain Theory Environment (DTE) Uses ODBC, Relational database (Microsoft Access) to store KB contents (inspired by Hendler’s PARKA-DB)

11. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Domain Theory Environment (DTE) Federated architecture, supports reasoning sourcesthat provide special-purposecapabilities efficiently

12. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Domain Theory Environment (DTE) Query-driven backchainerprovides basic reasoning services, integration mechanism

13. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Domain Theory Environment (DTE) KQML interface for building servers(e.g., analogy server,geographic reasoner)

14. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base DTE Problems High overhead,too many computational cliffs Too slow, not scaling well

15. Solution: Build next-generation system • Collaborating with Xerox PARC • John Everett, Reinhard Stolle, Bob Cheslow • Keeping good ideas in DTE: • Federated architecture/Reasoning sources model • Using database to implement KB • Query mechanism with simple backchainer as glue • Use of LTMS for justifications, reasoning • Overall structure of interfaces to applications using it will be similar • Internals will be very different

16. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Next-generation system Special-purpose C++ database,written by PARC. Built-in support for pattern matching.Adding new knowledge:DTE DB: 4 assertions/secondNew DB: 98 assertions/secondRetrieval:2-3 msec, in 111K assertion KB (preliminary data)

17. Analogical Reasoner Reasoner Spatial Reasoner GIS Knowledge Base Next-generation System Working memory = LTRE + discrimination tree indexing.Suggestions Architecture:Limit backchaining for “quick” reasoning. Expensive operations queued as suggestions, processed via agenda mechanism.Multithreaded, to exploit time user spends doing other things. Especially important for sketching, dialogue management

18. Next-Generation System Streamlined reasoning source interface, with constraint posting for query optimizer. Analogical Reasoner Reasoner Provide qualitative reasoningservices by embedding QP theory implementation Spatial Reasoner Gizmo Mk2 Create ink-based spatial reasoner, organized for incremental processing from the ground up Knowledge Base Perceptual Ink Processor

19. Current schedule • Halloween: First version turning over • Thanksgiving: DTE applications ported • Christmas: First round of performance tuning finished

20. Claim: There is a basic set of physical notions that need to be understood in order to interpret sketched explanations e.g., Simple notions of surfaces, volumes, forces, and materials Claim: Qualitative physics research can provide most of this knowledge Much of it has already been done, in isolated pieces Needs to be integrated, gaps filled Tied to sketch-based spatial representations Surface constraints on motion Will use Nielsen’s qualitative mechanics Fluid Ontologies Collins’ molecular collection ontology Kim’s bounded stuff ontology + usual contained stuff ontology Surface/fluid interactions Kim’s qualitative streamline theory Qualitative topology Cohn’s spatial algebras Qualitative Statics Nielsen & Kim’s qualitative vectors Everyday Physical Semantics domain theory

21. Multiple Perspectives: An example • How to reason about liquids? • Two models, due to Hayes • Contained stuff ontology: Individuate liquid via the space that it is in. • Piece of stuff ontology: Individuate liquid as a particular collection of molecules.

22. Fluid ontologies • Contained stuffs • Most detailed: Paper with John Collins, FSThermo domain theory • Pieces of stuff • Molecular collections (w/John Collins) • Plugs (Gordon Skorstad) • Bounded stuffs (H. Kim)

23. Molecular Collection ontology • Idea: Follow a little piece of stuff around a system • So small that when it reaches a junction, it never splits apart • Provides the perspective gained by tracing through a system of changes

24. Two containers example

25. Steam plant example

26. Refrigerator example

27. Bounded stuffs • Specialization of contained stuff ontology • Where something is within the space matters • Affects connectivity

28. Ontology zoo for liquids Contained Stuff Piece of Stuff Parasitic on Bounded Stuff Molecular Collection Plug

29. Ok Ok not Ok Qualitative Mechanics • Provides axioms for interaction of solids and surfaces • Qualitative vector representation • Assumes visual parsing of 2D shapes • Center of gravity, center of rotation critical • Surfaces broken at corners, points of contact not Ok

30. Qualitative Mechanics • Qualitative angles and vectors • How forces interact with surfaces, constraints on motion • Laminar flow fields

31. Engineering Thermodynamics • Basics of heat, mass flow • In-depth KB for supporting design, analysis • KB for supporting textbook problem solving • Includes control knowledge, analysis of roles for equations in problem-solving • Pisan’s Ph.D. thesis solves most problems in typical engineering thermodynamics textbooks • Teleological representations for thermodynamic cycles • No chemical interactions

32. sKEA: Sketch-based Knowledge Entry Associate Built on top of nuSketch + significant extensions Rich perceptual processing of digital ink Will support visual analogies and analogies using diagrams Speech I/O and specialized Dialogue Manager Can be used standalone or as component in larger system Ink Interpretation is key problem Collaborating with PARC vision group (Eric Saund, Jim Mahoney) for perceptual processing Developing domain theories that bridge perception and conceptual knowledge Multimodal Integrator Speech I/O RKF Team System DTE + Evidential Reasoner Current Sketches + Interpretations Graphical Symbology Domain Theory High-Level Visual Interpreter (GeoRep II) sKEA Everyday Physical Semantics Domain Theory Perceptual Ink Processor Digital ink Sketching for knowledge acquisition

33. Tools we will use in sketching GeoRep provides high-level visual processing for spatial reasoning Provides bridge between the visual and the conceptual Provides equivalent of Ullman’s universal visual routines MAGI models processes of symmetry and regularity detection GeoRep MAGI • Uses variation of structure-mapping laws to detect self-similarity • Same software operates on visual, functional, conceptual, and mathematical representations • Makes predictions consistent with human perceptual data

34. Represents conventions for displaying conceptual information graphically Includes What visual entities often depict boxes, blobs, arrows, etc. Conventional views side/top/bottom, 2D/3D, abstract/physical, cutaways Conceptual interpretation of visual relations proximity/alignment indicating grouping, inside indicating containment or partonomy,touching indicating contact (in-contact (protein-coat virus) (lysosome cell)) Virus DNA Cell DNA (Part-of cellDNA cell) Visual Symbology domain theory State (after) State (before) Process Binary Relationship Arg2 Arg1

39. Approach: Blob Semantics • Shape, object recognition irrelevant • Linguistic input provides labels and type information • Arrows may be exception wrt recognition • Spatial relationships between blobs is central • Topology • Touching or not, inside, overlap • Proximity • What arrows refer to • Orientation • Multiple reference frames • Quadrant plus relative inclination • Conceptual interpretation of spatial relationships • Hypothesis: Sufficient for • Process diagrams • Action sequences

40. Issues in blob semantics • Adequacy of visual primitives • User-defined diagram types • Kinds of objects participating • Conceptual interpretation of spatial relationships • Arrow recognition • Support different types of arrows?

41. Perceptual Ink Processor • Will use next-generation reasoner for conceptual side of reasoning • For visual reasoning, draw on three sources: • Our work on GeoRep and Magi (Ferguson’s Ph.D. work) • Eric Saund’s scale-space blackboard (Xerox PARC) • Stroke-based visual routines • Should provide robust proximity detection • Jim Mahoney’s MAPS ideas (Xerox PARC) • Bitmap-based visual routines • Should provide robust qualitative descriptions of free space

42. Example: eTDG10 Map

43. SR Regions for eTDG10 map (hand-sketched)

44. Hard constraints from SR regions

45. Voronoi diagram for free space

46. Junctions provide seeds for open regions

47. Regions extended from seeds

48. Edges outside regions form corridor seeds

49. Combined results for eTDG10

50. Speech or not? • Most multimodal systems use speech recognition • Hands, eyes busy with diagram • Potential problems with speech for RKF • Novel nouns, phrases could lead to distracting speech training during knowledge entry • How open-ended is grammar? Necessity versus user expectations • Trying both in RKF • NLP support with speech • LKB parser (Stanford CSLI) • Experiment: Speechless multimodal interface • Type (or write) label for instance, collection • Draw button, as in nuSketch COA Creator • Sacrifice fluidity for expressiveness