Some Thoughts on Machine Understanding

1. Some Thoughts on Machine Understanding Peter Clark Knowledge Systems Boeing Engineering and Information Technology

2. On Machine Understanding • Understanding = creating a situation-specific model (SSM), coherent with data & background knowledge • Data suggests model fragments which may be appropriate • Models suggest ways of interpreting data ? ? Garbled graph of relationships Coherent Model (situation-specific)

3. On Machine Understanding • Core theories of the world • Ton of common-sense/ episodic/experiential knowledge (“the way the world is”) • Only a tiny part of the target model • Contains errors and ambiguity • Not even a subset of the target model Assembly of pieces, assessment of coherence, inference ? ? Garbled graph of relationships Coherent Model (situation-specific)

4. What are some ingredients? • Elaboration (“scene building”) • Representing possibilities • Coherence assessment (“matching”?) • Viewpoints/context • Knowledge acquisition

5. Elaboration:The Parachute Sentences “Parachutes slow down a person falling through the air. This means that he or she can land safely when jumping out of a plane. When open, a parachute creates lots of drag as air pushes against its underside. This slows its fall.”

6. The Parachute Sentences “Parachutes slow down a person falling through the air. This means that he or she can land safely when jumping out of a plane. When open, a parachute creates lots of drag as air pushes against its underside. This slows its fall.”

7. 1. Elaboration (cont) “John chopped down the tree.” • A vivid picture comes to mind • John: adult male, out in woods • holding an axe (or chain saw?) • Tree is ~30ft high pine tree • or: a modification of that time I sawed a Christmas tree • or: that documentary on logging in Canada • 8:1 ratio of prior to explicit knowledge (Graesser, ’81) • Episodic/experiential knowledge plays a key role • Also core knowledge plays a key role • Not a deductive process!

8. 1. Elaboration: Using WordNet • Augment “semantic structure” with definitional “knowledge”. “The kid hit the ball hard.”

9. 1. Elaboration: Using WordNet • Augment “semantic structure” with definitional “knowledge”. “The kid hit the ball hard.”

10. 1. Elaboration: Another example “The Global Positioning System is a satellite navigation system designed to provide instantaneous position, velocity and time information almost anywhere on the globe.” • satellite: orbit around earth; receive/send radio messages • navigation: information about location • system: assembly of artifacts which together perform a task • people: often want to know where they are • (after more sentences): entire model on how GPS systems work.

11. 2. Representing Possibilities • Went to encode a space of possibilities • not what the model is, but constraints on what the actual models might be • enable actual models to be built and assessed where? “Most eucaryotic genes have their coding sequences interrupted by noncoding sequences, called introns. The scattered pieces of coding sequence, called exons, are usually shorter than the introns, and the coding portion of a gene is often only a small fraction of the total length of the gene. Most introns range in length from about 80 nucleotides to 10,000 nucleotides, although even longer introns exist.” p220, Alberts 1998.

12. 2. Representing Possibilities Got Want Possible (consistent) More likely/ preferred Impossible (inconsistent) Less likely/preferred Spaces of possible models

13. 2. Representing Possibilities • Are a few (feeble) methods in KM for this: • type restrictions (every Person has (spouse ((must-be-a Person))))) • (sometimes <x>) • (every Car has (parts ((sometimes (a Spare-Wheel))) • Cardinality constraints • (a Group with (min-cardinality (…)) (max-cardinality (…))) • Range of values • “size is between X and Y” • Still largely lacking in how to represent and reason with vague knowledge

14. 3. What Makes a Representation (Model) Coherent? • We don’t just blindly accept new knowledge: • Minsky: We proactively ask a set of pertinent questions about a scene, e.g., what is X for? What are the goals? etc. • What makes a representation coherent? • Simple consistency (“The man fired the gnu.”) • Purposefulness (for artifacts): • “The engine contains a thrust reverser.” • vs. “The engine contains an elephant.” • vs. “The engine contains a book.” • “Knowledge entry” is a serious misnomer! • Really talking about Knowledge Integration

15. 3. What Makes a Representation (Model) Coherent? “TRANSPONDER: A combination receiving and transmitting antenna on a communications satellite. TRANSPONDER: A combination receiver, frequency converter, and transmitter package, physically part of a communications satellite. Transponder parts: antenna purpose:receive, transmit Transponder parts: receiver frequency converter transmitter part-of: communications satellite Relay/Mediate

16. 3. A Catalog of Coherence Criterea • Volitional actions: • Agents must be capable of an action • legally, skill, fiscally, anatomically • Action serves a broader purpose/goal • Need equipment/resources/instruments, instruments must be adequate • Non-volitional actions; • There is a cause (inc. randomness) • Spatial: • statics: objects must be close • dynamics: objects can move in the required way • Temporal: objects exist at the same time

17. 3. A Catalog of Coherence Criterea (cont) • Objects: • physically possible • parts connected together at appropriate places • materials are appropriate • suspension/tension etc., gravity • physically normal/expected/standard • need to know normal shapes, sizes, etc. • Artifacts: • Purposefulness: • all parts play some role wrt. one of its intended functions (or subtasks thereof). Expect design to be optimized. • Could treat biological objects as “artifacts”

18. 3. Coherence and KM • KM unable to tolerate incoherence: • Current: “Error! Switching on the debugger…” • Desired: “This representation is generally ok, except this bit looks weird, and that bit conflicts with this bit.” • Problem compounded by long inference chains • (cf. Cyc: don’t think too hard ) • How could we change KM to be more tolerant?

19. 4. Viewpoints and Context Component theories/ ontologies(?) • Latter seems right, but: • can a big KB really be partitioned like this? (everything is connected!) • Models may vary by: • ignoring detail • making different approximations • using different ontologies vs. Reason over Giant KB Problem-specific KB, contains selected units

20. 4. Viewpoints and Context • e.g., DNA = sequence of different region types: • intron-exon-intron-exon… • promotor-gene-terminator • nucleotide pair-nuc pair-nuc pair… • Makes a difference: • Given: “The polymerase attaches to the promotor, and then moves down the strand.” • then answer: “Where will the polymerase be?” • nucleotide? gene? intron? • The point: It’s not simply a matter of having all viewpoints coexisting • Another example: “A satellite sends signals/messages/position information.”

21. 5. Knowledge Acquisition • Where do the core theories come from? • Hand engineered? • Where does all the “mundane” knowledge come from? • Schubert-style? • Dictionary/glossary definitions?

22. 5. Knowledge Acquisition:Can all this be Bootstrapped? Collection of coherent scenes Jungle of parse trees/ semantic graphs KB Text Domain Ontology List of compound nouns and verbs (entities and actions) Scene library