Spatial reasoning in a multi-modal user guide for a complex machine

1. Delft University of Technology, Faculty of Information Technology and Systems Spatial reasoning in a multi-modal user guide for a complex machine Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen

2. Contents • Problem introduction • Application • Geometry based reasoning • Techniques • Examples • Questions

3. Problem introduction • Multimodal interaction • Multimodal = speech + graphics + text • Computer speech needs feedback of text and graphics • Speech introduces geometric references, (e.g.the object in front of the building) • Solving of these references needs combination of geometric and functional reasoning

4. ? Application • Application: Interactive fax user guide • Speech & Reasoning & Graphics • Task: develop dialogue technologies

5. How can I make a copy of a document? • Put the document into the paper source tray • Press copy button Fax encyclopedia • Single questions & answers about the fax • Questions: Speech • Answers: Graphics & Speech

6. Not an intelligent system. Only pre-cooked question and answers are possible Possible solutions • Pre-programmed answers • develop set of possible user questions • develop answer + “movie” for every question

7. Possible solutions • Answers derived by reasoning • knowledge based system containing: • data about fax functionality • data about fax construction (3D model) • Reasoning system based on: • semantic • geometry • Answers derived by reasoning • knowledge based system containing: • data about fax functionality • data about fax construction (3D model) • Reasoning system based on: • semantics • geometry

8. System architecture ? Speech recognizer Parser Semantic based reasoning Dialogue manager Textual answer generation Geometry based reasoning Graphical answer generation

9. Graphics • Tasks: • Navigation though the virtual world • Solve geometric references Geometry based reasoning • Tasks: • Show components • Derive descriptions of a component location

10. How can I make a copy of a document? • Put the document into the paper source tray • Press copy button Geometry based reasoning • Show components

11. Geometry based reasoning • Show components • choose appropriate scale • choose appropriate view point

12. Geometry based reasoning • Deriving location descriptions Where is the power slot? Power slot is on the back side of the fax

13. Geometry based reasoning • Solve geometric references What is the button left of the start button? This is a copy button

14. Telefax Power plug .... Corpus Telephone .... Operational panel Telephone receiver LCD screen Start button Keypad Top face Corner Rib Corner Bottom face edge edge vertex face face Geometry based reasoning • Data about fax construction • hierarchy graph • feature graph • geometry graph

15. Techniques • Problem:search for a suitable reference object • Proposal:use of hierarchy • Result: reduction of reasoning complexity Telefax Power plug .... Corpus Telephone .... Operational panel Telephone receiver LCD screen Start button Keypad

16. Techniques • Analyze of dimensionality in view of children location • Objects that have children nodes: • 3D objects: fax corpus, telephone • 2D objects: operational panel, slot panel • 1D objects: cables • Set objects: group of slots, group of switches • Objects that do not have children nodes: • Leaf objects

17. Techniques • Use of applicability zone in for a geometric reference • when reason about a position on a surface of another object the spatial reference differ for 3D object and 2D object • use of fuzzy logic to process zones of applicability

18. Examples • Deriving location descriptions Where is the power slot? Power slot is on the back side of the fax

19. Deriving location descriptions • Search for the object parent • Select meaningful descriptions 2 priority Location relatively to neighbor objects 2.1 between (two other objects) 2.2 in front of , under, above 2.3 to the left of 2.4 to the left of 2.5 behind, under 2.6. near • 1 priority • Location on the surface • 1.1 on the corner • 1.2 on the right / left side • 1.3 on the top/ bottom face • 1.4 on the front / back face Developed answer : The printer slot is on the back face of the fax, to the left of the printer slot

20. Examples • Solve geometric references What is the button left of the start button? This is a copy button

21. 1. Search through the same hierarchy level 1.1 Search for a parent group of operational buttons 1.2 Analyze location left of the start button 1.3. Choose suitable objects 2. Go one level higher in the hierarchy 2.1 Search for a parent of the parent operational panel 2.2 Analyze location left of the start button 2.3 Select objects 3. Go one level higher……... Solve geometric references

22. Summary • Role of spatial reasoning techniques in the application of multi-modal user guides • A fax machine user guide system • Untilisng 3D model of a fax • Use of hierarchy • Use of dimentionality • Use of areference applicability zone

23. Conclusions • Geometry based reasoning is a new functionality for such a dialogue system • 3D model of the fax-machine is used as a source of data for reasoning • Several techniques for the geometry-based reasoning were developed and implemented using Java3D and OpenGL languages

24. Questions ?