Visual Communication: theoretical issues

1. Visual Communication: theoretical issues Research Seminar Master Informatiekunde 13/09/06 Leonie Bosveld-de Smet Humanities Computing, RuG

2. Question • Pictures can be worth ten thousand words. • Visual representations can be superior to verbal ones in thinking (problem solving), teaching, and communication. • Can benefits observed in other areas (esp. science) be exploited in other fields? Visual Communication

3. Outline talk • Characteristics of graphical versus sentential languages • Facilitating effects of visual representations • Visual communication: a semantic view • Conclusion and discussion Visual Communication

4. Theoretical concepts • Key words: • Representation system • Interpretation function • Form of representation • Perception of representation • Medium • modality Form of representation Piece of information Structured information Representation system link Visual Communication

5. Media and modality contrasts • Stenning and Inder(1995) • “Medium” how is a representation perceived? “Modality” how is a representation interpreted? Visual Communication

6. Graphical vs Linguistic Representations • Stenning and Inder (1995) • the more constrained, the less expressive, the more tractable • Stenning and Oberlander (1995) • limited abstraction, aided “processibility” • Shimojima (2004) • Free ride properties • Auto-consistency • Specificity • Meaning derivation properties Visual Communication

7. FOL Defeated(Jon,Bob) & Lost_to(Ken,Bob) PD Jon Bob Ken Free Ride Property Express information: Jon defeated Bob. Ken lost to Bob. English Jon defeated Bob and Ken lost to Bob. Visual Communication

8. FOL Defeated(Jon,Bob) & Defeated(Ken,Bob) Specificity Express information: Jon defeated Bob. Ken defeated Bob. PD ? ? Jon Ken Bob Ken Jon Bob English Jon defeated Bob and Ken defeated Bob. Visual Communication

9. Defeated(Jon,Bob) & Defeated(Bob,Ken) & Defeated(Gil,Jon) & Defeated(Ken,Ron) FOL Gil Jon Bob Ken Ron PD Meaning Derivation Property Express information: Jon defeated Bob Bob defeated Ken Gil defeated Jon Ken defeated Ron Jon defeated Bob and Bob defeated Ken and Gil defeated Jon and Ken defeated Ron. English Visual Communication

10. Benefits of graphical systems • Automaticity of inference • Ease of consistency inferences • Difficulty to express “abstract” information • Richness of semantic content • “Transparent” problem solving • Computational offloading • Amplification of cognition Visual Communication

11. Monk Puzzle (Winn, 1987) “A monk went to the temple at the top of a holy mountain to meditate and pray. He started out early one morning along the path that led up to the temple. Because he was an old man, and the way was steep and arduous, he frequently slowed his pace, and even sat and rested a while beside the path. Toward evening, he came to the temple at the top. After several days of meditation and prayer, it was time for him to leave. Early in the morning, he set off back down the path. Again, he frequently changed his pace and rested by the way. He arrived back at the bottom in the evening. Show that there is one single point on the path up the mountain where the monk will be at precisely the same time both when he goes up and when he comes down.” Visual Communication

12. “See” the solution Visual Communication

13. Larkin and Simon (1987) • “Why a Diagram is (Sometimes) Worth Ten Thousand Words” • Diagrammatic vs. Sentential representations • Informational equivalence, computational offloading • Pulley problem • Geometry problem • Grouping of information • Exploitation of space • Easy perceptual inferences Visual Communication

14. Using vision to think • Kosslyn (1994) Scatter plots...employ point symbols (such as dots, small triangles, or squares) as content elements. The height of each point symbol indicates an amount. These displays typically include so many points that they form a cloud; information is conveyed by the shape and the density of the cloud • Simultaneous presentation of local information and global information implied by the local information. Visual Communication

15. Visual Communication Data Tables Visual Structures Raw Data Views Data Transformations Visual Mappings View Transformations Visual Communication

16. Semantics of pictures: Wang’s view 1 2 Graphical Domain Application Domain 3 1: geometrical characterization of pictures 2: interpretations: graphical entities – application domain objects 3: specifications: application domain concepts – picture classes Visual Communication

17. Natural vs. Unnatural Link c c b a b a Visual Communication

18. Misleading Link Gregory Charles Bill Visual Communication

19. Kinds of visual communication • Via interpretations: • Illustration • Demonstration • Reasoning • Via picture specifications: • Geometric constraint maintenance • Design • Computational art Visual Communication

20. RequirementsMeaningful and Easily Tractable diagrams • Clear delimitation of critical information • Description of information by well-defined notions and properties (elements of application domain) • Graphical devices suitable for representation of information (elements of graphical domain) • Explicit, natural, not misleading link between graphical domain and application domain Visual Communication

21. Graphical domain • Research in information visualization and diagrams and diagrammatic reasoning • Bertin • Tufte • Shneiderman • Blackwell • Hegarty • Scaife & Rogers • Novick Visual Communication

22. Possible application areas • Language instruction • Linguistic research • History instruction • Historical research Visual Communication

23. Towards an automated system for diagram research e1: see e2: stop “When we saw the spaceship, we stopped the car.” t t’ now e1: deliver e2: pay s: have paid “Two men delivered the sofa. I had already paid for it.” t t’ now e1: make lunch s: making lunch “David was making lunch when the phone rang.” e2: ring t now Visual Communication

24. 3-D Syntactic trees Visual Communication