Expanding Minds: Integrating Images and Words in Visual Language

1. Pictures and Words • When should we use a visual display? • What is a visual language? • Dual coding theory? • How to integrate images and words

2. Consider that hieroglyphs gave way to more abstract symbols • Why turn back the clock? -17K years

3. Architecture for visual thinking 10 billion neurons Parallel,automatic Interaction Loop

4. Pavio’s dual coding theory Associate structure Imagens Visual Non-verbal responses Image Information Visual System Text Information Logogens Verbal responses Verbal Auditory System Information From speech

5. Capacity of visual working memory (Vogal, Woodman, Luck, 2001) 1 second • Task – change detection • Can see 3.3 objects • Each object can be complex Orientation Shape size, texture

6. Capacity of verbal working memory • Used to be though of a 7 +/- 2 • It is now thought of as more a duration of proto-verbal codes.

7. Theory: Graphics and Words • Graphics for structural logic • Words for procedural logic: conditionals, qualifiers, if-then else, while. + causality. • Not the formal logic of math but a kind of concrete logic. (A vowel should have an odd number on the back). 8 A 3 F

8. The nature of language • Chomsky, innate deep structures. • Common to computer languages • Critical period for language development • But being verbal is not essential to language development • Sign languages for the deaf are the most perfect examples of visual language

9. What is language • Description • Communication of intention • The ability to communicate procedures and sequences of operations – including logic – if, but, causes, do a then b then c • Thus far we have only dealt with description

10. Can there be a true visual language? Production Yes, But not for most of us!! Production Broca Understanding Wernicke Consider verbal language Auditory Cortex A critical period Abstraction, logic (if, while, perhaps) Based on speech Visual Cortex Sign languages are true Visual languages

11. Sign languages Danish SL Chinese SL ASL Arose spontaneously Are not related to verbal languages Have syntax Become more abstract NSL Goldin-Meadow

12. To be fluent in visual language we should be trained from early in life

13. The visual system gives us Rapid recognition and pattern finding

14. Abstraction Pattern • Jane is Jim’s boss • Jim is Joe’s boss • Anne works for Jane • Mark works for Jim • Anne is Mary’s boss • Anne is Mike’s boss

15. Visual and verbal pseudo-code • While letters in stack • Take a letter • Put a stamp on it • Put it in the ‘out tray’ Visual programming languages have a history of failure Data flow diagrams are defunct

16. Attaching images and words • Diexis and the diectic gesture • Can be a glance or a nod • Pre-speech • Shown to disambiguate verbal communications • Why the mouse is so powerful • The basis of shared environments

17. Attaching words to images- use gestalt laws

18. Integrated pictures and words moreEffective: Chandler and Sweller 1991

19. XPlane

21. Recap • Words for procedure, logic and abstract meanings/ images for pattern and structure

22. Linking images and wordsDeixis • Pointing is an elementary speech act. • Pointing links images and words • Put that (points) there (points) • Subject verb predicate

23. Other kinds of gestures • Beat gestures for emphasis • Verb gestures showing how to do something • McNeil Hand and mind

24. Issues in shared environments • Speech + Pointer + Visuals – most important components • Subtle ways of directing attention also important in meeting dynamics.

25. Examples of visual languages • Sanscrit • Petri-nets • Khoros

26. Count from 1 to 3 Sanscrit for i = 1 to 3 do

27. Petri Net languagePetri nets are stochastic – timed attributed (tokens on nodes, transitions)

28. Khoros

29. Peter