Translating between multiple representations: discussion

1. Translating between multiple representations:discussion EARLI Symposium Padua, August, 2003 Richard Cox HCT Group, University of Sussex richc@sussex.ac.uk

2. Overview • Definitions, framing issues, the translation ‘problem space’ • How do learners come to understand relationships between representations? • How we might discriminate between occasions when a learner is engaged in ER comprehension activity and when s/he is successfully reasoning with an ER ? • Supporting users at various levels of expertise

3. Framing issues… • 1. Translation “..convey..from one place to another..”, “..turn from one language into another retaining the meaning” (OED) • 2. Consider dimensions in which translation possible between (multiple) ERs - extending Palmer (1978) to the 2 ER case ...

4. Representedworld (Palmer, 78) a b c d

5. Representing world (ER) a b c d taller than --> longer than

6. Representing worlds ER 2 ER 1 a a c d b c d b taller than --> shorter than taller than--> longer than

7. a b c d a c d b Both ERs represent same aspects of represented world - YES Same aspect of representation doing the mapping - YES

8. b d a b c d a c taller than -> longer than taller than --> points to Both ERs represent same aspects of represented world - YES Same aspect of representation doing the mapping - NO

9. a a b c d b c d WIDER than -> longer than taller than -> longer than Both ERs represent same aspects of represented world - NO Same aspect of representation doing the mapping - YES

10. b d c d a b a c taller than --> points to WIDER than - -> LARGER than Both ERs represent same aspects of represented world - NO Same aspect of representation doing the mapping - NO

11. same aspect of world? YES NO sameERmapping? YES NO

12. a 3rd dimension - multimodality... LINGUISTIC GRAPHICAL

13. and a 4th, 5th, 6th... • static vs dynamic ERs (interactive, animated...) • not to mention other factors... • degree of abstraction of ER(s) • type of graphical representation (picture, non-picture) • number of ERs co-present in display • 3D, VR ... • combinatorial explosion of translation routes!

14. How do learners come to understand relationships between representations? • Depends on relationships and translation ‘route’ • If same aspect of world represented - can explore ER-to-ER links (dynalinks), perhaps without reference to the world and maybe solely via other (constraining) ERs • If different aspects of world represented - then ER-to-ER translation is usually via represented world

15. How do learners come to understand relationships between representations? • Learning may not be so much `more’ or `less’ but different in mode • exploration of dynalinks, constraining ERs and correlated displays ---> implicit learning (eg. Berry & Broadbent, 1984) • need to consider learning outcomes ... how important is need for explicit (verbalizable) knowledge? if implicit knowledge acquired - assess differently? • interaction of methodology with learning outcome - `think aloud’ likely to keep knowledge and reasoning explicit (with extra cognitive load perhaps)

16. `Staring at’ versus using - telling the difference.. • how can we discriminate between when a learner is engaged in a) ER comprehension activity and b) reasoning with the ER ? • 1. correlate ER behaviour with performance (as many researchers do!) here rich process data very useful - innovative methodologies pay off ... • DEMIST: logging, dynalinking, analyser • SDE: RFV, think aloud and logging • SimQuest: think-aloud and cognitive load probes

17. `Staring at’ versus using - telling the difference.. 2. Assess learner’s background knowledge of ERs beforehand • examine mental organisation of ER knowledge - category organisation differs in poor versus better reasoners (Cox & Grawemeyer, 2003) and in people with different backgrounds (Lohse et al 1994)

18. Supporting users at various levels of expertise • The `intermediate learner’ effect (du Boulay et al., Seufert) • some knowledge (of ERs, of domain) necessary to benefit from support • support should segue into integration thru’ degrees of compartmentalisation and complexity of domain k. and ERs • need to know more about what `knowing an ER’ actually means - characterise partial knowledge and misconceptions (eg viewing graphs as pictures der Meij & de Tong) - look at different levels of cognitive processing system - ie. perceptual, semantic memory organisation and output levels

19. Supporting users at various levels of expertise • by making aspects of world that are modelled explicit (very nice colour coding in der Meij & de Jong) assists learner to assess redundancy level of MERs in display, directs attention in dynalinking and exploration • sometimes a tension between learner-centred design and traditional ‘ease of use’ HCI ... ask who is system for, should there be different versions for different users?

20. References p.1of 2 • Berry, D.C. & Broadbent, D.E. (1984) On the relationship between task performance and associated verbalizable knowledge. The Quarterly Journal of Experimental Psychology, 36a, 209-231. • Cox, R. & Grawemeyer, B. (2003) The mental organisation of external representations. Proceeding of the European Cognitive Science conference (EuroCogSci03), Osnabruck, Sep.

21. References p. 2 of 2 • Lohse, G.L., Biolsi, K., Walker, N. & Rueter, H. (1994) A classification of visual representations. Communications of the ACM, 37(12), 36-49. • Palmer, S.E. (1978) Fundamental aspects of cognitive representation. In E. Rosch & B.B.Lloyd (eds) Cognition and categorization. Hillsdale, NJ: Lawrence Erlbaum Associates. • Other works cited were papers presented at this symposium