Dual Coding of Memory in Classical Texts

1. Dual Coding of Memory in Classical Texts Professor Han-liang ChangFudan University, ShanghaiNational Taiwan University, Taipei 2nd International Conference in Code Biology 2015 Jena, Germany 18th June 2015

2. What the paper is about? • This paper discusses the theory of dual codingand its application to information processing in memory • by rereading a few classical texts which deal with the literary function and philosophical implications of memory, with particular reference to the relationship between linguistic and visual signs, • such as the Platonic dialogues, Timaeus, Symposium and Phaedrus, and the Aristotelian treatise, On Memory and Recollection. • Given the time limit, the discussion of classical texts will be cut off from my presentation, and the allowed time will be used to give a general background of dual coding.

3. What is the argument? • Remote as they may seem, these classical texts demonstrate how information in memory may be claimed to have been dually encoded, stored and retrieved as both ‘visual’ and verbal representations. • The texts’ dual coding may have neuro-cognitive justification, but once when language’s meta-semiotic function as interpreting system is activated, all the representations, be they verbal or non-verbal, remain linguistic and thus compromising coding’s duality. • Finally, the philosophical insights in ancient texts are yet to be reconciled with recent advances in cognitive sciences.

4. A dilemma facing memory scientists Given the fact that the concept of memory has been deeply rooted since pre-scientific times, how should a memory scientist translate the concept of memory into terms that at once keep its philosophical meanings and yet permit analysis of its neuronal substrates and engineering principles?

5. Some of the keywords to be defined • What is ‘memory’? • What is ‘coding’ and what is ‘dual coding’? • What is ‘representation’? • At first glance, these terms look like vague commonalities, but they can be variously defined according to the disciplines which shape different bodies of knowledge.

6. What is memory? • Memory is the location where information is kept, as in a storehouse, or memory store. • Memory refers to the thing that holds the contents of experience, as in a memory trace or engram. • Memory is the mental process used to acquire (learn), store, or retrieve (remember) information of all sorts. (Radavansky 2011) • The tripartite definition above that covers three functions and/or loci of the brain raises more questions than it can claim to answer.

7. Metaphors of memory (Radvansky, 2011: 3) METAPHOR EXAMPLES Recorder of Experience Wax tablet, writing pad, record player, video camera Storage Locations House, library, dictionary Interconnections Switch board, network Jumbled Storage Bird in an aviary, junk drawer Temporal Availability Conveyor belt Content Addressability Lock and key Forgetting of Details Leaky bucket Reconstruction Building a dinosaur from fossils Active Processing Computer program

8. A few theoretical questions • Question 1: Metaphor notwithstanding, is there a ‘storehouse’ in the brain to store an infinite number of traces? • Question 2: Are there traces? And what are they? • Question 3: Are there both traces and the mental process, or just the latter? • Question 4: How can interpretations of mental process bypass the interference of subjective psycho-phenomenalism given that memory is always solipsistic, pour soi and ensoi?

9. Similar questions in the descending order of ‘psychologism’ can be seen in another definition • The capacity to encode, store, consolidate and retrieve information (i.e. memory processing) • A hypothetical store in which memory is held (i.e. a physical device in the brain) • The information in such a store (i.e. information content) • The process of retrieving information from a store (i.e. the act of remembering) (Tulving) • An individual’s phenomenal awareness of remembering something (i.e. self-reflectivity) (Tulving) (R. G.M. Morris 2007)

10. Memory systems and types of consciousness • The definitions above are inscribed in the various branches of Memory Studies, Memory Research or the so-called Memory Science. • For example, the meta-structure of self-reflexivity, i.e., ‘consciousness’ of memory, which may baffle the neuro-physiologist, serves to distinguish human memory from animal memories, and gives rise to‘autobiographical memory’, i.e., a conceptual entity in ‘episodic’ memory before its being textualised in a literary genre, such as autobiography. • This approach in terms of consciousness is also characteristic of phenomenological hermeneutics.

11. Distinction between engram & process is traceable to the classical tradition • The Platonic-Aristotelianpairs of νόησις (nóesis, ‘process of thinking’) versus νόημα (nóema, ‘that which is thought of’), φαντασις or φαντασια (phantasis or phantasia, ‘the process of imagining’) versus φαντασμα (phantasma, ‘that which is imagined’), and for our purpose here, μνήμησις (mnémesis, the action/process of remembering) versus μνημόνευμα (mnemóneuma, thing remembered) suggest a similar distinction between ‘mental process’ and ‘memory trace’ outlined above. • Witness how a leading cognitive psychologist on Memory Studies in our times, EndelTulving, postulates his typology of memory on the basis of νόησις, a concept juxtaposed with its twin, νόημα,which is made famous by Husserl but ultimately rooted in Plato.

12. A sample typology • Cognitive psychology’s interest in consciousness has allowed Tulvingto loan, albeit loosely, from philosophical discourse the term ‘nóesis’ in formulating his memory systems as follows: • (1) anoetic (non-knowing), (2) noetic (knowing) and (3) autonoetic (self-knowing) consciousness, • which correspond roughly to three kinds of memory: (a) procedural, (b) semantic and (c) episodic (1983, 1985, 2005). • This typology has dated and made obsolete Ebbinghaus’s short-term and long-term memory in 1885.

13. Correspondence of three memory systems and three kinds of consciousness MEMORY SYSTEM CONSCIOUSNESS EPISODIC  AUTONOETIC SEMANTIC  NOETIC PROCEDURAL  ANOETIC (From EndelTulving 1985. The arrows mean ‘implications’.)

14. Two systems rather than three? • Since the procedural memory, such as the skill of bicycle-riding or swimming, is governed more by the cerebellum than the hippocampus, only two kinds of memory that process information can be said to exist. They are termed, collectively, ‘propositional’ memory. • And these two types of propositional memory, the episodic and the semantic, enjoy a strange ‘joint-coding’ (i.e., dual-coding) when taking flight.

15. How Tulving defines the two terms • Tulvingdefines the episodic memory as ‘the recording and subsequent retrieval of memories of personal happenings and doings’ and the semantic as ‘the knowledge of the world that is independent of a person’s identity and past.’ • One is not sure if the distinction is essential and functional. • The next slide will give a long quotation from his now classical Elements of episodic memory, Oxford: Clarendon, 1983.

16. According to Tulving, ‘… the registration of information into the episodic system is more direct than that into the semantic system. The episodic system is capable of recording and retaining information about perceptible properties of stimuli that can be apprehended immediately by the senses. It is in this sense that we could say that registration in the episodic system is ‘experiential’. The information registered in the semantic system, on the other hand, is frequently given to us in symbolic form, expressed in natural or some other language. What is registered in the semantic system is not perceptible properties of the input signals, but rather the information about the cognitive referents of the signals. Episodic memory registers what we may call immediate, or first-hand knowledge, whereas semantic memory records mediate or second-hand knowledge (1983: 41).

17. How are episodic and semantic memories coded? • By so doing, the author has turned the episodic memory into nóemaand the semantic memory nóesis, the former being content-oriented, and the latter expression-oriented. • In semiotic terms, then, their relationship becomes one between meta-semiotics and object-semiotics, or, as far as ‘propositional’ goes, between metalanguage and object-language. • This last pair of terms was made famous by Tarski and Carnap in the early 1930s, and the latter having been Frege’s student here at Jena University before emigrating to Chicago.

18. Typologies of memory • Short-term and long-term memory (Ebbinghaus, 1885) • Episodic, semantic, and procedural memory (Tulving, 1983) • Explicit and implicit memory (Schacter, 1987) • PRS – ‘perceptually-based representational system’ (Schacter and Tulving, 1994) • Memory from external sources versus memory from internal sources (Johnson & Raye, 1981).

19. What other questions can be raised and should be answered? • There are quite a few other questions one ought to address, e.g., semiosic coding as distinguished from hermeneutical deciphering, the relationship between coding (the ‘translation’ of one language into another) and representation (the mapping of event space in cognitive/neuronal space), etc. • For our purpose, the crucial question remains the possibility and feasibility of dual coding, andwhether the relationship between visual and verbal coding is meta-semiotic, i.e., one mapping the other, or inter-semiotic, i.e., they are mapped by a third entity.

20. Problems of visual and verbal dual coding • As early as the 1960s, Roland Barthes already raised the question in his analysis of the ‘rhetoric’ of ‘image’: Can visual representation be doubly articulated like language? • The question can be otherwise posed: Can analogical ‘language’ produce true systems of signs? • Or, is it possible to conceive of an analogical ‘code’, as opposed to a digital one which is founded on a combinatory systems of digital units, such as phonemes? • Allan Paivio (2007) argues that Marcel Proust’s example of tasting a madeleine cookie to evoke, unwittingly, memory shows that memory can be gustatorially (i.e., non-verbally) encoded, but, one may retort that it’s eventually verbally recoded (i.e. via Proust’s writing).

21. Introducing the Dual Coding Theory • The Dual Coding Theory or Dual Code Theory (DCT), launched by Paivio (1969, 1986, 2007), is a theory of memory which suggests that visual and verbal information act as two distinctive systems and humans are capable of storing information in either visual codes and/or verbal codes. • The theory has been so popular in cognitive sciences that its usage has been stretched from its original formulation and applied to any message that is doubly or dually encoded. • Thus a memory trace in Tulving that can be both read as ‘episodic’ and ‘semantic’ is dually coded.

22. Imagery and language function as alternative coding systems • Paivio’s assumption is that ‘images and verbal processes are assumed to function as alternative coding systems, or modes of symbolic representation’. (1969: 243) • And the two interconnected processes perform independent functions in cooperative activity (1986, 2007). • The verbal and image systems are correlated, as one can think of the mental image of an automobile and then describe it in words, or read or listen to words and then form a mental image.

23. How does the DCT function? • The processing of DCT involvesthree procedures, in Paivio’s words: • ‘representational’ -- the direct activation of verbal or non-verbal representations; • ‘referential’-- the activation of the verbal system by the nonverbal system or vice versa, and, finally, • ‘associative’-- the activation of representations within the same verbal or nonverbal system. • In other words, both the linguistic sign and the ‘analogical’ or ‘iconic’ sign are used and their mutual transcoding is required.

24. More recently, Paivio has refined the processing as including five aspects: 1. verbal and nonverbal symbolic systems that cut across sensorimotor systems; 2.the representational units of each system; 3.connections and activation processes within and between systems; 4.organizational and transformational processes; 5.conscious versus unconscious processes (2007).

25. How Paivio’s DCT got started in the 1960s? • It’s important to know that Paivio’s DCT evolved empirically, from specific experiments on the role of imagery in associative learning (Paivio, 1963, 1965). • As such, the approach departed from the verbal learning and memory tradition of the time. • And it remains questionable whether the empirical research could expand and lead to a general theory of memory and cognition.

26. Structural model of dual coding theory (A. Paivio, Mental representations: A dual coding model. Oxford: 1986)

27. Dual processes in memory retrieval • Process 1: the ‘familiarity’ process that uses a signal detection-like principle in which information is identified as old (remembered) when it exceeds some threshold. • Process 2: The ‘recollection’ process that involves the conscious retrieval of different components associated with the to-be-retrieved information. • The 2 processes operate simultaneously but independently of each other. E.g., You see a familiar face but do not remember the person’s name. (Radavansky 216-218)

28. Information contents (traces) have to be coded • The act of remembering and self-reflection suggests the semiotic subject’s discursive practice of encoding and decoding. • Though smacking ‘psychologism’, it belongs to semiotics of the second-order, involving the agent’s conscious and active participation. • Beyond the borders of folk psychology, or rather, underlying it, coding can be regarded as a series of autotelic but involuntary molecular and cellular mechanisms and neuronal processes. • Therefore, it seems there is an unbridgeable gap between the phenomenology of memory and the implicit biological forms.

29. References Morris, Richard G.M. 2007. Memory: Distinctions and dilemma.Roediger III, Dudai & Fitzpatrick, eds. Science of memory: Concepts. Oxford: Oxford University Press. 29-34. Paivio, Allan 1969. Mental imagery in associative learning and memory. Psychological Review, 76:3,241-263. Paivio, Allan1986. Mental representations: A dual coding approach. New York: Oxford UniversityPress. Paivio, Allan2007. Mind and its evolution: A dual coding theoretical approach. Mahwah, N.J.: Lawrence Erlbaum Associates. Radavansky, Gabriel 2011. Human memory 2nd ed. Boston: Allyn & Bacon. Tulving, Endel 1983. Elements of episodic memory. Oxford: Clarendon Press. Tulving, Endel1985. How many memories are there? American Psychologist 40: 385-398. Tulving, Endel2005. Episodic memory and autonoesis: Uniquely human? In: H. Trrace & J. Metcalfe (Eds.) The missing link in cognition: Evolution of self- knowing consciousness. New York: Oxford University Press. 3-56.

30. A ton of language is still weightier than an ounce of mathematics! • Thank you!