Presentation given at the London Knowledge Lab 10-10-2007

1. Philosophical Aspects of Multimedia Communication:Towards a New RationalityZsuzsanna Kondor Institute for Philosophical Research of the Hungarian Academy of Sciences Presentation given at theLondon Knowledge Lab 10-10-2007

2. Some preliminary remarks • The title • The evolution of the idea • Language • Communication • Representation • Cognition • Back to philosophy (epistemological relevance)

3. Thesis • Communications technology has a much more far-reaching effect on our thought than we used to suppose. • Multimodality provides the basis of a common conceptual background. • This possibility opens up the way for a new rationality.

4. Questions • Why recently? • How is it possible? • What does this mean as regards philosophical considerations? • Does it have any practical relevance at all?

5. Why recently? • István Hajnal and the Toronto Circle • Beyond verbal representation (pictorial vs. verbal representation) • Secondary literacy – Ubiquitous multimedia

6. István Hajnal and the Toronto Circle • Objectification • Primary orality • Literacy • Secondary orality

7. Objectification Alphabetical writing as the technical basis of rationality • “With the appearance of literacy, we know that what had been happening instinctively to this point in a human being's inner and outer life, now starts to become conscious. This sphere of life becomes objectified and abstracted; the human being projects this sphere in front of himself and examines it consciously and from the outside. There arises the possibility for methodical purposefulness, for the conscious handling of concepts and for combinational and complicated work.”(I. Hajnal, „Európai kultúrtörténet – írástörténet” (1932), in: F. Glatz (ed.), Technika, művelődés, Budapest 1993, p. 18) • ”Movements and sounds do disappear, still, humans can use them and their matter-relatedness to produce something that is objective, something that functions as an extrinsic intermediator for inner life.”(I. Hajnal, „Történelem és szociológia”, (1939), in: Glatz (ed.) op. cit. p. 203)

8. Objectification Objectification induces over-mechanisation • “Letters produced letters, writing produced writing. Purely speculative thinking was highly refined, even in the smallest of tasks, and more perfectly so since the more one-sidedly it functioned, the more it divided the professions into mechanical details, the more it exempted them from sensing the heavy material of life.” (Hajnal, „Évforduló” (1948), in: Glatz (ed.) op. cit. p. 449)

9. Orality • A primary oral culture is one which does not possess any knowledge of writing. • conservative and traditional. • “storage language” (usage of rhyme and rhythm, formulaic.) • special technology to weave ideas together, and to transmit awareness of the new facts of life: • The way of expression was • additive • redundant, and • the expressions and words used were very closely embedded in concrete situations. • Intercourse was • empathetic • participatory and • agonistic.

10. Literacy • The age of written records • transmission of ideas was liberated from certain restrictions • only text could provide context • potential to create concepts free of emotions, distanced from the human life-world and • possibility of the systematic analysis of ideas, • possibility of regarding events as linearly structured in time and thereby • possibility to recognize the eternally human. • ideas became remarkably easier to handle and to elaborate. • replacing the live situation with mute words required considerable intellectual effort, and caused plenty of difficulties.

11. Secondary orality • Secondary orality set in with telephone, radio, television,and the various kinds of sound tapes and similar electronic technologies. • “This new orality”, as Ong puts it, “has striking resemblances to the oldin its participatory mystique, its fostering of communal sense, its concentrationon the present moment, and even in its formulas. But it is essentiallya more deliberate and self-conscious orality based permanently onthe use of writing and print, which are essential for the manufacture andoperation of the equipment and for its use as well.”(Walter J. Ong, Orality and Literacy: The Technologizing of the Word, London: Methuen, 1982, p. 136)

12. Verbal representation Linearly ordered High degree of generality (far from experience) distancing Pictorial representation Holistic Close to mundane experience (immersive) Pictorial vs. verbal representation from the point of view of a scholar

13. William M. Ivins Jr.,Prints and Visual Communication, Cambridge, Massachusetts: Harvard University Press 1953 “Plato’s Ideas and Aristotle’s forms, essences, and definitions, are specimens of this transference of reality from the object to the exactly repeatable and therefore seemingly permanent verbal formula. An essence, in fact, is not part of the object but part of the definition. Also, I believe, the well-known notions of substance and attributable qualities can be derived from this operational dependence upon exactly repeatable verbal descriptions and definitions – for the very linear order in which words have to be used results in a syntactical time order analysis of qualities that actually are simultaneous and so intermingled and interrelated that no qualitycan be removed from one of the bundles of qualities we call objects without changing both it and all the other qualities.“ (p. 63)

14. Ancient Egyptian Holistic Complementary logic Cyclic comprehension of reality Creating – writing/drawing – seeing/grasping Western Analytic Dualisticlogic Linearity Disruption of the unity of creation and description of the world Comparing ancient Egyptian and Western metaphysics

15. Secondary literacy • Multimedia • Handheld devices • Secondary literacy is an epoch which is characterized by the rationality of literacy but due to the changes in communications technology allows for multimodal enhancement

16. How is it possible? • Cognitivebackground • Cognitive processing of visual percepts • Conceptual processing of perceptual information • Main differences between imagery and verbal processing from cognitive point of view

17. George Lakoff, Women, Fire, and Dangerous Things. What Categories Reveal about the Mind, Chicago and London: The University of Chicago Press, (1987) 1990

18. Pierre Jacob and Marc Jeannerod, Ways of seeing: The scope and limits of visual cognition, Oxford: Oxford Univ. Press (2003) 2004

19. Cognitive processing of visual percepts • Visual percepts serve as input to higher human cognitive processes, including memory, categorization, conceptual thought, and reasoning • Visuomotor representation is at the service of human action.

20. Cognitive processing of visual percepts • Conceptual transformation means a loss of some details (The content of visual perceptual representations is “both more fine-grained and informationally richer than the conceptual content of thoughts.” (Jacob et al. 22))

21. Excursion “In order to describe experience more fully language must be less precise. But greater imprecision brings more effectively into play the powers of inarticulate judgement required to resolve the ensuing indeterminacy of speech. So it is our personal participation that governs the richness of concrete experience to which our speech can refer. Only by the aid of this tacit coefficient could we ever say anything at all about experience – a conclusion I have reached already by showing that the process of denotation is itself unformalizable.” (M. Polanyi, Personal Knowledge - Towards a Post-Critical Philosophy, London: Routledge & Kegan Paul, 1962, pp. 86f.)

22. Cognitive processing of visual percepts • “Now once the visual percept has been turned into a thought by a process involving a selective elimination of information, further conceptual processing can yield a still more complex thought involving, not a two-place relation between pairs of objects, but a three-place relation between a pair of objects and an egocentric perspective.” (Jacob et al. 31)

23. Cognitive processing of visual percepts • Visuomotor representations are also egocentric, but in a strictly functional sense: the spatial reference is the actor’s body. • Verbal or conceptual transformation is egocentric, but this means a reflexive relation in accordance with some general goals.

24. Forerunning • “As perspectivally based cognitive representations, then, linguistic symbols are based not on the recording of direct sensory or motor experiences, as are the cognitive representations of other animal species and human infants, but rather on the ways individuals choose to construe things out of a number of other ways they might have construed them, as embodied in the other available linguistic symbols that they might have chosen, but did not.” (Michael Tomasello, The Cultural Origins of Human Cognition, Cambridge, MA: Harvard University Press, 1999, p. 9)

25. Excursion • Social brain hypothesis (Robin Dunbar) • Cognitive evolution (Merlin Donald)

26. Conceptual processing of perceptualinformation • “conceptual structure is meaningful because it is embodied, that is, it arises from and is tied to, our preconceptual bodily experiences. In short, conceptual structure exists and is understood because preconceptual structures exist and are understood.” (Lakoff, Ibid. p. 267)

27. Conceptual processing of perceptual information • thepreconceptual level is determined by kinesthetic image schematicstructures (They originate from everyday bodily experiences and mean certain, relatively simple spatial relations such as: containers, paths (movement in space with a starting and ending point including direction), links (regarding security and its source), forces, balance, different kinds of orientation (up-down, front-back, part-whole, center-periphery, etc. ) • and basic-level categories which • do not overload memory capacity • are closely bound to motor activity. • emerge as gestalt (they emerge in overall general forms and at the same time their structures are identifiable as well)

28. Conceptual processing of perceptualinformation Our conceptualizing capacity consists of: The ability to form symbolic structures that correlate with preconceptual structures in our everyday experience. Such symbolic structures are basic-level and image-schematic concepts. (That is, we can speak about basic-level categories in a twofold sense: categories that emerge out of the network of motor activities, and categories that are in accordance with the former and refer to concepts in linguistic form.)The ability to project metaphorically from structures in the physical domain to structures in abstract domains, constrained by other structural correlations between the physical and the abstract domains. This accounts for our capacity to reason about abstract domains such as quantity and purpose. The ability to form complex concepts and general categories using image schemas as structuring devices. This allows us to construct complex event structures and taxonomies with superordinate categories.(George Lakoff, Ibid. p. 281)

29. image schemas are crucially important since they play “two roles: They are concepts that have directly understood structures of their own, and they are used metaphorically to structure other complex concepts.”(Lakoff, Ibid. p. 283)

30. Idealised cognitive models • Image-schematic • Bodily experiences • Spatial relations • Metaphoric (describes ways of mapping) • Metonymic (describes ways of mapping) • Propositional • no use of imaginative devices as inthe previous models • elements are basic-level and concepts characterized by cognitive models of other types • Symbolic • linguistic elements are associated with conceptual elements in ICM

31. “Propositional models have an objectivist flavour, since they contain entities, with their properties and the relations holding among them. … It seems to me that when we understand our experience by projecting propositional models onto it, we are imposing an objectivist structure on it.” (Lakoff, Ibid. p. 285)

32. Imagery closely related to motor activity richin iconic content Verbal processing not as closely related to motor activity narrower domain propositional linear Main differences between imagery and verbal processing from a cognitive point of view

33. “Systems of thought emerge from this reflective activity [i.e.,representational redescription] because self-observation employs all ofthe categorization and analytic skills that are employed in perceiving,understanding, and categorizing the outside world – in effect the subjectperceives, understands, and categorizes her own cognition facilitated bythe fact that it is expressed externally in language.”(Tomasello, Ibid. p. 195)

34. Summing up I. • Language inherently has a distancing effect • and tendentious character. • Written language radicalizes these features. • Imagery has strong immersive potential.

35. Summing up II. • using multimedia means a nearly direct access to others’ experiences • decreasing dominance of verbal processing • access to mundane particularities • the conceptual background is changing due to ubiquitous multimodality

36. What does this mean as regards philosophical considerations? • The birth and evolution of atomistic individualism/objectivism • A turning-point in the fields of ontology, epistemology, philosophy of language, and even in metaphysics • Convergence of long-established opposed traditions

37. The birth and evolution of atomistic individualism/objectivism • Heraclitus • Plato • Descartes

38. A turn in the fields of ontology, epistemology, philosophy of language, and even in metaphysics • Bergson • Dewey • Heidegger • Wittgenstein

39. Bergson • ”The mistake of ordinary dualism is that it starts from the spatial point of view: it puts, on the one hand, matter with its modifications, in space; on the other hand, it places unextended sensations in consciousness. Hence the impossibility of understanding how the spirit acts upon the body or the body upon the spirit. … But hence also the impossibility of constituting either a psychology of memory or a metaphysic of matter.”(Henri Bergson, Matter and Memory, New York: Zone Books, (First published in French, 1896) 1991p. 220) • “The reality of matter consists in the totality of its elements and their actions of every kind. Our representation of matter is the measure of our possible action upon bodies.”(Ibid. p. 38)

40. Dewey • “All of these separations culminate in one between knowing and doing, theory and practice, between mind as the end and spirit of action and the body as its organ and means.”(J. Dewey, Democracy and Education. The Middle Works of John Dewey 1899-1924. Vol. 9, Carbondale and Edwardsville : Southern Illinois Univ. Press 1985. p. 346) • Relying on the findings of physiology and psychology, Dewey concludes that “[n]o one who has realized the full force of the facts of the connection of knowing with the nervous system and of the nervous system with the readjusting of activity continuously to meet new conditions, will doubt that knowing has to do with reorganizing activity, instead of being something isolated from all activity, complete on its own account”.(Ibid. p. 347) • Considering the results of biology and contemplating the “doctrine of evolution”, Dewey claims that “the living creature is a part of the world”, which implies a new comprehension of knowledge. “If the living, experiencing being is an intimate participant in the activities in the world to which it belongs, then knowledge is a mode of participation, valuable in the degree in which it is effective. It cannot be the idle view of an unconcerned spectator.”(Ibid.)

41. Heidegger • Being-in-the-world • Ready-at-hand vs. present-at-hand • Referential totality

42. Wittgenstein • “[O]ur language is tempting us to draw some misleading analogy. This should remind us of the case when the popular scientist appeared to have shown us that the floor which we stand on is not really solid because it is made up of electrons”. (Ludwig Wittgenstein, The Blue and Brown Books. Preliminary Studies for the Philosophical Investigations, Oxford: Basil Blackwell (1958) 1984, p. 48)– different language games • “The problem may seem simple, but its extreme difficulty is due to the fascination which the analogy between two similar structures in our language can exert on us.”(Ibid. p. 26) – analogy • „There are the sounds of the words, and all sorts of bodily sensations connected with gesture and intonation. Where we are liable to go wrong is in supposing that sensations connected with words are somehow ‘in the mind’. The phrase ‘in the mind’ has caused more confusion than almost any other in philosophy.” (Ludwig Wittgenstein, Wittgenstein’s Lectures. Cambridge, 1932-1935 Oxford: Basil Blackwell (1979), p. 114) – one painful example

43. Atomistic individualism dualism of mind and body focus on abstract reasoning the human mind works as a machine which manipulates symbols symbols are internal representations of external reality (correspondence) Towards a new rationality embeddedness focus on action and perception emphasis on the activity of the brain language in a new light the importance of imagery Summing up thedifferences

44. Does it have any practical relevance at all? • Conceptual framework • Institutional setting, especially education

45. Conceptual framework Think of the question of reality in the mobile age. Is there any reason to distinguish virtual and real in the case of a location-sensitive and at the same time location-independent device? What should be considered real, the given circumstances and the concomitant tasks, or those engendered by the mediated-from-afar via mobile devices? Is there any relevance to the distinction between real and virtual when considering function and effect?

46. Conceptual framework The intuitive idea of so-called local realism (which Einstein shares as well) suggests that “a particle cannot be instantly influenced by a distant event, and that its properties exist independently of any measurements”.(M. Brooks, ”Reality Check”, in: New Scientist 23 June 2007, p. 31)Although the implications of quantum physics suggest that “[t]he world could not be local and real”, poses the question of whether it is either local or real. The various experiments supporting quantum theory led to the conviction that it is possible that “there is nothing inherently real about the properties of an object that we measure. In other words, measuring those properties is what brings them into existence. …So does the universe exist independently of measurements?”(Ibid. pp. 32) ”We need to rethink and radically revised our basic physical concepts beforewe make the next big breakthrough in physics”(Ibid. p. 32)

47. Institutional setting, especially education • “[E]verywhere the school system has the same structure, and everywhere its hidden curriculum has the same effect. …]This identity of theschool system forces us to recognize the profound world-wide identity of myth, mode of production, and method of social control, despite the great variety of mythologies in which the myth finds expression.” (Ivan lllich, Deschooling Society, Marion Boyars, London, (1971) 1978 p. 74) That is, the influence of literacy is recognisable everywhere in the world.

48. Taming vs. integrating new technology • “instead of the computer cutting across the disciplines in the subjects, the computer is now confined to a computer room and it is a subject of its own, taken out of the mainstream of the learning environment. There’s now a specialized computer teacher. There’s a curriculum even for the study of the computer. It has been normalized by the system; it has been tamed. That’s not the only way in which it can be tamed.” (Papert, S. 1998 Child Power: Keys to the New Learning of the Digital Century. http://www.connectedfamily.com/main_alt.html)

49. Thanks for your attention kondor@webmail.phil-inst.hu