Unit 6: Word and image

1. Antar Abdellah Unit 6: Word and image

2. Introduction • This chapter deals with a range of printed literary texts which use visual communication as a meaning-making resource. Different aspects of texts, such as typography and images will be considered with a view to understanding how their analysis can illuminate aspects of literary creativity. • semiotics is a ‘literary studies’ approach, and a look at what postmodern theory can illuminate about visual playfulness in literature. • Multimodality, it's the use of different forms of communication in a single text, or texts utilizing image and words, from the genres of poetry and narrative including different genres , such as advertisement texts

3. 1 Semiotics/Semiology

4. Signifier & Signified Signs • B-Signifier & Signified Signs ; Syntagmatic & Paradigmatic Signs • In his analysis of linguistic signs, Ferdinand de Saussure showed that there are two components to every sign. One is signifier : the vehicle which expresses the sign, • The other part of the sign is called the ‘signified’: the concept which the signifier calls forth when we perceive it. • This horizontal movement is called the ‘syntagmatic’ aspect of the sentence. If we reverse the order into ‘The man bites the dog’, the meaning is obviously different. • These lists of signs are called ‘paradigms’. We could replace ‘dog’ with ‘cat’ or ‘tiger’, and replace ‘bites’ with ‘licks’ or ‘kicks’ or ‘chews’.

5. Visual signs • C- Visual signs • Charles Peirce[1958]. For him, the sign also comprises signifier and signified, but he divides signs themselves into three types. • 1-Symbolic signs are those where the signifier does not resemble the signified , and their meaning is arbitrary and they are culturally learnt and understood [such as the use of the color red for a Stop sign]. • 2-Iconic signs are those where a resemblance can be perceived, such as a portrait of someone. • 3-Indexical signs often have some kind of causal relationship between signifier and signified: • smoke is an index of fire.

6. Denote / Connote • Because we use signs to describe and interpret the world, their function is to ‘denote’ something, to label it. • The linguistic sign ‘Rolls-Royce’ denotes a particular trend of car, But along with the denotative, or labeling function of these signs to communicate a fact, come some extra associations which are called ‘connotations’. Because Rolls-Royce cars are expensive and luxurious, they can be used to connote wealth and luxury. • The connotations of a sign are often multiple and unstable.

7. D-Semiotic Domains • D-Semiotic Domains • A Semiotic domain is“words, symbols, images, and artifacts have meanings that are specific to particular semiotic domains and particular situations… (James Paul Gee, 2003:25). • Here are some examples of semiotic domains: cellular biology, postmodern literary criticism, first-person- shooter video games, high-fashion advertisements, Roman Catholic theology, modernist painting, midwifery, rap music. • In any semiotic domain, words, symbols, images, and/or artifacts have meanings and combine together due to design grammar of the domain to take on complex meanings.

8. 2-Semiotics and paralanguage in literature • Linguists generally define paralanguage as features of speech medium which are combined with words to create additional meaning, such as intonation, pitch, tempo and tone. • In face-to-face conversation, or in a stage production, visual non-linguistic features such as gesture, facial expression or movement may also be included in paralanguage. • Paralinguistic features do not occur in written texts. But literature and poetry are in fact perfectly capable of utilizing paralinguistic signs, and semiotics gives us a way of analyzing these.

9. ARURMS ---------- (‘you are up in arms over it’) IT TRAVEL ---------- (‘travel over C’s = travel overseas) ccccccccc

11. A-Concrete poetry • Concrete poetry or pattern poetry’ is a type of poetry in which the lines are arranged in a specific shape on the page in a meaningful way. • Mosaics are amongst the earliest examples of it. The poems of E.E. cummings could be considered concrete, as the spatial layout is significant. • In view of David Crystal [ 1987, p. 75], concrete poetry refers to poems in which the visual shape is paramount

14. 3 Multimodality in Literary texts

15. Advertisements are a simple example of such texts, in their common employment of image, color, layout, text, and typography for attention-grabbing and persuasive effects. Printed multimodal literary fictions employ a similar range of codes, being limited to the printed novel form, but as part of communication of a narrative. These text interact with images, symbols, shapes and colors. Accordingly, multimodal stylistics aims to combine multimodal theory and methodology with that of literary stylistics in an attempt to systematically take into consideration all modes involved in literary meaning-making. Others, often more modern texts, employ paralanguage ,i.e. images and words , and these kinds of texts are the subject of an increasing academic interest in how words and pictures are used in books, particularly those for children [Nodelman, 1988]

16. A- Text world : • A- Text world : • In view of Elena Semino [1997], when we read a narrative, we create a ‘text world’ as ‘the context, scenario or type of reality that is evoked in our minds during reading and that (we conclude) is referred to by the text’. • Text world theory is based on the idea of cognitive linguistics that when humans communicate they visualize a kind of scene, space or world in their mind, and it provides a framework to analyze discourse in any situation. The basic principle of Text World Theoryis that humans process and understand all discourse by constructing mental representations of it in their minds ,and it provides the analytical tools for the systematic examination and discussion of these mental representations, or text-worlds. • In view of Joanna Gavins [ 2007], the world-building elements are time, location, enactors and objects, while the function-advancing propositions are verbs which can describe the enactors in relation to each other and the objects surrounding them, propel the action, or present an argument of some kind.

17. B- The literary functions of images in fictions • 1- Connotative function • Other uses of images in fiction seem to function at the level of connotation rather than denotation: they add affective meaning but don’t seem to have an explicitly narrative function. Denotation is the direct meaning of a word, sign, or image. Connotation is a second level of meaning that is conveyed or suggested in addition to the denotation. Connotation - is about image meaning. • Connotation is related to describing all the things we associate with the image as a whole as well as its elements. The process may involve looking at objects and subjects in both foreground and background, at the colors used, at expressions, at gestures (body language), at clothing, hairstyle etc….

18. C- Picturebooks and Multimodality • In picturebooks, the images assume a central role in telling the story and creating the central meaning of the narrative. This is achieved in a variety of ways. Images are often wholly integrated with the words, and layout, image and typography are intertwined. • In reading B Clare Bradford [2001] looks at combinations of word and image in postcolonial literature. She is concerned with representations, both linguistic and visual, of racial politics in children’s hooks • Clare Bradford’s texts provide further evidence of such texts emerging from postcolonial contexts. In Nikolajeva and Scott’s terms then, picturebooks are counterpointing dynamic as long as additional meanings are generated by the interaction of words and images.

19. 4 Postmodern multimodal literature • Postmodernity or postmodernism can be defined as a cultural condition of living in an increasingly technologically orientated society, with lower levels of trust in authority and ‘truth’ than previously, where the meaning of things is unstable and open to interpretation. Postmodernism, as it relates to literature, refers to texts that can be seen to represent such instability and unreliability. • It is largely a reaction to the assumed certainty of scientific, or objective, efforts to explain reality. In essence, it stems from a recognition that reality is not simply mirrored in human understanding of it, but rather, is constructed as the mind tries to understand its own particular and personal reality • Postmodernism is "post" because it denies the existence of any ultimate principles, and it lacks the optimism of there being a scientific, philosophical, or religious.

20. features of postmodernism • A-IndeterminacyIn the postmodern world we are less sure about the nature of objective reality, our own selves and the products of our hands and minds than we used to be. The more we know about the world the less stable and certain it seems. • Literature has responded to such developments by placing an increased emphasis upon undecideable outcomes and irresolvable dilemmas. In place of the obscurities of the modernist text we now have the indeterminacies of the postmodern text. Readers of the postmodern are uncertain about which way to turn. • B-FragmentationPostmodernism suspects totalization, attempts to unify and synthesize being considered the imposition of an ideological, and thus spurious, order. The metaphor of ‘unjustified margins’ is a useful one for it suggests graphically the refusal to tidy up loose ends. Rather than attempt to pull everything into shape at the last minute, and thus create an illusion of order where none in fact exists, the postmodern artist or writer is likely to let the ends remain loose and visible.

21. C-DecanonizationIn view of Jean-FranoisLyotard[1984],the most widely disseminated principle of postmodernism is that the governing narratives of our culture have broken down . We are less likely to trust blindly in authority than were the citizens of previous ages. Fewer people believe the overarching stories we tell ourselves about ultimate values, truth, progress and reason because the authorities that underwrote such stories are no longer viable. • D-IronyIndeterminacy, fragmentation and decanonization inevitably lead to irony. High Street and television advertisements are today so much more ‘knowing’ than they were and the consumer at whom they are aimed is expected to understand the allusions and get the jokes in the act of reading the text.

22. E-HybridizationThe dissolving of boundaries, the fragmentation of wholes, the flattening out of differences between high and low are all held to be characteristic of our postmodern condition, and they have all contributed to the rise of bizarre hybrid genres and artifacts. • F-Performance and participation • The more authorities dissolve and the more authors and artists cancel responsibility for leading readers and viewers towards sense and meaning, then the more readers have to write the text they read. • Much art is now conceived in terms of performance and participation, the role of the viewer or participant in the process being deemed as important as any product.

23. 5 Valuing multimodal texts So on what basis do we as readers judge multimodal literature as ‘good’ or ‘bad’? • Judgments in this regard are necessarily subjective. Perhaps we learn to ascribe value to multimodal literary texts depending on the same notions of ‘good’ and ‘bad’, or ‘high’ and ‘low’, discussed in earlier chapters. In a sense, then, our aesthetic judgments depend at least in part on what we have learned to value in our society and culture. • It has been argued by Richard Bradford that ‘good literature’ is distinguished from ‘bad literature’ by the extent to which form and meaning are held in balance, a complex interplay that allows a poem to resist closure. • Basically, we take meaning from texts depending on what they are made of , either pen on notepaper or graffiti on a wall, and on where we encounter them. Overall, ‘‘Materiality’ ,or the stuff that texts are made of ,can be seen as significant in terms of literary value.

24. Conclusion • Conclusion This chapter has explored some of the many ways in which authors and illustrators can use visual communication in their work. There is a huge range of possible signifiers, from non-standard punctuation to concrete poetry, to whole multimodal books where an understanding of the visual meaning is just as important, if not more so, than the words. • Multimodal literature has been investigated through different approaches, from the Formalist or inherency-based to the more sociocultural. • Semiotics is also used to study multimodal literature; and it relies on an understanding of social and cultural connotations to find the meaning of the linguistic or visual sign in the text. As with language, visual elements of a text are often intertextual. These may be allusions to other visual texts, or deliberate connections made across semiotic modes such as in punning.

25. Antar Abdellah Unit 7: Literature and technology

26. Introduction • Introduction This chapter focuses on the impact of technology on literature. • It will consider how technology may afford new possibilities for literary creativity. • Basically, the term ‘technology’ is used in this chapter to mean relatively recent technologies such as filmand the internet. • It will consider novels adapted for CD-ROM and film, an oral narrative made into a movie by a group of schoolchildren, literature created as electronic hyper fiction.

27. 1. Adapting literature for technology • How can a passage of a novel be adapted for new technology, and how does the reader construct meaning while interacting with this visual text? • Adaptation from one medium to another is fundamentally the art of making choices. • Adapting a written literary text into a visual text or a movie is the transfer of a written work, in whole or in part, to a feature film. It is a type of derivative work. • Because visual signs are necessarily arranged spatially, they ‘mean’ differently than they would if the same ideas were conveyed in writing. • There are of course many possible ways of changing a written passage from literary text into semiotic modes, i.e., making it work as an adaptation for television. However, there’s no right or wrong way to do this. • Overall, adapting a written text into a visual version necessitates distinguishing between modes, media and affordances (P. 305)

28. Communication media • Media is the plural of medium , and a medium of communication implies a way of communicating with different people . • Communication media include audio , video, printed material and photos. Communication can be broadly split into two categories: • Physical Media : • A person who is talking can be seen and heard by the audience , we only hear the messages but we also see body language and feel the climate • Examples of Physical media: Large meetings , and town hall meetings • Mechanical Media • communication that is conveyed through written or electronic channels. It is used as archives for messages.

29. Communication Modes • Mode relates to the means by which a message is represented , using for example the sounds of speech, the graphic system of writing or the gestures of sign language. Communication modes can be split into four types: • 1-InterpersonalModeThe Interpersonal Mode is characterized by active negotiation of meaning among individuals. • 2- Interpretive Mode • The Interpretive Mode focuses on the appropriate cultural interpretation of meaning that occurs in written and spoken forms where there is no recourse to the active negotiation of meaning with the writer or the speaker.

30. 3-PresentationalModeThe Presentational Mode refers to the creation of messages in a manner that facilitates interpretation by members of the other culture where no direct opportunity for active negotiation of meaning between members of the two cultures exists , e.g., TV programs, Internet, videos, currency and give personal reactions. • 4-Kineikonic Mode The kineikonic mode is a term used to denote the moving image as a multimodal form. kineikonicmode unifies what is culturally understood as a form of the moving image.

31. Affordances • They are properties of the relationship between an organism and the material environment. • Objects afford possibilities to humans and animals; for instance, a tree affords shade to an animal. • A hypertext could be a series of static screens, or scrolling text. The words or images could contain clickable links to descriptions of their characters, or reminders of past events in the narrative, or other books by the same author, or location maps to show the wider scene.

32. 2 Adapting narrative elements across semiotic modes • A filmadaptation is the transfer of a written work, in whole or in part, to a feature film. It is a type of derivative work. • A common form of film adaptation is the use of a novel as the basis of a feature film. • Text producers make some creative choices and these choices have effects when text produces adapt literary narratives for different media and semiotic modes. • Among the choices the adapter or creator of a text must make are those relating to the presentation of narrative elements.

33. Macbeth Original Script last scene. • Adaptation: • https://www.youtube.com/watch?v=KTyqbts4oT0 • Macbeth adapted script last scene. • Another adaptation:

34. Adapting literature for film [Reading B] • Porter Abbott describes the process of adaptation as creative destruction’. • The film producer has to make ruthless decisions in order to create, or recreate, a coherent narrative for the new medium. • Porter Abbott notes some important aspects of novels which have to be recreated in new ways for moving images such as films duration, characterization, metaphor and gaps. • What the author says about the fidelity’ of the text when reproduced in a different medium?

35. 3. Creative involvement in the text A-What does a reader/viewer do? • Reading or viewing a novel requires engagement from us in some way. • In reading a novel, we enter into the world of the characters, we piece together ‘mind maps’ of locations from the author’s descriptions, we decode metaphors, we form opinions about characters and predict events. • In other words, readers are involved in meaning making. • Fish (1980, p. 89) rejects the notion that meaning resides in the text, insisting on its location in the process of reading: ‘meaning is not the property of a timeless formalism but something acquired in the context of an activity’. • Overall, the roles of a reader/viewer in terms of written and multimodal texts can be outlined as follows:

36. Reader /viewer roles (P. 310-311) • 1-Active : • Both the reader and viewer make connections and seek creating meaning from a text in a traditional novel or multimodal text. • 2-Interactive • The reader usually makes decisions , chooses elements from the selection at hand. Likewise, viewers can interact with the multimodal text by changing , for instance changing channels or pressing buttons…. • 3-Selective/Agentive • The reader / viewer usually makes choices as to what to read/view, and in what order including the possibility of backtracking and making alternative choices , creating a different pathway through the text. • 4-Transformative • The reader is more clearly involved in the text’s production to the extent that he/she is acting like a writer . Depending on how text is designed, hypertext allows the reader to write back , intervene and change the text.

37. 5-The designer/producer • Designis currently important in multimodality theory, and, while it encompasses interpretation. Design suggests that we are all designers of meaning: faced with an increasing number of multimodal texts in all aspects of our lives, we are called upon to construct meaning from all semiotic modes and from the ways these modes are articulated and combined. • Constructing meaning from texts is what it means to talk of a shift from ‘reading as interpretation’ to ‘reading as design’. • Design,then, implies engagement with creative processes as well as creative texts. The degree, and location of activity and creative intervention may be difficult to identify, but the point is that it is difficult to see creativity as mainly located in the text.

38. Reading C, • Andrew Burn and David Parker analyze the semiotic modes used to redesign an oral narrative, a Ghanaian Anansi story. Anansi is a trickster spider, common in oral folktales in many different African countries and the Caribbean. • Burn and Parker analyze a short section of the film in terms of the overall ‘communicative metafunctions’: ideational, interpersonal and textual which derived from the functional linguistic model of Michael Halliday (1985). (P. 314).

39. 4. Hypertext & Hyperfiction • This section is concerned about literature created on the internet, in hypertext and in internet relay chat (IRC). Hypertext term was coined by Ted Nelson around 1965, it is a text which is not constrained to be linear, and which contains links to other texts. • Aweb page on a related subject may load, a video clip may run, or an application may open. Hypertext is different from hyperlink, the latter implies reference or navigation element in a document to another section of the same document or to another document that may be on or part of a different domain. • In producing literature for hypertext and IRC, every act of authorship can be potentially seen as an act of play.

40. A-Hypertext, the author and playfulness "Hypertext differs from the printed text by offering readers multiple paths through a body of information: it allows them to make their own connections, incorporate their own links, and produce their own meanings. Hypertext consequently blurs the boundaries between readers and writers [..] The extent of hypertext is unknowable because it lacks clear boundaries and is often multi-authored. "

41. Reader of hypertext (P. 316-17) • The reader of hypertext can be seen,: • as active: as the embodiment of poststructuralist notions of the ‘death of the author’, an influential idea put forward by Roland Barthes in 1977 . • as a writer: ‘In the electronic medium readers cannot avoid writing the text itself, since every choice they make is an act of writing’ [Bolter, 1991. p. 144] • as an author , but not a writer: In view of Montfort [2003],in much interactive fiction, the reader at most types a couple of commands, or simply clicks on the mouse. This is not writing, although it might be authoring. • as an interactor, rather than an author.

42. B-Hyperfiction (P. 317). • Hyperfictionis fiction that uses the linking capabilities of hypertext. Hyperfiction is also called hypertext fiction, tree fiction, non-linear fiction, and electronic fiction. • Hypertextual fiction is non-linear; it cannot be represented on a printed page. The reader takes an active role in the hyperfiction, choosing which links to click on and which paths to follow. • The story, then, may be very different from one reading to the next, depending on the choices made by the reader. Hyperfiction can be distributed on the World Wide Web or through standalone. • The choices made by readers of hyperfiction vary according to the amount of control retained or given up by the originating author. • Hyperfiction that gives more control and more choices to the reader will obviously give them greater opportunities for creativity, an example of hyperfiction is Afternoon by Michael Joyce [1987]: P. 318-320.)

43. c-The features of hyperfiction • 1-Less stability and fewer conventions • The technological nature of hypertext is one reason for its unstableness. There is no tangible entity that by its immediate physical presence offers the reader simple access to it. Instead, the reader must access it through a complex system of computer hardware that seems to have power on its own. • 2-Open text • Hyperfiction is often literally open, that is, without a clear start and end point. The reader stops reading simply when he does not feel like reading it anymore, if the text loops too much, or if it simply becomes too enigmatic and frustrating. • 3- intertextuality • Hypertext can easily direct the reader to other texts in a way that books cannot.

44. 4-Disorientation • Multilinearity and the rapidly changing contexts of Hypertext makes its reading confusing to a mind accustomed to one-way reading in print. • 5- Multivocality • The reader of hyperfiction often experiences a blurring of the identity of who is narrating or which character is speaking. • 6- Less authorial power and Liberation of reader • In hypertext there are more possible paths of reading, but the book reader always reads linearly, word by word, Most readers of hyperfiction experience probably skipping words, paragraphs or pages…etc. The reader cannot turn to a certain passage as easily as in a book and in most hyperfiction the author dictates what paths the reader may take in which order, except in the few hyperfiction.

45. C-Hypertext poetry (p. 322-324) • Like hyperfiction, hyperpoetry obscures the boundaries between readers and authors, and between texts and readers. • Meaning is created in the process of reading when the reader encounters the text. • Example: mountain rumbles: • http://www.cddc.vt.edu/journals/newriver/larsend/yama/index.html • POP UP POEMS. (P. 324).

46. D-Performance on the Internet • Online theatre highlights the aspects of creativity in texts produced with newer technology: design, the utilization of modes and affordances to creative effect, and the collaborative nature of such creativity. • In this regards, Brenda Danet’s book Cyberpl@yis worth being considered. • In HamnetPlayers [Reading D] the actors played with the affordances of writing, image, layout and punctuation, and they played with the medium of computer communication, and its sub-medium, IRC or Internet Relay Chat. (p. 353-356) • Unlike hypertext, Hamnetwas produced synchronously, in real time. The actors were online simultaneously, and could therefore use the creative affordances of time as a semiotic mode. • Brenda Danet shows how IRC provides extensive opportunities for playful and creative uses of language.

47. Conclusion • Modes, media, affordances, • Types of media • Types of modes • Hypertext.. Hyperlink • Roles of reader /viewer of the hypertext • Hyperfiction • Features of hyperfiction • Hyper poetry • Internet performance

48. Good Luck • Antar Abdellah