Musico -Literary Miscegenations : words and music synergies in new media writing ``1

1. Motions: Hazel Smith (text); Will Luers (image and coding) Roger Dean (music) Musico-Literary Miscegenations: words and music synergies in new media writing ``1 Hazel Smith Writing and Society Research Centre University of Western Sydney

2. Trajectories • Focus: words and music relationships in new media writing. •  Relationship between words and sound in the new media writing field neglected by critics. Tend to be more interested in the relationship between word and image. •  Word/sound relationships have tended to be less explored by new media writing practitioners than word/image. • Aspirational as well as analytic. Grigar, Dene. "The Role of Sound in Electronic Literature." trAce Online Writing Center. Spring 2006. Barber, John F. Internet radio and electronic literature: locating the text in the act of listening, Electronic Book Review, http://www.electronicbookreview.com/thread/electropoetics/internetradio

3. What is new media writing? • New media writing (electronic literature, digital writing, networked and programmable writing) is a form of literature that exploits the possibilities of new technologies to evolve the way we write and read literature. • Usually has screened words, may also have spoken words. •  Some features: kineticism, interactivity, processing of text, textual variability, multimedia. Hayles, Katherine N. Electronic Literature: New Horizons for the Literary. Notre Dame: University of Notre Dame Press, 2008.

4. Will Luers (video), Roger Dean (sound), Hazel Smith (text), Film of SoundCordite Poetry Review , 2013. • http://cordite.org.au/ekphrasis/film-of-sound/

5. Words and music: formal and cultural crossings • Cross-overs of words and music in new media writing can create hybrid structureswhich produce emergent meanings. • These hybrid structures can facilitate various forms of cultural crossing and cross-cultural encounter. • Conceptualise this process of formal and cultural crossing as musico-literary miscegenation (Sexual inter-racial relationships).

6. word ethnicity Musico-literary miscegenation in new media writing image textsound imagetext sound Nation state culture Intermedia Processes Parallelism Coordination Interactive variability Algorithmic Synaesthesia Heterogeneity Semiotic and perceptual exchange Cultural processes Transnationalism Cross-cultural encounter Cultural borrowings and appropriations Translation Intermedia hybrid space Musico-literary miscegenated space Intracultural hybrid space

7. Formal aspect ; Zbikowski; conceptual blending •Formal aspect of the model; builds on theories of musical multimedia, work in the area of musico-literary discourse: Nick Cook, Laurence Kramer, Lawrence Zbikowski. • Zbikowski:draws on ideas about conceptual blending from cognitive linguistics (Turner and Fauconnier) and maps them to the way words and music interrelate. —  generic space, input spaces, blended space. • differences from Zbikowski; multimedia not song and opera; interested in juxtapositions, frictions, hybrids, not blends; cultural framework. Zbikowski, L. M. (2002/3 ). Music Theory, Multimedia, and the Construction of Meaning. Integral 16/17, 251-268. Cook, N. (2001). Analysing musical multimedia, Oxford: Oxford University Press. Kramer, L. (2002). Music Meaning: Toward a critical history, Berkeley: University of California Press.

8. New media writing: what kinds of sound/music? (1) • Ecological sounds (sounds taken from or mimicking the environment, from everyday life). • Often have an iconic or illustrative connection with the text. Clarke, E.F. Ways of Listening: An Ecological Approach to the Perception of Musical Meaning, Oxford University Press, 2005

9. Example (ecological sound )MD Coverley Afterimage • http://califia.us/Afterimage/end.htm

10. New media writing: what kinds of sound/music? (2) •  Musical sounds: often used for their affective, emotional, atmospheric or immersive properties. — from acoustic instrumental music to electroacoustic music. (Film of Sound) —  independent piece of music, separately composed; music composed for the piece including algorithmically generated music. Smith, H. soundAFFECTs: translation, writing, new media, affect in Sounds in Translation eds. Amy Chan , Canberra: Australian National University Press.

11. New media writing: what kinds of sound/music? (3) •Voiced sounds: —straight — technologically manipulated — computer - synthesised voices Smith, Hazel , and Roger T. Dean. "Voicescapes and Sonic Structures in the Creation of Sound Technodrama, Performance Research 8.1 (2003): 112-23.

12. Intermedia processes Word/sound juxtapositions in new media writing the intermedia hybrid space (1) Coordination, Parallelism, Interactive Variability, Algorithmic Synaesthesia, Heterogenity, Semiotic and Perceptual Exchange (may be more than one type of conjunction in any one piece) Coordination: —  associative, representational, illustrative linking between sound and words — (often ecological sounds (e.g. MD CoverleyAfterimage ). Most emergence when associations are multiple and are indirect. Parallelism: — music and screened text run continuously together but are independent entities. The text or music or both, may be algorithmically organised to be different each time the piece is played, so there may be different points of coordination each time.

13. Brian Kim Stefans, Suicide in an Airplane 2014 — Example of parallelism: Brian Kim Stefans (Asian American writer), Suicide in an Airplane 2014 written in Flash, music Suicide in an Airplane by Leo Ornstein, futurist composer 1919. —  Source text for the words is an article from The New York Times ; American combat operations in Afghanistan; algorithms for the text chose different words each time from the source text. — Different points of coordination between the words and music each time. Energetic music, futurist celebration of mechanisation, engines and flight; a text which can be read as a critique of the bombings in Afghanistan. Parallelism can bring out both similarities and differences.

14. Example 2 (Brian Kim Stefans: Suicide in an Airplane, 2013) • http://www.arras.net/?p=572..

15. Intermedia processes Word/sound juxtapositions in new media writingthe intermedia hybrid space (2) Interactive variability: —the user can interact with, and vary, the sound (possibly in addition to the text: WilksTailspin, BouchardonLoss of Grasp ). — Or the performer can interact with the sound during real-time performance. Creates constantly changing relationship between text and sound. Algorithmic synaesthesia: — Music and words may be composed using the same algorithmic basis for both; the algorithms from the words can be translated into algorithms for sound (e.g. Smith, Dean, Brewster, soundAFFECTs; CayleyTranslation). Dean, R.T., et al. The mirage of real-time algorithmic synaesthesia: Some compositional mechanisms and research agendas in computer music and sonification.Contemporary Music Review

16. Hazel Smith, Anne Brewster (text, Roger Dean (sound and coding) : soundAFFECTs, 2004. • On CD Rom accompanying, Hazel Smith The Erotics of Geography, Tinfish Press, 2008.

17. Intermedia processes Word/sound juxtapositions in new media writingthe intermedia hybrid space (3) Heterogeneity: — Sound and text composed of multiple, heterogeneous and dynamic elements. Continuously couple and uncouple in different ways. — The more heterogeneous the elements, the more emergence . (Luers, Dean, Smith Film of Sound). Semiotic and perceptual exchange: —Different media seem to take on each other’s characteristics when juxtaposed. —  Words highly referential, while music is more abstracted. When words and sound are juxtaposed become more aware of sonic/abstracted aspect of words, referential aspects of sound. — perception as well as semiotics. (e.g. Young Hae Chang Heavy Industries) Smith, H. & Dean, R. (1997). Improvisation, Hypermedia and the Arts Since 1945, London and New York: Harwood Academic.

18. Young Hae Chang Heavy Industries: Nippon (music: Thelonius Monk quartet , Japanese Folk Tale, 1967) • http://www.yhchang.com

19. Young Hae Chang Heavy Industries: Nippon (2) • Conscious use of semiotic and perceptual exchange – the movement of the language is programmed to respond to the rhythm of the music. • after the opening the Japanese is moving with the piano and the saxophone with the English. • the word-music exchange is not only creating a semiotic dialogue, it is also creating a cultural interaction.

20. word ethnicity Musico-literary miscegenation in new media writing image textsound imagetext sound Nation state culture Intermedia Processes Parallelism Coordination Interactive variability Algorithmic Synaesthesia Heterogeneity Semiotic and perceptual exchange Cultural processes Transnationalism Cross-cultural encounter Cultural borrowings and appropriations Translation Intermedia hybrid space Musico-literary miscegenated space Intracultural hybrid space

21. Intercultural processes and the intercultural hybrid space: Musico-literary miscegenation Mixings/crossings of word and sound can and may facilitate cultural crossings : — crossing of national borders (transnationalism) —  crossing over of different ethnicities and ethnic cultures — various forms of cultural encounter (western, non-western cultures) — various forms of cultural appropriation

22. Musico-literary miscegenation Results in musico-literary miscegenation: • Reactivating the word miscegenation more positively: — maps the ethnic, erotic and reproductive onto the mixing of words and music —  brings the histories of colonialism, cultural imperialism and unequal power relationships to bear on our thinking about the word-sound relationship —  in intermedia/multimedia work, word and music relationships most successful when non-hierarchical.

23. Theoretical Background Cross–cultural aspect of the model builds on: • theories of globalisation/cosmopolitanism (e.g. George Ritzer— glocalisation and grobalisation as the two faces of globalisation, Ulrich Beck, Robert Fine, Berthold Shoene). Posthuman cosmopolitanism. • theories of biraciality (Zack, Funderberg, Piper, Romano, Sollers); cross cultural encounter, hybridity (e.g. Paul Gilroy, Stuart Hall, Robert Young). Work of Jackie Lo and Helen Gilbert. • the positive retrieval by others of the idea of miscegenation (Dunning; Kim Stefans) Three examples which show different forms of cultural crossing: cross cultural encounter, transnationalism; cultural appropriation. Lo, Jacqueline, and Helen Gilbert. "Toward a Topography of Cross-Cultural Theatre Praxis." TDR/The Drama Review 46.3 (2002): 31-53. Dunning, Stefanie K. Queer in Black and White: Interraciality, Same Sex Desire, and Contemporary African American Culture. Indiana: Indiana University Press, 2009. Schoene, B. (2009). The Cosmopolitan Novel, Edinburgh: Edinburgh University. Press. Smith, H. A Posthuman Cosmopolitanism in new media writing, Hyperrhiz: new media cultures, 09, http://www.hyperrhiz.net/hyperrhiz09/essays/a-posthuman-cosmopolitanism-and-new-media-writing.html Ritzer, G. (2007). The Globalization of Nothing, London: Pine Forge Press. Zack, N. (1994). Race and Mixed Race, Philadelphia: Temple University Press. Kim Stefans(2013) Viva Miscegenation: New Writing (MakeNowBooks)

24. Example. Young Hae Chang Heavy Industries: Operation Nukorea 2003 / • http://www.yhchang.com

25. Operation Nukorea • Cross-cultural group based in Seoul. Young Hae Chang, Korean artist and translator; Marc Voge. American poet.Strong interest in the word-music relationship. • Formally, movement of text constructed to respond to two different rhythmic levels of the music: the regular pulse of the ostinato; the rubato effects of the melody: semiotic and perceptual exchange. •  Text about an imagined devastating attack by North Korea on South Korea, precipitated by a pre-emptive US attack on North Korean nuclear facilities. Text about east-west antagonisms. •  Juxtaposed with US jazz pianist Bill Evans’s “Peace Piece”,. Everyone Digs Bill Evans, 1958,followed by a trio item “What is there to say”. From Everyone Digs Bill Evans, 1958. • Thematically the music is meditative, calm and reconciliatory, opposite to content of the text. •  Juxtaposition of different media speaks to the conflict between east and west: emergent meaning. —  Whatever the level of aggression between nation states and different cultures, always the possibility of peace, exchange and cooperation.

26. Example. Translation John Cayley (text) Giles Perring (music) • http://collection.eliterature.org/1/works/cayley__translation.html. First published 2004.

27. Example: John Cayley/Giles PerringTranslation (2) • John Cayley: British new media artist currently living in the US. The text undergoes ‘transliteral morphing’ — the building up of, and substitution of letters, within each individual textual block; moves between English, German and French; and often mixes languages and inhabits an in-between language. • The main text that is translated is ‘ Translation is removal from one language into another through a continuum of transformations. Translation passes through continua of transformation, not abstract ideas of identity and similarity’. ‘On Language as such’ Benjamin.

28. John Cayley/Giles Perring: Translation (3) • Music, ‘sung vocabularies’, and the words share the same algorithms as the music (algorithmic synaesthesia).   • The music cursor scans the text and is programmed to respond musically to what is happening linguistically. • the translation between the media both facilitates and reinforces the translation between languages, and emphasisesthe broader cultural dynamic of transnational (European) crossings and inter-relationships. Emergence — the endless process of translation between languages, music and cultures, and the inhabitation of an in-between language.

29. Motions: Hazel Smith (text), Will Luers (image and coding), Roger Dean (music) 2014http://taylorstreetstudio.com/motions/ • About human trafficking and contemporary slavery (draws on many different genres: poetry, narrative, documentary, photography). Interactive multimedia book, different each time you read it. • Juxtaposes texts and images with music (through parallelism, interactivity illustration, heterogeneity) • Four sections of music: flight sounds, sampled Macedonian music , train sounds, Americanised music. Also cross-fertilisation of the American and East European musics. • Uses transnational appropriation of sounds (e.g. sampling of Macedonian music). • Uses transnational appropriation to evoke other cultures but also their suppression. Brings out the darker, exploitative side of globalisation/cosmopolitanism/transnationalism.

30. Motions (Smith, Luers, Dean) http://taylorstreetstudio.com/motions/

31. Coda • Different formal crossings between words and music can facilitate cultural crossings. •  this crossover creates the musico-literary miscegenated space. •  model reflects sonic-verbal interactions in pre-existing new media writing but can suggest future possibilities.