‘Joining in’ spontaneous conversation and improvisational music-making

1. ‘Joining in’ spontaneous conversation and improvisational music-making Sarah Hawkins1, Richard Ogden2, Ian Cross1 1Centre for Music & ScienceUniversity of Cambridge 2Dept. of Language and Linguistic Science,University of York {sh110, ic108}@cam.ac.uk rao1@york.ac.uk Language, Music and Interaction, Philological Society,QMUL, Nov. 2012

2. What processes underpin interaction in music-making and speech? with unfamiliar instruments / objects Spontaneous interaction:conversation and musical improvisation • A controlled environment in which to elicit (relatively!) natural joint (inter)action in unrehearsed • talking • music-making • non-musical play • Initial observations and hypotheses

3. Speaker–listener neural coupling underlies successful communication Stephens, Silbert, and Hasson (2010) PNAS 107(32) 14425-14430 Neural: Charles Schroeder group; Edward Large group e.g. Schroeder et al. (2008)TICS 12(3), 106-113 Fujioka, Trainor, Large & Ross (2012) J. Neurosci. 32(5), 1791-1802 Sociophonetics: Garrod & Pickering (2004) TICS 8(1), 8-11 Stephens, Silbert, and Hasson (2010) fMRI as speaker tells a story; another P listens. a speaker’s brain activity is spatially and temporally coupled with the listener’s activity but only when the listener understands the speaker most correlated patterns in listener are delayed relative to speaker’s; but some are anticipatory greater anticipatory speaker–listener coupling→ greater understanding (independent comp. measure)

4. Hypothesis: if coupling of neural oscillations underpins successful communication, then we should find: similar processes in music and speech differing only by demands of the medium function of the particular interaction our aim: find a set of comparable tasks & measures in music-making and conversing CA framework: alignment and disalignment

5. Alignment and disalignment in talk hasn’t he got nice eyes? In music, we can expect similar patterns (Turino) Superordinate, multidimensional terms We don’t expect binary classification every time

6. demographic questions; initial consent Minutes conversation ≥ 5 • How did you get here? • What do you think of the room? • 9/11; Princess Diana’s death… • important event you shared • xylophone, kalimba • drums, claves • card houses • tallest tower: blocks • market stall: playdough at least 2 10 non-musical play 10 musical play no shakers! 10 conversation detailed musical questions; final consent; £8

7. Who? “Pilot”: 5 dyads, • 3 musician pairs • 2 non-musician • various tasks “Experiment 1” 6 dyads, • 3 musician pairs • 3non-musician • 2-3 prescribed tasks Dyads: • friends • same-sex • native speakers of English • 18-30 • university educated • both musicians,or bothnon-musicians

8. Starting set-up: familiarisation • card house • tallest tower of blocks • playdough market stall • Sri Lankan drums (2 types, one not shown) • claves • circular xylophone • kalimba(mbira)

9. non-musicians • ≥ 10 years’ formal training • currently actively engaged in music at least once a month • ≤ 7 years’ formal training • no active music-making in past 4 years musicians

10. Recording • 4 video cameras • overhead omni mike • stereo pair (music) • 2 close-talking mikes

11. Recording • 4 video cameras • overhead omni mike • stereo pair (music) • 2 close-talking mikes

12. Looking for co-ocurrences • Focus: alignment and disalignment • rhythmically: entrainment and failure to entrain • Body movement is well established as marking important events (beats) in both speech and music • What happens when such beats carry across the two modalities? • A single framework for labelling events • tracking beats in speech and music: currently, Cummins

13. Example(s) of what we found placeholder • Alignment “magic”: E1_MF1 29:43-29:51 • beat continuation across modalities, and between participants • perfect coordination • unscripted (music not ‘counted in’) • Disalignment: E1_MF1 26:31-26:47

14. Alignment: beat continuation across modalities, and between participants • The criterion is (for alignment): there is speech before or after where • the music changes, and the music 'works'. • The questions are: to what extent is the speech and music beat coordinated? • and how does this compare when there is breakdown or less successful interaction? • what happens with body movements and eye gaze?

15. Tentative Hypotheses (to be completed) • look at the effector: • hands when playing • faces when speaking • presumably faces when singing together • look at times of uncertainty….

16. Next steps • Quantify: proportion of positive instances of categories • Theory: which? is there only one? • Why? (Causes) do people entrain willy nilly or element of prediction from one or other • if we can’t tell bottom up from top down, and there’s not a clear listener vs clear talker, what are we dealing with – the holy spirit? • Theory: top down and bottom expectancies mesh: me, Narmour, Pearce/Wiggins….we need to work actively to get this working for a general theory – and using sp and music as our test bed seems an exciting way forward. Form a working group???

17. Satinder Gill Thanks to! David Greatrex Rein OveSikveland Daniel Halford Hannah Leach Newton Trust, U. Cambridge; BA/Leverhulme Foundation Small Grant

18. Some random slides from past presentations that may help orient you re our way of thinking

19. So why do music and speech seem fairly different, but we feel they are the same? • music and speech typically differ in the relative balance accorded to conveying phatic vs referential meaning—but this is a very loose difference • functions of both modalities dictate what is important, and where we should look for guiding principles • languages will differ – as will musics (structure) • these different balances in large part dictate the greater predictability of rhythm in music than in speech

20. some things that music and speech share • wide range of rates, affected by many factors • phrase-final lengthening • predictable tonal endings: cadence, nuclear tone • internal/local rate change: rubato, asides, emotion, floor holding • deviation from rhythmicity indicates • emotion • phrasing • in speech, the demands of the actual words used • regular rhythms are constructed, in both speech and music • consistent with most other perceptual approaches

21. Summary of a video clip (not included) • the background speaker, L, maintains a beat of about 460 ms in speaking; and appears to lead the transition into music: • though R talks more, and is talking very casually, she seems to entrain to L’s speaking beat • they start playing about on ‘the current beat’: c. 800-900 ms • gradually increase tempo to c. 700 ms • look at the effector: • hands when playing • faces when speaking • presumably faces when singing together