Review of MUSIC PERFORMANCE

1. Review of MUSIC PERFORMANCE Professor, Dept of Psychology Canada Research Chair Cognitive Neuropsychology of Performance McGill University, Montreal, Quebec, Canada by Caroline Palmer Ann. Rev. Psychol. 1997, 48:115-38 Presented by Elaine Chew On January 11, 2006, as part of ISE 599: Topics in Engineering Approaches to Music Cognition - Computational Models of Expressive Performance

2. AGENDA • INTRODUCTION • INTERPRETATION • PLANNING • MOVEMENT ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

3. Forms of Performance • Sight-reading • Performing well-learned music from memory or notation • Improvising • Playing by ear ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

4. Serial Order and Timing Issues • Skilled serial action: • speaking, typing, performing music • Activity must be centrally linked • Little time for feedback for planning • Can be performed w/o kinesthetic feedback • Accurate temporal control: rhythm • Basis for dev models of timing mechanisms • Consensus on requirements for accuracy ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

5. Purpose of psychological studies • Develop theories of performance mechanisms (cognitive/motor constraints) • Explain treatment of structural ambiguities • Understand relationship between performance and perception ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

6. Components of performance • Interprete piece conceptually • Retrieve musical structures and units from memory • Prepare for production • Transformed into appropriate movements ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

7. Methodological Issues • Wealth of data • Separating signal from noise • Focus on movement-based information • Judgement of representative piece • Recognized level of performer expertise • Large samples of data hard to find • Rely on converging evidence from both small and large sample studies ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

8. Performance Expression • Variations in timing, intensity (dynamics), timbre, and pitch • form the microstructure of a performance • differentiate it from another of the same pc • Measurements • deviation from fixed or regular values as notated in score • Relative to performance itself, e.g. pattern of deviation with repect to a unit s.a. a phrase ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

9. AGENDA • INTRODUCTION • INTERPRETATION • PLANNING • MOVEMENT ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

10. System of Communication • Chain of events … ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

11. System of Communication • Composers code musical ideas in notation • Performers recode from notation to acoustical signal • Includes performer’s conceptual interpretation of composition • Listeners recode from acoustical signal to ideas ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

12. Interpretation • Performer’s individualistic modeling of a piece according to their own ideas or musical intentions • In western music notation: • Pitch and duration (clear) • Intensity and tone quality (approx) • Group boundaries, metrical levels higher than the bar, patterns of motion, tension, and relaxation (unspec, implicit) • Could explain inter- and intra-performer performances of the same piece ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

13. Role of Analysis • Every performance involves some kind of interpretation or analysis • Analysis offers explanations for the content of a composition as a • Hierarchy of whole/part relations • Linear course following harmonic tension • Series of moods that result in unity of character • Analysis does not indicate how a performer actually produces a desired interpretation ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

14. Goal of Interpretation • Convey the meaning of the music • Structure, emotion, and physical movement • Highlight particular structural content • Highlight particular emotional content ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

15. Highlighting structural content • Nakamura (1987): • Compared musicians’ performances of baroque sonata with notated interpretations of dynamics • Perceived dynamics matched intended fairly well, even when underlying acoustic changes were not identifiable • Palmer (1989): • Compared pianists’ notated intepretations of phrase structure and melody with expressive timing patterns • Melody lead and slowing of tempo at phrase boundaries observed • Expressive timing patterns decr when attempting to play w/o interpretation, incr in exaggerated interp • Palmer (1988): • Expressive timing patterns incr from novices to experts, during practice of unfamiliar piece, changed in diff interp by same perf ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

16. Implication of structural content interp • Palmer (1992): • Pitch deletions tend to occur within phrases, and pitches tend to persevere at phrase boundaries • Interpretations strengthen phrase boundaries relative to other locations • Palmer & van de Sande (1993, 1995): • Melodic events are correctly retrieved and produced relative to nonmelodic events ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

17. Goal of Interpretation • Convey the meaning of the music • Structure, emotion, and physical movement • Highlight particular structural content • Highlight particular emotional content ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

18. Highlighting emotional content • Langner (1953): Music shd sound the way moods feel • Gabrielsson (1995), G. & Juslin (1996): • Compared performers’ interp of emotional content with their use of expression • Happy/angry - faster, larger dynamic range • Soft/sad - slower, smaller dynamic range • Ashkenfelt (1986): • Similar results in tender/aggressive experiments • Schmalfeldt (1985), Shaffer (1995): • Emotional content as part of narrative, dramatic char, thematic content, conceptions of large-scale structures ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

19. Role of experience • Musical experience enhances ability to use and identify interpretations • Nonmusicians can pick up interpretative aspects of performance • Discern general differences among mechanical, expressive, exaggerated perf • Can hear intended phrase structure • Cannot always find melody interpretation • Sufficiency of expressive features to convey intepretations ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

20. AGENDA • INTRODUCTION • INTERPRETATION • PLANNING • MOVEMENT ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

21. Planning and Memory • Related to melodic, harmonic and diatonic structures • Chord errors occur more in homophonic mus • Single note errors more in polyphonic music • Mistakes originate more from key of piece • Mistakes tend to be of same chord type • Child singing pitch errors tend to be harmonically related to intended events • Pianists’ sight-reading errors in pcs with deliberate pitch alterations indicate tacit melodic/harmonic knowledge ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

22. Subsequence Partitioning • Partitioning into phrases • Errors originate more from same phrase • Interacting errors rarely crossed phrase boundaries (like in speech) • Errors increased when melodic, metrical, rhythmic accents unaligned • Planning ahead • Eye-hand span 7-8 events, or to phrase end • Range of planning affected by serial & struct ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

23. Syntax of Musical Structure • Events at most salient levels are commonly emphasized in performance • Tactus: foot tapping metric level • Phrase: partitioning of melody • More important events are processed at deeper hierarchical (structural) levels • Improvisations tend to retain only structurally important events from abstract hierarchical levels of reduction ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

24. Structure-Expression Link: Phrases • Decrease of tempo/dynamics at end of phrases • Amt of slowing at a boundary reflects depth of phrase embedding • More important segments have greater phrase-final lengthening • Greatest corr bet expr timing and intensity found at interm phrase level • Performers’ notated/sounded interpretations differ most at levels lower than phrase ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

25. Structure-Expression Link: Meter • Events on strong beats often lengthened, have delayed onsets • Events on metrical accents louder, longer, more legato • Listeners’ judgements of metric interpretation aligned best with experienced pianists’ intended meter • Articulation most often used as metric cue, loudness not always present • No one set of necessary and sufficient expressive cues to denote meter exist ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

26. Structure-Expression Link: Rhythm • Systematic deviations in Vienese Waltz: • Short 1 - long 2 - 3 ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

27. Expressive Timing Patterns • Structure • Meter, accent pattern, simplicity (dur ratios) • Motion • Rapidity, tempo, forward movement • Emotion • Vitality, excitedness, playfulness ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

28. Comments • Melodic/metrical accents sometimes altered by presence of rhythmic accents or each other • Melody lead may serve to separate voices perceptually ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

29. Generative model of expr synthesis • Clarke (1993, 1995): • Systematic patterns of expression result from transformations of the performer’s internal representation of musical structure • Support for view: the abilities to • Replicate same expressive timing profile with little variation across performances • Change interpretation and produce different expression with little practice • Sight-read with appropriate expression ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

30. Rule-based models • Sundberg et al (1983ab): • Differentiation, grouping, ensemble rules affect event durations, intensities, pitch tunings, and vibrato • Clynes (1977,1983,1986): • Composer-specific inner pulses applied to different levels of musical structure • Piece-specific factors contribute as much as piece-transcendent factors captured by rules ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

31. Arguments against generative models • Performers can imitate expressive timing patterns with arbitrary relationships to musical structure • Accuracy worse with more disruptive structure-expression relationship, improved with repeated attempts • Suggests expression not generated solely from structural relationships ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

32. Perceptual Functions • Communicate particular interpretations and resolve structural ambiguities • Compensate for perceptual constraints of auditory system ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

33. Other explanations for expression • Compensatory explanation: • Some notes played louder/longer because they would be heard softer/shorter otherwise • Musical structure elicits expectations: • Detection of lengthening more difficult where expected • Detection accuracy inversely related to performer’s natural use of lengthening in same piece • Structure constrains both perception and performance ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

34. Music Theories • Narmour (1990,1996): • Model of melodic expectancy • Lerdahl (1996): • Model predicting tonal tension and relaxation • Listeners can apprehend predicted structures • Expressive cues emphasize computed structures • Interpretations constrained by composition ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

35. AGENDA • INTRODUCTION • INTERPRETATION • PLANNING • MOVEMENT ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

36. Movement • Musical rhythm often defined relative to body movement • Different views on relationship: • Motor control - movement generating timing • Timekeeper - internal clock for anticipation and coordination of gestures ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

37. Timekeeper models • Role: regulate and coordinate complex time series, such as those produced between hands or between performers • Constructs beats at abstract level, providing temporal reference for future movements • Evidence: rhythm reproduction better for integer duration ratios ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

38. Internal clocks • Single clock model • Multiple timekeepers (Jones 1990 review) • Attributed to perceptual encoding • Attributed to production mechanisms ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

39. Clock operation level • Tactus: most salient metrical level • Preferred tactus ~ 600ms (spontaneous clapping period) • Typical inter-step interval ~ 540ms • Listeners use motion to describe rhythmic patterns when interbeat intervals ~ 650ms • Time periods derived are multiples or fractions of beat periods ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

40. Source of temporal variance • Early models: partitioned temporal variance to lack of precision of timekeeper vs. motor response delay • Extended to hierarchical organizations of timekeepers at multiple metrical levels • Performed durations at metrical level less variable than durations of residual nested events within that level ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

41. Hierarchical clocks • Timekeeping most directly controled at intermediate metrical levels of the sub-beat, the beat, or the bar • Solo piano music: timekeeping controlled at the beat level (hands have independence in coordinating events below beat level) • Separate timekeepers controled timing of individual hands • Duet piano performance: highest precision (least variance) at bar level • (above studies assumed constant global tempo) ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

42. Performance timing stability • Not only at tactus / beat / bar level • Exists at level of entire piece. Durations of string quartets over repeat performances highly consistent. • Std dev of piece duration ~1% • Less than variations in movement lengths • Proportional tempos theory: tempos of successive sections of music form simple integer ratios • Phase synchrony, esp at structural boundaries • May reflect performer’s memory for tempo ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

43. Movement • Musical rhythm often defined relative to body movement • Different views on relationship: • Motor control - movement generating timing • Timekeeper - internal clock for anticipation and coordination of gestures ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

44. Motor Programs • Contains representations of internal actions and processes that translate them into movement sequence • Accounts of motor equivalence across contexts • Possible proof: Relational invariance - tempo changes as parameter change ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

45. Relational invariance • Relative durations of notes tend to vary across performances played at different tempi • Hypothesis: structural interpretation does not remain constant across performance tempo • # group boundaries incr at slower tempo • Practicing at different rate than intended performance might be counterproductive • Lesson: do not draw conclusions from average of performances over diff tempi ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

46. Tempo changes perceived structure? • Tempo affects perception of duration patterns • Different perceptions may result for same relative expressive timing pattern at different tempo • Repp (1995b): • Manipulated degree of expressive timing and global tempo • Listeners preferred reduced expression with fast tempo and augmented expression w slow ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

47. Kinematic models • View: music performance and perception have origins in kinematic/dynamic characteristics of typical motor actions • E.g. walking -> beat • Aesthetically satisfying performances should satisfy kinematic constraints of biological motion ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

48. Kinematic models • Final ritards modeled as variable curve followed be linear decrease in tempo • Feldman et al (1992): cubic polynomial models used to minimize jerk/jumpiness in connecting points of tempo changes • Repp (1992b): used quadratic ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

49. Models with dynamics • Studies suggest coupling bet expr timing and dynamics • Todd (1992): proposed model where intensity proportional to square of vel. Used constant acceleration • Todd (1995): proposed auditory model of rhythm performance and perception • Temporal segmentation of onsets • Periodicity analysis • Sensory-motor feedback: tactus, body sway ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance

50. Arguments agains kinematic models • Physical notions of energy cannot be equated with psychological concepts of musical energy • Tempo changes guided by perceptual rather than kinematic properties: • Large tempo changes cannot occur too quickly (perception to rhythmic categories) ISE599 (ISE575b / CSCI575b / EE675b pending): Computational Modeling of Expressive Performance