1 / 55

An I ntroduction to A ffective M usic, T heory and S ome A pplications

Roberto Bresin http://www.speech.kth.se/music/performance. An I ntroduction to A ffective M usic, T heory and S ome A pplications. Outlook. Aim To explain how it is possible to communicate different emotions with the same music score Part I: The science of music performance

natara
Download Presentation

An I ntroduction to A ffective M usic, T heory and S ome A pplications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Roberto Bresin http://www.speech.kth.se/music/performance An Introduction to Affective Music, Theory and Some Applications

  2. Outlook AimTo explain how it is possible to communicate different emotions with the same music score Part I: The science of music performance Analysis & synthesis of music performance • The most important techniques for measuring and modelling a performance • The acoustical cues of importance for communicating expressivity • How the use of acoustical cues can influence the performance style • A rule-based system for the synthesis of music performance Part II: Emotion in music performance • Emotionally expressive music performance • Real-time Visualization of Musical Expression • Examples • Applications • Emotional colouring of music performance • expressive ringtones in mobile phones • visual display of emotion in music performance

  3. Part IThe Science of Music Performance

  4. composer musician instrument Computational models score gestures sound listener Musical Communication

  5. The Musician • The musician Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression

  6. What is communicated? The musician • Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression • The music • Emotions • Imagined and real motion • …

  7. Different factors in the communication Auditive Social/cultural Score - Notes, harmony, melody, rhythm, pitch, texture, instruments Performance - Tempo, phrasing, articulation, intonation Environment - Concert, club, home, live/recording Audience Past experience Visual Body movements and gestures People, clothes, stage lightning, etc. Memory Musical knowledge

  8. What can be studied? The musician • Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression • What is a musical performance? • Emotional communication: Accuracy, musical factors • Emotional affect • Couplings to motion: Musicians gestures and the resulting sound • Visual perception of musicians body movements

  9. The Score and the Performance The musician • Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression How important is the performance? • Dead-pan by computer and sampler • Schumann’s Träumerei by Alfred Brendel IOI (%) Brendel Time deviation from score

  10. The Score and the Performance The musician • Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression IOI (%) Horowitz 65 IOI (%) Schnabel Time deviation from score IOI (%) Brendel

  11. Collecting Data of Expressive Performances The musician • Musical communication Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression • Expert musicians (Lars Frydén for KTH) • Expertise is translated into rules • Measurements of recorded performances • Commercial recordings (CDs) • Computer controlled acoustical instruments (Disklavier, Böserndorfer)

  12. Design of Performance Rules The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Performance rules obtained mainly with 2 methods: • analysis-by-synthesis • analysis-by-measurement Generative grammar for automatic music performance

  13. DIRECTOR MUSICES (performance rules) MUSIC SCORE (MIDI) PERFORMED MUSIC (MIDI) NEW / MODIFIED RULE K values (Rule quantity) PROGRAMMER PROFESSIONAL MUSICIAN Analysis-by-Synthesis of Music Performance

  14. Duration contrast rule Dead-pan, K=0 Exaggerated, K = 4.4 Moderate, K = 2.2 Interonset Interval deviations (%) Inverted, K = -2.2

  15. Analysis-by-MeasurementDesigning Articulation Rules The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression 3 main classes of articulation: • Legato (overlapping) • Staccato (detaching) • Repetition

  16. Legato and StaccatoTones Legato IOI = Inter-onset-Interval DR = Tone Duration KOT = Key Overlap Time KDT = Key Detached Time Staccato

  17. Data 5 pianists played the same score 9 times on a Disklavier The first sixteen bars of the Andante movement of W A Mozart’s Piano Sonata in G major, K 545 • Natural • Glittering • Dark • Heavy • Light • Hard • Soft • Passionate • Flat 1 pianist played 13 Mozart piano sonatas on a computer-monitored Bösendorfer

  18. Mean Legato (KOR) Legato articulation rule

  19. Mean Staccato(Key Detached Ratio, KDR) 19 Staccato articulation rule

  20. Context Influence in Staccato Production Amount of staccato (KDR) in different contexts for the 2nd note in a threenotes pattern. (S = staccato note, N = Non–staccato note)

  21. Legato and StaccatoAllude to Walking and Running? Control model for step sounds Legato and walking Staccato and running

  22. Footsteps Walking Running

  23. Controlling Footsteps The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Pd model for crumpling sounds controlled with performance rules

  24. KTH Performance Rules The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression • Descriptions of different performance principles used by musicians • General applicability • K values change the overall quantity of each rule • Context dependency • ~ 30 rules • 30 years ofresearch at KTH Score Rules Performance K values

  25. Director Musices The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression A program for modelling music performance http://www.speech.kth.se/music/performance

  26. Performance Rules The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Phrasing Phrase arch Final ritardando Punctuation High loud Harmonic/melodic tension Harmonic/melodiccharge Repetitive patterns and grooves Swing Articulation Punctuation Staccato/legato Accents Accent rule Ensemble timing Ensemble swing Melodic sync

  27. Phrase Arch Rule Dead-pan ΔIOI (%) Exaggerated

  28. Phrase Arch Rule Moderate ΔIOI (%) Inverted

  29. The DM system (~30 rules) The musician Musical communication • Modelling music performance Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Differentiation Rules Example: Duration Contrast Rule Grouping Rules Example: Phrase Articulation Rule Synchronization/Ensemble Rules Example: Ensemble timing Other Rules Example: Repetition Articulation Rule

  30. Part IIEmotion in Music Performance

  31. Happyorsadmusic?

  32. Mapping from Emotional Expression to Rule Parameters The musician Musical communication Modelling music performance • Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Emotional expression Rule parameters Mapping For each emotion: Select a palette of rule parameters according to previous findings

  33. No expression Tenderness Solemnity Happiness Sadness Anger Fear Synthesis of Emotional Expression The musician Musical communication Modelling music performance • Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Cues for thesimulation of emotions in music performance (byA.Gabrielsson and P.Juslin, Psychology of Music, 1996, vol. 24)

  34. Positive Valence From Juslin (2001) • HAPPINESS fast mean tempo (Ga95) small tempo variability (Ju99) staccato articulation (Ju99) large articulation variability (Ju99) high sound level (Ju00) little sound level variability (Ju99) bright timbre (Ga96) fast tone attacks (Ko76) small timing variations (Ju/La00) sharp duration contrasts (Ga96) rising micro-intonation (Ra96) • TENDERNESS slow mean tempo (Ga96) slow tone attacks (Ga96) low sound level (Ga96) small sound level variability (Ga96) legato articulation (Ga96) soft timbre (Ga96) large timing variations (Ga96) accents on stable notes (Li99) soft duration contrasts (Ga96) final ritardando (Ga96) Low Activity High Activity • ANGER high sound level (Ju00) sharp timbre (Ju00) spectral noise (Ga96) fast mean tempo (Ju97a) small tempo variability (Ju99) staccato articulation (Ju99) abrupt tone attacks (Ko76) sharp duration contrasts (Ga96) accents on unstable notes (Li99) large vibrato extent (Oh96b) no ritardando (Ga96) • SADNESS slow mean tempo (Ga95) legato articulation (Ju97a) small articulation variability (Ju99) low sound level (Ju00) dull timbre (Ju00) large timing variations (Ga96) soft duration contrasts (Ga96) slow tone attacks (Ko76) flat micro-intonation (Ba97) slow vibrato (Ko00) final ritardando (Ga96) • FEAR staccato articulation (Ju97a) very low sound level (Ju00) large sound level variability (Ju99) fast mean tempo (Ju99) large tempo variability (Ju99) large timing variations (Ga96) soft spectrum (Ju00) sharp micro-intonation (Oh96b) fast, shallow, irregular vibrato (Ko00) Negative Valence

  35. Lens model: quantifies the expressive communication between performer and listener

  36. Example: SADNESS The musician Musical communication Modelling music performance • Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression

  37. Score Model Example: SADNESS IOI deviations articulation dB deviations

  38. Synthesis of Emotion: Listening Test Results The musician Musical communication Modelling music performance • Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression

  39. Conclusions The musician Musical communication Modelling music performance • Emotional colouring Applications The ListenerRecognition of emotion Visualisation of musical expression Emotional expression can be derived directly from the music score, simply by enhancing music structure

  40. Dead-pan Natural Happy Angry Sad Solemn Better Monophonic Ringtones! The musician Musical communication Modelling music performance Emotional colouring • Applications The ListenerRecognition of emotion Visualisation of musical expression Mozart G minor Today (nom.) Natural :-) Happy >:-( Angry :-( Sad =|:-| Solemn www.notesenses.com

  41. Better Polyphonic Ringtones! The musician Musical communication Modelling music performance Emotional colouring • Applications The ListenerRecognition of emotion Visualisation of musical expression Usher, Burn Mechanical Musical Happy Jennifer Ellison, Bye Bye Boy Mechanical Musical Romantic www.notesenses.com

  42. pDM –performance rules in real-time Anders Friberg + MEGA project (IST EU) The musician Musical communication Modelling music performance Emotional colouring • Applications The ListenerRecognition of emotion Visualisation of musical expression

  43. Real-time Visualization of Musical Expression

  44. Background The musician Musical communication Modelling music performance Emotional colouring Applications • The ListenerRecognition of emotion Visualisation of musical expression • Feel-Me project • Design a computer program for teaching students to play expressively • The system includes a tool for automatic extraction of acoustic cues (CUEX): • pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum

  45. Aim The musician Musical communication Modelling music performance Emotional colouring Applications • The ListenerRecognition of emotion Visualisation of musical expression Design a tool for real-time visual feedback to expressive performance Mapping of acoustic cues: • Non-verbal • Intuitive • Informative (including emotional expression) Previous studies: cross-modality speeds stimuli discrimination

  46. Recognition of Emotion The musician Musical communication Modelling music performance Emotional colouring Applications The Listener • Recognition of emotion Visualisation of musical expression TempoSound levelArticulation... Cue analysis Expression mapper Audio Emotion Implementations Mult. Regression CUEX Simplified real-timeversion Fuzzy inspired

  47. Experiment The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion • Visualisation of musical expression 2 melodies, Brahms (minor) & Haydn (Major) 3 instruments (piano, guitar, saxophone) 12 performances per instrument (12 emotional intentions) 24 colour nuances 8 levels of hue 2 levels of brightness 2 levels of saturation 2 groups of 11 subjects each (1 group per melody)

  48. Experiment: main results The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion • Visualisation of musical expression HUE Happiness Yellow Fear  Blue Sadness  Violet & Blue Anger  Red Love  Blue & Violet BRIGHTNESS Observed tendency: Minor tonality  Low brightness (Dark colours) Major tonality  High brightness (Light colours) Interaction involving sadness: Even for major tonality low brightness is preferred for all instruments

  49. Experiment: main results The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion • Visualisation of musical expression Similar colour palettes within instruments

  50. Experiment: main results

More Related