Performing expressive music using Case-Based Reasoning

1. Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es www.iiia.csic.es/~mantaras

2. Outline • Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

3. Case-based reasoning (CBR) Solving problems by means of examples of already solved similar problems (reasoning from precedents) The task of our system is to infer, via CBR and musical knowledge, a set of expressive transformations to be applied to the notes of inexpressive musical phrases given as input The precedents are examples of expressive human interpretations

4. Saxex Components

5. SMSSnapshot

6. Saxex-CBR

7. Outline • Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

8. Case representation • Score • Musical knowledge • implication-realization, metrical structure, time-span reduction & prolongational reduction • Performance representation(solution description) • sound transformation operations: • eg: high dynamics, medium rubato, very legato, etc. SOLUTION

9. Transformations • Transformations (for each note) • Dynamics (5 possible values) • Rubato (5 possible values) • Vibrato (5 possible values) -----> 1250 possibilities • Articulation (5 possible values) • Attack (2 possible values) Vibr. Vibr. Din. Rub Art.

10. Score

11. Musical knowledge • Implication/Realization model (Narmour) • Basic structures: • Melodic direction, durational cumulation • GTTM theory (Lerdahl & Jackendoff) • Metrical structure (metrical strength of notes) • Time-span reduction (relative importance of notes within phrases or sub-phrases) • Prolongational reduction (tensions, relaxations) • Jazz Theory • Harmonic Progressions (duration, harmonic stability)

12. Implication/Realization Model

13. GTTM Theory

14. Performance

15. Outline • Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

16. A Retrieval Perspective

18. Retrieval Example Identify Search Select Problem Case Memory

19. Outline • Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

20. Saxex-Reuse • Criteria • Most similar • Majority • Minority • Continuity • Random • Fuzzy combination (DEFAULT) • Transformations • Dynamics • Rubato • Vibrato • Articulaction • Attack Vibr. Vibr. Din. Rub Art.

21. Din. Rub Art. Saxex-Reuse Example Problem Single case retrieved Din. Rub Art.

22. Saxex-Reuse (Fuzzy Combination) The notes in the human-performed musical phrases are qualified by means of five ordered linguistic values. Those for rubato are: Very Very Low Low Medium High High 1 0 20 320 Tempo Assume that SaxEx has retrieved and selected two notes whose rubato values are 72 and 190 respectively. The fuzzy combination followed by a defuzzification gives the rubato value to be applied to the input note: COA Low Medium 0.9 0.7 72 123 190

23. Outline • Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

24. Saxex Results Autumn Leaves Expressive Output phrase Inexpressive Input phrase SaxEx

25. Affective Labels • Three orthogonal dimensions • Tender-Aggressive • Sad-Joyful • Calm-Restless • Relating to notions such as • activity • tension vs. relaxation • Brightness . . .

26. SaxEx Results All of me Inexpressive Input phrase Joyful SaxEx Sad Aff. values

27. Reminding CBR & Introducing Saxex main components • Case representation • The musical knowledge • Retrieval using perspectives • Reuse • Fuzzy combination • SaxEx Results • TempoExpress • Conclusions and future work

28. TempoExpress Goal: • Changing the original performing tempo of a melody, preserving expressiveness, in the context of jazz standards. Application: Audio editing software Video / Audio post-production (video constrains audio) Why not applying uniform time stretching to the audio? Timing of notes w.r.t. beat may have to change Other expressive phenomena (e.g. ornamentations, consolidations, fragmentations) may have to change as a function of the tempo

29. TempoExpress Musical explanation: Expressivity is a result of the conception of the music by the performer, and this conception changes with tempo [Desain & Honing, 1994] Melody: “Up Jumped Spring” Uniform time stretching Recording Original tempo (180 ) Transformed tempo (90)

30. Expressive Transformations Some basic music performance concepts and their relations

31. Onset deviations at different tempos(Body and Soul A1)

32. Approches to expressive music generation • “Hand crafted” • Let a music expert formulate rules for music performance (Friberg, CMJ 1991, Friberg et al. CMJ 2000) • Machine learned • Derive expressivity rules automatically from examples (Widmer, ICMC 2000, JNMR 2002) • Eager approach: Builds a model based on many training examples and uses the learned model to solve new problems • Imitate expressivity using examples of concrete human performances by means of CBR (Arcos & Lopez de Mantaras, JNMR 1998, Lopez de Mantaras & Arcos, AI Mag 2002) • Lazy approach: Take the solution of the training example that resembles most to the new problem, and adapt it to solve it “That an expressive effect is applied only once does not mean it is insignificant” (Sundberg, MP 2001)

33. TempoExpress Architecture Desired Tempo

34. Performance Annotation Expressivity in jazz is more than timing / dynamics deviations. It is also spontaneous note ornamentations, fragmentations, etc. To model this, we define a set of Performance Events: And we use them as edit operations to obtain an edit-distance-based alignment between the score and the performance

35. Goal of the annotation process • Automatic case base acquisition Comparing Score vs recordings

36. Once I Loved Examples Body and Soul I I F C C

37. Edit (Levenshtein) distance • Goal: Assessing the distance between two sequences <S1 , S2 > • Calculated as the minimal cost of transforming S1 into S2 • Requires: • Edit operations • Cost functions

38. R I R R Edit (Levenshtein) distance

39. Case T T F Annotation examples (I)

40. Case C T T C T T T Annotation examples (II)

41. Case T T I T T Annotation examples (III)

42. Representing melodic context • Rationale: the expressivity of a performed note is not just determined by the note itself. • Ergo: Some representation of the melodic context of the note is needed • We use the Implication / Realization model of melodic structure (Narmour, 1990) • It captures the pattern of fulfillment / violation of expectations created by the melodic surface • Groups notes based on gestalt principles

43. Case Representation Repeated for each tempo

44. Retrieval • 1. Filter cases by tempo: keep cases containing performances at relevant tempos (one of the tempos is similar to the original tempo of the target melody and there is another performed tempo similar to the desired tempo to which the target melody has to be transformed) • 2. Rank the cases that passed the previous filter by I/R similarity to the score of the target melody (using edit-distance) • 3. Partition the phrases of the most similar cases into segments using the I/R parser or any other melodic segmentation algorithm (for instance Temperley, 2001) • 4. Form a “new” case base containing the obtained segments (space of partial solutions) as cases

45. Reuse • Solutions for the target melody are generated segment-wise via a best first search through the space of partial solutions (segments) • Procedure: • Retrieve best matching segment (using edit-distance) • Align target melody and retrieved segment • Transfer performance events For aligned notes T and R, let Ti(R) -----> To(R) represent the tempo transformation of note R; use the annotations differences between Ti(R) and To(R) to generate the solution To(T) from Ti(T) 4. For non-aligned target notes use UTS to transform Ti(T) into To(T)

46. TempoExpress overall view

47. Example of TempoExpress Result 55 100 bpm Uniform time stretching CBR Human

48. Experimental comparison to UTS • Four jazz standards recordings by a professional musician (12 tempos for each: 48 recordings) • 14 different phrases containing a total of 64 different melodic segments • More than 8000 tempo-transformation problems in the case base

49. TempoExpress vs. UTS as a function of the ratio of original tempo to transformed tempo. The lower plot shows the probability of incorrectly rejecting the hypothesis (that there is no difference between TempoExpress and UTS) for the Wilcoxon signed-rank test.

50. Conclusions & Future • CBR is a powerful technique to imitate human solutions (performances): Human-like output • SaxEx successfully retrieves relevant cases • Fuzzy combination increases output variation • SaxEx as a pedagogical tool: • Users can experiment with the system • Helps understanding how to use the different expressive resources • TempoExpres: an application to audio post-production that clearly outperforms UTS • Further TempoExpress experimentation with fast tempos (more example cases at fast tempos are needed) • Add within-note descriptions: • Energy envelpe features: attack, sustain, decay, tremolo • Pitch envelope features: vibrato, glissando • Add between-notes descriptions: • Articulation (legato,, staccato)