Ching-Hua Chuan Department of Math and Computer Science Barry University February 20th 2009

1. Automatic Style-Specific Accompaniment Ching-Hua Chuan Department of Math and Computer Science Barry University February 20th 2009 MUE Forum, Music Engineering, University of Miami

2. Agenda • Problem Description • Motivations, definitions, requirements, and uniqueness • ASSA System Design • Experiments • A melody accompanied in two different styles • A turing test • Quantitative Evaluations • Other research projects MUE Forum, Music Engineering, University of Miami

3. Questions • Have you ever had these experiences? • A original tune pops up in your mind • Struggling to make your own tune into a complete song and share with your friends? Description-1 MUE Forum, Music Engineering, University of Miami

4. Importance of Accompaniment A melody could be a hit A hit Without sophisticated chord arrangements and accompaniments, some creative melodies would languish as plain and immature utterances, soon discarded. Description-2 MUE Forum, Music Engineering, University of Miami

5. Not Just Any Accompaniment Accompanied by Microsoft Songsmith Radiohead’s Creep Good accompaniments should realize the musical ideas conveyed by the melody, creating a suitable atmosphere to help a story develop. Description-3 MUE Forum, Music Engineering, University of Miami

6. Style Matters Roxanne by The Police accompanied by Microsoft Songsmith “Here's what The Police's hit "Roxanne" would sound like if they'd used Microsoft Songsmith. Microsoft felt this song was Latin, so who am I to tell them they're wrong?” - azz100c on YouTube Description-4 MUE Forum, Music Engineering, University of Miami

7. Requirements • User input • Take newly created melodies by users as input. • Let users specify style without using formal musical terms. • Accompaniment • Identify salient notes in a given melody. • Identify features important to the style in examples. • Chord transitions must follow style. • Accompaniment needs to support phrase structure of melody. Description-5 MUE Forum, Music Engineering, University of Miami

8. Uniqueness of ASSA • User input • Users can specify the style they prefer by providing examples, as less as one song. • Accompaniment • Determine the importance of melodic notes according to the style. • Capable of generating new chord transitions while ensuring stylistic consistence. • Various musical features, such as scale, metrical strength, phrase structure and etc, are taken into account. Description-6 MUE Forum, Music Engineering, University of Miami

9. Agenda • Problem Description • Motivations, definitions, requirements, and uniqueness • ASSA System Design • Experiments • A melody accompanied in two different styles • A turing test • Quantitative Evaluations • Other research projects MUE Forum, Music Engineering, University of Miami

10. ASSA System Chord Tone Determination Triads Construction Chord Progression Generation chords melody Key Finding • Accompaniment Style Modeling • Chord Tone Determination: modeling the chordal tendency in melody • Chord Progression Generation: modeling the relation between adjacent chords ASSA-1 MUE Forum, Music Engineering, University of Miami

11. Chord Tone Determination Chord Tone Determination Triads Construction Chord Progression Generation chords melody • Chord Tone Determination • Chord tone vs. non-chord tone • The classifier: decision tree • Pitch in a bar is represented in terms of 73 attributes (as shown in the table) ASSA-2 MUE Forum, Music Engineering, University of Miami

12. Triads Construction Chord Tone Determination Triads Construction Chord Progression Generation chords melody • Given the generated chord tones, triads are first chosen for harmonizing the bar which shows strong triad tendency in the chord tones. input melody: e g chord tones: g d c f C or Em G triads: checkpoint checkpoint ASSA-3 MUE Forum, Music Engineering, University of Miami

13. Chord Progression Generation Chord Tone Determination Triads Construction Chord Progression Generation chords melody Chord candidate selection Neo-Riemannian Transform Tree construction and pruning Markov Chains ASSA-4 MUE Forum, Music Engineering, University of Miami

14. C Cm F Fm Am F Fm B Bm Dm Chord Progression Generation Chord Tone Determination Triads Construction Chord Progression Generation chords melody Chord candidate selection melody: Neo-Riemannian Transform CTs: e g g d c f Tree construction and pruning C or Em G Markov Chains ASSA-5 MUE Forum, Music Engineering, University of Miami

15. Chord Progression Generation Chord Tone Determination Triads Construction Chord Progression Generation chords melody Chord candidate selection Neo-Riemannian Transform Tree construction and pruning Markov Chains Example: chord C to G in C major  LR ASSA-6 MUE Forum, Music Engineering, University of Miami

16. Chord Progression Generation Chord Tone Determination Triads Construction Chord Progression Generation chords melody Chord candidate selection Triad: I bar i bar i+1 Neo-Riemannian Transform bar i+2 Tree construction and pruning bar i+3 Markov Chains Triad: I bar i+4 ASSA-7 MUE Forum, Music Engineering, University of Miami

17. Chord Progression Generation Chord Tone Determination Triads Construction Chord Progression Generation chords melody Chord candidate selection Ci bar i P(Ci, …, Cn) = P(Ci)P(Ci+1 | Ci)…P(Cn | Cn-1) = P(Ci)P(NROi, i+1)… P(NROn-1, n) Neo-Riemannian Transform . . . . . . Tree construction and pruning bar n Cn Markov Chains ASSA-8 MUE Forum, Music Engineering, University of Miami

18. Agenda • Problem Description • Motivations, definitions, requirements, and uniqueness • ASSA System Design • Experiments • A melody accompanied in two different styles • A turing test • Quantitative Evaluations • Other research projects MUE Forum, Music Engineering, University of Miami

19. melody accompanied in style 1 melody accompanied in style 2 Accompaniments in Different Styles • Experiment design ASSA system Experiment-1 MUE Forum, Music Engineering, University of Miami

20. Learning Radiohead’s Style • Melody • The training example • Anyone can play guitar by Radiohead (Pablo Honey, 1993) Experiment-2 MUE Forum, Music Engineering, University of Miami

21. Generated Accompanimentsin Radiohead’s Style G D G G D Em B Em Em Bm C G F#m G G Em Em C Em Em F#m Em G G F G F Experiment-3 MUE Forum, Music Engineering, University of Miami

22. Learning Fiona Apple’s Style • Melody • The training example • Never is a promise by Fiona Apple (Tidal, 1996) Experiment-4 MUE Forum, Music Engineering, University of Miami

23. Generated Accompanimentsin Fiona Apple’s Style G G C C C#m G Em G C Gm C G Am D G G Em C Bm G Am G D G C C C Experiment-5 MUE Forum, Music Engineering, University of Miami

24. G G D G G C C G D C#m Em G Em B Em G C Em Gm Bm C C G G Am F#m G D G G G Em Em Em C C Bm Em Em G F#m Am G Em G D G G C F G C C F Differences Experiment-6 MUE Forum, Music Engineering, University of Miami

25. A Turing Test • Choose one song from an album, train the system using the rest songs, and re-harmonize the melody of the selected song in the style learned. An example Experiment-7 MUE Forum, Music Engineering, University of Miami

26. Agenda • Problem Description • Motivations, definitions, requirements, and uniqueness • ASSA System Design • Experiments • A melody accompanied in two different styles • A turing test • Quantitative Evaluations • Other research projects MUE Forum, Music Engineering, University of Miami

27. Quantitative Evaluation • Musical metrics • Entropy and Perplexity Evaluation-1 MUE Forum, Music Engineering, University of Miami

28. Data set: songs in 5 pop/rock albums Experiments Inter-System Temperley-Sleator Harmonic Analyzer ASSA Naïve chord generator Evaluation-2 MUE Forum, Music Engineering, University of Miami

29. Results Chord Map Distribution Entropy and Perplexity Evaluation-3 MUE Forum, Music Engineering, University of Miami

30. Agenda • Problem Description • Motivations, definitions, requirements, and uniqueness • ASSA System Design • Experiments • A melody accompanied in two different styles • A turing test • Quantitative Evaluations • Other research projects MUE Forum, Music Engineering, University of Miami

31. Other Projects • Phrase Structure Analysis in Expressive Performances • Audio Onset Detection Using Machine Learning Techniques • Audio key finding • Guitar Score Interpretation • http://euclid.barry.edu/~chchuan/ • chchuan@mail.barry.edu MUE Forum, Music Engineering, University of Miami