1 / 46

Songsmith:

Songsmith:. Using Machine Learning to Help People Make Music. Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research. Computational User Experiences (CUE) group. In general: HCI + (sensors, devices, machine learning, health, physiology).

penha
Download Presentation

Songsmith:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Songsmith: Using Machine Learning toHelp People Make Music Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research

  2. Computational User Experiences (CUE) group In general: • HCI + • (sensors, devices, machine learning, health, physiology)

  3. Computational User Experiences (CUE) group • Using physiological signals for input

  4. Computational User Experiences (CUE) group • Using physiological signals for input • Health and wellness

  5. Computational User Experiences (CUE) group • Using physiological signals for input • Health and wellness • Creativity support tools

  6. Songsmith: Using Machine Learning toHelp People Make Music Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research

  7. What is Songsmith? You, singing... Your music “Automatic accompaniment generation for vocal melodies” Songsmith

  8. High-Risk Live Demo (What other kind of live demo is there?)

  9. Who is Songsmith for?

  10. Today’s Talk • Overview and demo • How Songsmith works • Exposing machine learning parameters • What are people doing with Songsmith? • Creativity support tools @ Microsoft Research

  11. Songsmith: 5000’ Overview G Amin C Daug

  12. Chords from Melody • Songsmith’s core: Hidden Markov Model Observations (note vectors) Song start Chord 1 Chord 2 Chord 3 Chord 4 Song end Hidden states (chords)

  13. HMMs in 5 Minutes or Less • What does an HMM do for me? • What does an HMM want from me? States Observations

  14. HMMs in 5 Minutes or Less Possible states:C major C minor C diminished … … … … Transition probabilities:P(C Major | A Minor)? P(C Major | D Major)? P(F# Major | F# Major)? Observation probabilities:P( | A Minor)? P( | D Major)? States Observations

  15. Building our HMM • Things my HMM wants from me: • Possible states • Transition probabilities • Observation probabilities • Observations

  16. Finding the Probabilities • Data-driven • Not heuristic-driven

  17. Training Data ~300 lead sheets(vocal melodies with chords)

  18. Processing the Database • Convert all chords to five basic triads • Transpose every song into the same key • Count the number of transitions from every chord to every other chord (transition probabilities): • Count the total duration of each melody note occurring while each chord is playing (observation probabilities): Notes played over C Major:

  19. Building our HMM • Things my HMM wants from me: • Possible states • Transition probabilities • Observation probabilities • Observations

  20. Observations: what did the user sing? Input: Hard!

  21. Building a Pitch Histogram FFT

  22. Building our HMM • Things my HMM wants from me: • Possible states • Transition probabilities • Observation probabilities • Observations • Run Viterbi algorithm, get “best” sequence of chords… thank you, HMM! • One hitch: key determination…

  23. Today’s Talk • Overview and demo • How Songsmith works • Exposing machine learning parameters • What are people doing with Songsmith? • Creativity support tools @ Microsoft Research

  24. So we can choose optimal chords... so what? • My demo took 10 seconds… is that fun? • What would a songwriter do next? • How can we build creative exploration into a learning-driven system?

  25. UI: Exposing Learning Parameters • There are always hard-coded “magic numbers” in machine learning • Machine learning also use lots of learned parameters • Can we let users control those numbers?

  26. A Bad User Interface Songsmith: A fun way to make music (if you have a PhD in math and/or computer science) Observation weight: Conjugate prior: Frequency smoothing:

  27. UI: Exposing Learning Parameters • Can we let users control those numbers? • Can we let users intuitively control those numbers?

  28. Exposing Model Parametersin Songsmith

  29. The “Happy Factor”

  30. Happy Factor: Implementation • Partition database into two databases (major and minor songs) using clustering • Build separate transition probability matrix for each database • When we actually run our HMM, blend the two transition matrices together according to user input…

  31. The “Happy Factor” • Bonus question: what’s wrong with this equation?

  32. Another “Happy Factor” Example Happy Sad

  33. The “Jazz Factor”

  34. Jazz Factor: Implementation • When running our HMM, we need to make chords match the voice and each other • Computing how well each chord fits at a given position: k • log( P(this chord | what the user sang) )+ (1-k) • log( P(this chord | the previous chord) ) • Just put k on a slider!

  35. Chord Locking • Global sliders are very coarse • Chords can be “locked” by the user

  36. “Suggested Chords” • Songwriters will often explore “chord substitutions”… • …but we’re assuming our audience doesn’t know that much music theory… • Expose suboptimal marginal probabilities at each node as “suggestions”

  37. Interactive Machine Learning • Songsmith is one example of IML • Roughly: moving what used to be in the domain of ML experts into users’ hands • Related work: image classification • Fogarty et al, CHI 2008: CueFlik • Fails and Olsen, IUI 2003: Crayons • Why? • Harness end-user knowledge • Use ML as a tool for data exploration • Use ML as a tool for creative expression

  38. Today’s Talk • Overview and demo • How Songsmith works • Exposing machine learning parameters • What are people doing with Songsmith? • Creativity support tools @ Microsoft Research

  39. Today’s Talk • Overview and demo • How Songsmith works • Exposing machine learning parameters • What are people doing with Songsmith? • Creativity support tools @ Microsoft Research

  40. Dynamic Mapping of Physical Controls for Tabletop Groupware(Fiebrink, CHI 2009) • One project, two problems: • Direct-touch, tabletop input is great for collaboration… • …but suffers from serious precision issues. • Working on music alone is boring.

  41. Dynamic Mapping of Physical Controls for Tabletop Groupware(Fiebrink, CHI 2009) • Incorporate high-precision controllers into a tabletop environment • Evaluate in a collaborative audio-editing app

  42. Data-Driven Exploration of Musical Chord Sequences(Nichols, IUI 2009) • Problem: let people create and explore music by moving around in a reduced-dimensionality space • Genres, artists make for intuitive labels

  43. Data-Driven Exploration of Musical Chord Sequences(Nichols, IUI 2009) • Why isn’t this easy? • Solution(s): • Divergence-maximizing clustering, PCA

  44. Future work Future work? • Make Songsmith freakin’ amazing, and port it to a bajillion platforms, and make an amazing community Web site, and build it into audio hosts… etc… • Other applications of machine learning in creativity support tools… • Writing? Painting? Web Design? CAD? Music?

  45. Songsmith research.microsoft.com/songsmithdan@microsoft.com Live/Google: songsmith Dan Morris, Ian Simon, Sumit Basu, and the MSR Advanced Development Team Microsoft Research

More Related