Visually and Acoustically Exploring the High-Dimensional Space of Music

1. Visually and Acoustically Exploring the High-Dimensional Space of Music Lukas Bossard Michael Kuhn Roger Wattenhofer SocialCom 2009 Vancouver, Canada

2. organization by album History • Storage media • Vinyl records • Compact cassettes • Compact discs • An Album is stored on a single physical storage medium • Sequence of songs given by album • Album is typically listened to as a whole Michael Kuhn, ETH Zurich @ SocialCom 2009

3. Music today • Huge offer, easily available • filesharing, iTunes, amazon, etc. • Large collections • The entire collection is stored on a single electronic storage medium • Organization by albums (and other lists) is no longer appropriate organize by similarity Michael Kuhn, ETH Zurich @ SocialCom 2009

4. Organization by Similarity • Our Goals • Mobile application (portable player) • Play songs the user likes • Overview of a collection • Problems on mobile devices • Limited input • Limited output • Limited processing power • Limited memory • Solution • Use song coordinates provided by www.musicexplorer.org Michael Kuhn, ETH Zurich @ SocialCom 2009

5. Which songs are similar? • Goussevskaia et al., WI 2008: • Each song is positioned in a Euclidean „Map of Music“ • Similar songs are close to each other in this Euclidean space Michael Kuhn, ETH Zurich @ SocialCom 2009

6. The Map of Music • Based on usage data • „behaviour of the crowd“ • Gathered from social music platform (last.fm) • NO audio-analysis! • Underlying similarity measure • Item-to-item collaborative filtering (Amazon)[Linden et al., IEEE Internet Computing] • „users who listen to song A also listen to song B“ • Coordinates available through webservice • www.musicexplorer.org Michael Kuhn, ETH Zurich @ SocialCom 2009

7. rock Hey Jude Praise you Imagine Galvanize electronic I want it that way My Prerogative pop Using the Map • Similar songs are close to each other • Quickly find nearest neighbors • Span (and play) volumes • Create smooth playlists by interpolation • Visualize a collection • Low memory footprint • Well suited for mobile domain convenient basis to build music software Michael Kuhn, ETH Zurich @ SocialCom 2009

8. That‘s easy – is it? 10 dimensional! Michael Kuhn, ETH Zurich @ SocialCom 2009

9. Contributions Proof-of-concept application for Android devices („Google-phone“) Visual and acoustic guide to the high-dimensional music galaxy Michael Kuhn, ETH Zurich @ SocialCom 2009

10. Visual Exploration Visual Exploration Michael Kuhn, ETH Zurich @ SocialCom 2009

11. fast happy sad slow The Reference: SensMe (Sony Ericsson) Create playlist by selecting areas Based on audio-analysis Michael Kuhn, ETH Zurich @ SocialCom 2009

12. Global Overview Orientation Local Overview Requirements Our problem: 10 dimensions! Michael Kuhn, ETH Zurich @ SocialCom 2009

13. Lens Metaphor Detailed view in the center Few details in the border rings Michael Kuhn, ETH Zurich @ SocialCom 2009

14. Lens: Recursive Clustering Few details in the border regions High resolution in the center Michael Kuhn, ETH Zurich @ SocialCom 2009

15. Cake Metaphor Used to represent song clusters Michael Kuhn, ETH Zurich @ SocialCom 2009

16. The Visual Exploration Interface • Browsing • Touch cluster to bring it to the center • Playlist Generation • Select a number of seed songs • Playlist will consist of songs around these seeds • Similar to SensME (but songs are selected in a different interface) Touch to make this area the new center Michael Kuhn, ETH Zurich @ SocialCom 2009

17. Evaluation (1) • User Experiment • 9 participants • Collection (1400 songs) • 5 minutes to create playlist of 20 songs (for both systems) • Evaluation: Participants had to... • ...rate each individual song in the playlists • ...fill in a questionaire vs. Michael Kuhn, ETH Zurich @ SocialCom 2009

18. Evaluation (2) • Average song rating (scale: 0..10): • 5.5 (SensMe) • 6.3 (this paper) • Questionaire (scale: 1..5): Trade-off: Accurracy of high-dimensional space versus simplicity of interface Michael Kuhn, ETH Zurich @ SocialCom 2009

19. Acoustic Exploration Michael Kuhn, ETH Zurich @ SocialCom 2009

20. Shuffle (play songs in random order) Idea Can we do better? Yes! Idea: Learn on the fly which songs the user likes! Skip = bad song Listen = good song Michael Kuhn, ETH Zurich @ SocialCom 2009

21. Realization Supposed to be the user‘s region of interest Basic algorithm: Voronoi Tesselation First song was skipped Michael Kuhn, ETH Zurich @ SocialCom 2009

22. Improvements • Weighting • Account for strong/weak feedback • Aging • Allows to adapt to changing mood • Centering • Border regions are risky => go to center • Escaping • Sometimes play random song to avoid getting stuck somewhere Rating bar (left = skip, right = good) Michael Kuhn, ETH Zurich @ SocialCom 2009

23. References • Random shuffling (e.g. iPod-Shuffle) • Pampalk et al. (ISMIR, 2005) • Designed for (Euclidean) audio feature spaces If there are songs with dg < db: select such song with smallest dg Else: select song with largest ratio dg/db dg db Michael Kuhn, ETH Zurich @ SocialCom 2009

24. Evaluation • 9 Participants • Song ratings are used as input and for evaluation Ratings clearly better than random Diversity clearly better than Pampalk Michael Kuhn, ETH Zurich @ SocialCom 2009

25. www.musicexplorer.org Conclusion Michael Kuhn, ETH Zurich @ SocialCom 2009

26. Questions? Michael Kuhn, ETH Zurich @ SocialCom 2009