Music information retrieval systems

1. Music information retrieval systems Author: Amanda Cohen

2. Music Information Retrieval Systems • Based on content from the following webpage: http://mirsystems.info/index.php?id=mirsystems • Other good sources on MIR and MIR systems • http://www.music-ir.org - Virtual home of music information retrieval research • http://www.ismir.net - The International Symposium on Music Information Retrieval

3. Audentify! • Developers: F. Kurth, A. Ribbrock, M. Clausen • Relevant Publications: • Kurth, F., Ribbrock, A., Clausen, M. Identification of Highly Distorted Audio Material for Querying Large Scale Data Bases. 112th Convention of the Audio Engineering Society, May 2002, Munich, Convention Paper • Kurth, F., Ribbrock, A., Clausen, M. Efficient Fault Tolerant Search Techniques for Full-Text Audio Retrieval. 112th Convention of the Audio Engineering Society, May 2002, Munich, Convention Paper • Ribbrock, A. Kurth, F. A Full-Text Retrieval Approach to Content-Based Audio Identification. International Workshop on Multimedia Signal Processing. St. Thomas, US Virgin Islands, December 9-11, 2002 • Kurth, F. A Ranking Technique for fast Audio Identification. International Workshop on Multimedia Signal Processing. St. Thomas, US Virgin Islands, December 9-11, 2002 • Clausen, M., Kurth, F. A Unified Approach to Content-Based and Fault Tolerant Music Recognition, IEEE Transactions on Multimedia. Accepted for publication

4. Audentify • System Description • Takes signal queries (1-5 seconds, 96-128 kbps) • Searches by audio fingerprint • Returns a file ID that corresponds with a song in the database • Currently a part of the SyncPlayer system

5. SyncPlayer • Developers: F. Furth, M. Muller, D. Damm, C. Fremerey, A. Ribbrock, M. Clausen • Relevant Publications: • Kurth, F., Müller, M., Damm, D., Fremerey, Ch. Ribbrock, A., Clausen, M. SyncPlayer - An Advanced System for Multimodal Music Access, Proceedings of the 6th International Conference on Music Information Retrieval (ISMIR 2005), London, GB • Kurth, F., Müller, M., Ribbrock, A., Röder, T., Damm, D., Fremerey, Ch. A Prototypical Service for Real-Time Access to Local Context-Based Music Information. Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR 2004), Barcelona, Spain. http://www-mmdb.iai.uni-bonn.de/download/publications/kurth-service-ismir04.pdf • Fremerey, Ch., SyncPlayer - a Framework for Content-Based Music Navigation,Diplomarbeit at the Multimedia Signal Processing Group Prof. Dr. Michael Clausen, University of Bonn, 2006, Bonn, Germany • URL: http://audentify.iai.uni-bonn.de/synchome/index.php?pid=01

6. SyncPlayer • System Description • Query type(s): audio files (mp3, wav, MIDI), lyrics, MusicXML, Score scans (primary data) • Generates “derived data” from query • extracts features • generates annotations • compiles synchronization data • Submitted to SyncPlayer Server, which can perform three services (at present) • audio identification (through audentify) • provide annotations for a given song • retrieval in lyrics annotation • SyncPlayer Client: audio-visual user interface, allow user to playback, navigate, and search in the primary data

7. ChoirFish • Developers: A. Van Den Berg, S. Groot • Relevant Publications: • Groot, S., Van Den Berg, A., The Singing Choirfish: An application for Tune Recognition, Proceedings of the 2003 Speech Recognition Seminar, LIACS 2003 • URL: http://www.ookii.org/university/speech/default.aspx

8. ChoirFish • System Description: • Query by humming • Contour features used for matching • Used Parson’s Code to determine contour • Code is based on the direction of note transitions • 3 characters for each possible direction: • R: The note is the same frequency as the previous note • D: The note is lower in frequency than the previous note • U: The note is higher in frequency than the previous note • Generated by changing the audio to the frequency domain via Fast Fourier Transform and using the highest frequency peak to determine pitch and pitch change

9. CubyHum • Developers: S. Pauws • Relevant Publications: • Pauws, S., CubyHum: A Fully Operational Query by Humming System, ISMIR 2002 Conference Proceedings (2002): 187--196, doi:10.1.1.108.8515 • PDF of paper: http://ismir2002.ismir.net/proceedings/02-FP06-2.pdf

10. CubyHum • System Description: • Query by Humming: user queries system by humming the desired song • Pitch is estimated by computing the sum of harmonically compressed spectra (sub-harmonic summation, or SHS). • Musical events (note onsets, gliding tones, inter-onset-intervals) are detected • Query is transformed via quantization into musical score, which is used to create a MIDI melody for auditory feedback • Approximate pattern matching used to find matching song • Distance between melodies defined based on interval sizes and duration ratios to compensate for imperfect query (people don’t always hum the correct melody in the correct key)

11. Fanimae • Developers: Iman S.H. Suyoto, Alexandra L. Uitdenbogerd and Justin Zobel • Relevant Publications • Suyoto, I.S.H., Uitdenbogerd, A.L., Simple efficient n-gram indexing for effective melody retrieval, Proceedings of the Annual Music Information Retrieval Evaluation eXchange, 2005 • URL: http://mirt.cs.rmit.edu.au/fanimae/

12. Fanimae • System Description: • Desktop Music Information Retrieval System • Search by symbolic melodic similarity • Query: a melody sequence that contains both pitch and duration information • Melody sequence is standardized • Intervals are encoded as a number of semitones, with large intervals being reduced • Coordinate matching used to detect melodic similarity • Query is split into n-grams of length 5, as are any possible answers • count the common distinct terms between query and possible answer • return results ranked by similarity

13. Foafing the Music • Developers: Music Technology Group of the Universitat Pompeu Fabra • Relevant Publications: • Celma, O. Ramírez, M. Herrera, P., Foafing the music: A music recommendation system based on RSS feeds and user preferences Proceedings of 6th International Conference on Music Information Retrieval; London, UK, 2005, http://ismir2005.ismir.net/proceedings/3119.pdf • URL: http://foafing-the-music.iua.upf.edu

14. Foafing the Music • System Description: • Returns personalized music recommendations based on a user’s profile (listening habits, location) • Bases recommendation information on info gathered across the web • Similarity between artists determined by their relationships between one another (ex: influences, followers) • Creates RSS feed for news related to favorite artists • Computes musical similarity between specific songs

15. Meledex/Greenstone • Developers: McNab, Smith, Bainbridge, Witten • Relevant Publications: • McNab, Smith, Bainbridge, Witten, The New Zealand digital library MELody inDEX, D-Lib Magazine, May 1997 • URL: http://www.nzdl.org/fast-cgi-bin/music/musiclibrary

16. Meldex/Greenstone • System Description: • Receives audio queries (hummed, sung, or played audio) • Filters audio to get fundamental frequency • Input sent to pitch tracker, which returns average pitch estimate for each 20ms • Note duration can optionally be taken into account, as well as user defined tuning • Results found using approximate string matching based on melodic contour

17. Musipedia/Melodyhound/Tuneserver • Developer: Rainer Typke • Relevant Publications: • Prechelt, L., Typke, R., An Interface for Melody Input. ACM Transactions on Computer-Human Interaction, June 2001 • URL: http://www.musipedia.org

18. Musipedia/Melodyhound/Tuneserver • System Description • Query by humming system • Record sound, system converts into sound wave • Converts query sound wave into Parson’s Code • Match by melodic contour • Determine distance between query and possible results via editing distance (calculate the number of modifications necessary to turn one string into the other) • Return results with smallest distance

19. MIDIZ • Developers: Maria Cláudia Reis Cavalcanti, Marcelo Trannin Machado , Alessandro de Almeida Castro Cerqueira, Nelson SampaioAraujoJúnior and Geraldo Xexéo • Relevant Publications: • Cavalcanti, Maria Cláudia Reis et al. MIDIZ: content based indexing and retrieving MIDI files. J. Braz. Comp. Soc. [online]. 1999, vol. 6, no. 22008-11-02]. http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0104-65001999000300002&lng=&nrm=iso ISSN 0104-6500. doi: 10.1590/S0104-65001999000300002

20. MIDIZ • System description • Database that stores, indexes, searches for and recovers MIDI files based on the description of a short musical passage • Allows for non-exact queries • Musical sequence is based on intervals between notes • Uses wavelet transform and a sliding window in the melody • Window defines a note sequence of a given size (2^k) and moves through the song note by note • Each sequence in the window is converted into a vector storing the interval distances • First note in a sequence is assigned the value 1 • Values of the following notes are determined by their chromatic distance in relation to the first note • Those values are added together in pairs, and the result is converted into coordinates in the final vector • Songs in database are stored in a BD Tree, determined by Discriminator Zone Expression • Completed vector of query is submitted to tree, similar results are returned

21. Mu-seek • Developers: Darfforst LLP • URL: http://www.mu-seek.com/ • System Description: • Search by title, lyrics, tune fragment, or MIDI • Uses pitch, contour, and rhythm to find matches

22. MusicSurfer • Developers: Music Technology Group of the Universitat Pompeu Fabra • Relevant Publications: • Cano et al. An Industrial-Strength Content-based Music Recommendation System, Proceedings of 28th Annual International ACM SIGIR Conference; Salvador, Brazil 2005. http://mtg.upf.edu/files/publications/3ac0d3-SIGIR05-pcano.pdf • URL: http://musicsurfer.iua.upf.edu/

23. MusicSurfer • System Description: • Automatically extracts features from songs in database based on rhythm, instrumentation, and harmony • Uses spectral analysis to determine timbre • Uses those features to search for similar songs

24. NameMyTune • Developers: Strongtooth, Inc • URL: http://www.namemytune.com/ • System Description: • User hums query into microphone • Results are found by other users determining what the song is

25. Orpheus • Developers: Rainer Typke • Relevant Publications: • Typke, Giannopoulos, Veltkamp, Wiering, van Oostrum, Using Transportation Distances for Measuring Melodic Similarity, ISMIR 2003 • URL: http://teuge.labs.cs.uu.nl/Ruu/?id=5

26. Orpheus • System Description: • Query can be example from database, hummed or whistled melody, or a MIDI file • All queries are converted into internal database format before submission • Similarity between query and results based on Earth Mover’s Distance • Two distributions are represented by signatures • Distance represents the amount of “work” required to change one signature to the other • Work = user defined distance between two signatures

27. Probabilistic “Name That Song” • Developers: Eric Brochu and Nando de Freitas • Publications: • Brochu, E., Freitas, N.D., "Name That Song!": A Probabilistic Approach to Querying on Music and Text. NIPS. Neural Information Processing Systems: Natural and Synthetic 2002 (2003)

28. Probabilistic “Name That Song” • System Description: • Query is composed of note transitions (Qm) and words (Qt). A match is found when a corresponding song has all elements of Qm and Qt with a frequency of 1 or greater. • Database songs are clustered. Query is performed on each song in each cluster until a match is found

29. Query by Humming (Ghias et al.) • Developers: Asif Ghias, Jonathan Logan, David Chamberlin, Brian C. Smith • Relevant Publications: • Ghias, A., Logan, J., Chamberlin, D., Smith, B.C., Query by Humming - Musical Information Retrieval in an Audio Database, ACM Multimedia (1995) • URL: http://www.cs.cornell.edu/Info/Faculty/bsmith/query-by-humming.html

30. Query by Humming (Ghias et al.) • System Description: • Hummed queries are recorded in Matlab • Pitch tracking is performed • Converted into a string of intervals similar to Parson’s Code (U/D/S used as characters instead of R/D/U) • Baesa-Yates/Perleberg pattern matching algorithm used to find pattern matches • Find all instances of the query string in the result string with at most k mismatches • Results returned in order of how they best fit the query

31. Search by Humming • Developers: Steven Blackburn • Relevant Publications: • Blackburn, S. G., Content Based Retrieval and Navigation of Music Using Melodic Pitch Contours. PhD Thesis, 2000 • Blackburn, S. G., Content Based Retrieval and Navigation of Music. Masters, 1999 • DeRoure, D., El-Beltagy, S., Blackburn, S. and Hall, W., A Multiagent System for Content Based Navigation of Music. ACM Multimedia 1999 Proceedings Part 2, pages 63-6. • Blackburn, S. G. and DeRoure, D. C., A tool for content based navigation of music. Proceedings of ACM Multimedia 1998, pages 361—368 • DeRoure, D. C. and Blackburn, S. G., Amphion: Open Hypermedia Applied to Temporal Media,Wiil, U. K., Eds. Proceedings of the 4th Open Hypermedia Workshop, 1998, pages 27--32. • DeRoure, D. C., Blackburn, S. G., Oades, L. R., Read, J. N. and Ridgway, N., Applying Open Hypermedia to Audio, Proceedings of ACM Hypertext 1998, pages 285--286. • URL: http://www.beeka.org/research.html

32. Search by Humming • System Description: • Takes query by humming, example, or MIDI • Queries and database contents represented by gross melodic pitch contour • Within database, each track is stored as a set of overlapping sub-contours of a constant length • Distance between songs is determined by the minimum cost of transforming one contour into another (similar to EMD) • Query is expanded into a set of all possible contours of the same length as the database’s sub-contours • A score is calculated for each file based on the number of times a contour in the expanded query set occurs in the file. Results are sorted in order of score

33. SOMeJB (The SOM enhanced JukeBox) • Developers: Andreas Rauber, Markus Frühwirth, E. Pampalk, D. Merkl • Relevant Publications: • A. Rauber, E. Pampalk, D. Merkl, The SOM-enhanced JukeBox: Organization and Visualization of Music Collections based on Perceptual Models, Journal of New Music Research (JNMR), Swets and Zeitlinger, 2003 • E. Pampalk, A. Rauber, D. Merkl, Content-based Organization and Visualization of Music Archives In: Proceedings of ACM Multimedia 2002, pp. 570-579, December 1-6, 2002, Juan-les-Pins, France • A. Rauber, E. Pampalk, D. Merkl, Using Psycho-Acoustic Models and Self-Organizing Maps to Create a Hierarchical Structuring of Music by Musical Styles, Proceedings of the 3rd International Symposium on Music Information Retrieval (ISMIR 2002), pp. 71-80, October 13-17, 2002, Paris, France. • A. Rauber, E. Pampalk, D. Merkl, Content-based Music Indexing and Organization, Proceedings of the 25. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 02), pp. 409-410, August 11-15, 2002, in Tampere, Finland • A. Rauber, and M. Frühwirth, Automatically Analyzing and Organizing Music Archives, Proceedings of the 5. European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2001), Sept. 4-8 2001, Darmstadt • URL: http://www.ifs.tuwien.ac.at/~andi/somejb

34. SOMeJB (The SOM enhanced JukeBox) • System Description: • Interface is a static, web-based map where similar pieces of music are clustered together • Music organized by a novel set of features based on rhythm patterns in a set of frequency bands and psycho-acoustically motivated transformations • Extracts features that apply to loudness sensation (intensity), and rhythm • Self-organizing map algorithm is applied to organize the pieces on a map (trained neural network)

35. SoundCompass • Developers: Naoko Kosugi, Yuichi Nishihara, Tetsuo Sakata, Masashi Yamamuro and Kazuhiko Kushima, NTT Laboratories • System Description: • User sets a metronome and hums melody in time with clicks • Database songs have three feature vectors • Tone Transition Feature Vector: contains the dominant pitch for each 16-beat window • Partial Tone Transition Feature Vector: Covers a time window different from the Tone Transition Feature Vector • Tone Distribution Feature Vector: histogram containing note distribution • Query is matched against each of the vectors, results are combined by determining the minimum

36. Tararira • Developers: Ernesto López, Martin Rocamora, Gonzalo Sosa • Relevant Publications: • E. Lopez y M. Rocamora. Tararira: Sistema de búsqueda de música por melodía cantada. X Brazilian Symposium on Computer Music. October, 2005. • URL: http://iie.fing.edu.uy/investigacion/grupos/gmm/proyectos/tararira/

37. Tararira • System Description: • User submits a hummed query • Pitch tracking applied to query • Audio segmentation determines note boundaries • Melodic analysis adjusts pitches to tempered scale • Results found by coding query note sequence, find occurrences using flexible similarity rules (string matching), and refining the selection using pitch time series

38. TreeQ • Developer: Jonathan Foote • Publications: • Foote, J.T., Content-Based Retrieval of Music and Audio, C.-C. J. Kuo et al., editor, Multimedia Storage and Archiving Systems II, Proc. of SPIE, Vol. 3229, pp. 138-147, 1997 • URL: http://sourceforge.net/projects/treeq/

39. TreeQ • System Description: • Primarily query by example, can also search by classification • Tree based supervised vector quantizer is built from labeled training data • Database audio is parameterized via conversion into MFCC and energy vectors • Each resulting vector is quantized into the tree • Vector space divided into “bins”, any MFCC vector will fall into one bin • A histogram based on the distribution of MFCC vectors into each bin is created for query and database audio • Songs matched based on histograms of feature counts at tree leaves • Distance is determined using Euclidian distance between corresponding templates of each audio clip • Results sorted by magnitude and returned as a ranked list

40. VisiTunes • Developers: Scott McCaulay • Further Information: www.slis.indiana.edu/research/phd_forum/2006/mccaulay.doc • URL: http://www.naivesoft.com/ • System Description: • Uses audio content of songs to calculate similarity between music and creates playlists based on the results • Converts sample values of each frame to frequency data • Extracts sum total of sound energy by frequency band • Uses results to simplify audio data into 256 integer values for fast comparison