The Effectiveness Study of Music Information Retrieval

1. The Effectiveness Study of Music Information Retrieval Arbee L.P. Chen National Tsing Hua University 2002 ACM International CIKM Conference

2. Outline • Motivation • The Ultima Project • The 1D-List approach • The APS approach • The Effectiveness Study • Estimating all relevant references • Conclusion

3. Motivation • Various approaches were proposed to provide efficient and effective content-based retrieval of music objects • Music representations • pitch, rhythm, contour, chord • Index structures • tree-based index, list-based index, n-gram index • Query processing methods • exact match, partial match, approximate match

4. Motivation (Cont.) • A platform is needed for the evaluation of various music information retrieval methods • Efficiency • response time • Effectiveness • recall-precision curve • The Ultima project builds such a platform • Same data set and query set • Also serves as a testbed whenever new approaches are proposed

5. The Ultima Project (Cont.) Architecture • Mediator • Query processing module • Report module • Summarization module • Query generation module • Data store

6. The Ultima Project (Cont.) • Two approaches have been compared • 1D-List • APS

7. The 1D-List Approach • The 1D-List approach • Music objects are coded as melody strings • “so-mi-mi-fa-re-re-do-re-mi-fa-so-so-so” • Melody strings are organized as linked lists • Both exact and approximate matching can be handled • Exact link, insertion link, dropout link, transposition link

8. The 1D-List Approach (Cont.)

9. The APS Approach • The APS approach • Music objects are coded as sequences of music segments • four segment types to model the music contour • pitch and duration are considered • Index structures • one-dimensional and two-dimensional augmented suffix tree • Both exact and approximate matching can be handled

10. The APS Approach (Cont.) • Representation type A type B type C type D

11. The APS Approach (Cont.) A A B $ A B 3 $ B 2 $ • An example of suffix tree • A 1-D augmented suffix tree 1 The suffix tree of the string S=“ABCAB”

12. The APS Approach (Cont.) • Similarity measure • Given a query sequence Q = (i1, j1, k1) (i2, j2, k2) ... (in, jn, kn), and a candidate sequence from the database C = (i1, x1, y1) (i2, x2, y2) ... (in, xn, yn).

13. The Effectiveness Study • Traditional measures of effectiveness are precision and recall • However, the number of relevant references are usually unknown • it is unrealistic for the user to make relevant judgments to all music objects in the database

14. The Effectiveness Study (Cont.) • How to estimate the number of relevant references NR? • ASx is the set of relevant objects from the top x ranked results • RSx is the set of the top x ranked results retrieved by an approach • Assumption 1: , , • the number of the retrieved results is a function of the number of retrieved relevant objects • Assumption 2: , where B is a positive integer • Based on the two assumptions, NR can be derives as follows:

15. The Effectiveness Study (Cont.) |RSx| = 8, |ASx| = 5, |DB| = 20

16. The Effectiveness Study (Cont.) Experiment setup

17. The Effectiveness Study (Cont.) • Experiment results

18. The Effectiveness Study (Cont.) • Experiment results

19. The Effectiveness Study (Cont.) • Experiment results • 1D-list achieves a high precision in the limited range of recall, while a moderate precision for the APS family can be obtained • Comparing the APS family, the precision in a descending order is: 1D-AST (pitch), 2D-AST, and 1D-AST (duration) • In average, the effectiveness of “incipit” queries is better than “refrain” queries

20. Conclusion • The Ultima project builds a platform for evaluating the performance of various approaches of music information retrieval • A new measure for estimating the number of relevant references is proposed • Future work • Design and implement the summarization module as well as the query generation module • Extend the project for evaluating polyphonic music retrieval methods