1 / 42

The Representation, Indexing and Retrieval of Music Data at NTHU

The Representation, Indexing and Retrieval of Music Data at NTHU. Arbee L.P. Chen National Tsing Hua University Taiwan, R.O.C. http://www.cs.nthu.edu.tw/~alpchen. Outline. Content-based media data retrieval Music data retrieval Features of music data Feature indexing and matching

waneta
Download Presentation

The Representation, Indexing and Retrieval of Music Data at NTHU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Representation, Indexing and Retrieval of Music Data at NTHU Arbee L.P. Chen National Tsing Hua University Taiwan, R.O.C. http://www.cs.nthu.edu.tw/~alpchen

  2. Outline • Content-based media data retrieval • Music data retrieval • Features of music data • Feature indexing and matching • Prototypes • Reference

  3. Content-based Media Data Retrieval • Representation of media contents • features • Feature extraction from media data • Feature indexing • Query interface

  4. Content-based Media Data Retrieval • Matching query features against the feature index • approximate/partial matching • similarity measure • precision: how many of the answers are in fact correct • recall: how many of the correct answers are in fact retrieved • relevance feedback

  5. Music Data Retrieval: System Architecture

  6. Features of Music Data

  7. Features of Music Data • Static music information • The intrinsic music characteristics of music objects • Key, beat, and tempo • E.g., the Beethoven Symphony No. 5, Op. 67, C minor, 4/4, Allegro con brio • Acoustical features • Loudness, pitch, duration, bandwidth and brightness • Can be computed and represented as numericalvalues

  8. Features of Music Data • Thematic features • Themes, melodies, rhythms, and chords • Can be derived from the staff information of a music object • Melody • The melody of a song is the sequence of the pitches of all notes in the songs • E.g., the melody of the theme of the Beethoven’s Symphony No.5 is “sol – sol – sol – mi – fa – fa – fa - re”

  9. Features of Music Data • Rhythm • The rhythm of a song is the sequence of the durations of all notes in the songs • E.g., the rhythm of the theme of the Beethoven’s Symphony No.5 is “1/2-1/2-1/2-2-1/2-1/2-1/2-4” • Chord • A chord consists of three (root, third, and fifth) or more notes which sound together in harmony

  10. Features of Music Data • Coding scheme: a music object  a sequence of music segments • music segment = (segment type, segment duration, segment pitch) • four segment types: ┌┐(type A), └┘(type B), ┌┘(type C), and └┐(type D)

  11. Features of Music Data • For example, the sequence of music segments: (B,3,-3) (A,1,+1) (D,3,-3) (B,1,-2) (C,1,+2) (C,1,+2) (C,1,+1)

  12. music segment = (type, duration, pitch)

  13. Features of Music Data • Repeating Pattern • A sequence of notes appearing more than once in the music object • Efficient content-based retrieval • Semantics-rich representation • Extracting repeating patterns • Tree-based approach • Matrix-based approach

  14. Features of Music Data • Experiment 1

  15. Features of Music Data • Dissimilarity of melody strings

  16. Features of Music Data • Dissimilarity of repeating patterns

  17. Features of Music Data • Experiment 2

  18. Features of Music Data • Validity of classes

  19. Finding Repeating Patterns: Tree-based Approach • Construct an RP-tree for RP’s with lengths 2n, n  0, 1, ... • S = “ABCDEFGHABCDEFGHIJABC”

  20. Finding Repeating Patterns: Tree-based Approach • Length 1 • {A, 3, (1, 9, 19)} • {B, 3, (2, 10, 20)} • {C, 3, (3, 11, 21)} • {D, 2, (4, 12)} • {E, 2, (5, 13)} • {F, 2, (6, 14)} • {G, 2, (7, 15)} • {H, 2, (8, 16)}

  21. Finding Repeating Patterns: Tree-based Approach • Length 2 • {AB, 3, (1, 9, 19)} = {A, 3, (1, 9, 19)} 0{B, 3, (2, 10, 20)} • {BC, 3, (2, 10, 20)} = {B, 3, (2, 10, 20)} 0{C, 3, (3, 11, 21)} • {CD, 2, (3, 11)} = {C, 3, (3, 11, 21)} 0{D, 2, (4, 12)} • …

  22. Finding Repeating Patterns: Tree-based Approach • Length 4 • {ABCD, 2, (1, 9)} = {AB, 3, (1, 9, 19)} 0{CD, 2, (3, 11)} • {BCDE, 2, (2, 10)} = {BC, 2, (2, 10, 20)} 0{DE, 2, (4, 12)} • … • Length 8 • {ABCDEFGH, 2, (1, 9)} = {ABCD, 2, (1, 9)} 0 {EFGH, 2, (5, 13)}

  23. Finding Repeating Patterns: Tree-based Approach

  24. Finding Repeating Patterns: Tree-based Approach • Prune trivial patterns of length 2n, n = 0, 1, … • Let X be an RP of S, Y a substring of X, and Z a substring of Y If freq(X) = freq(Z), Y is trivial

  25. {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {CD, 2, (3, 11)} {DE, 2, (4, 12)} {EF, 2, (5, 13)} {FG, 2, (6, 14)} {GH, 2, (7, 15)} Finding Repeating Patterns: Tree-based Approach • Length 1

  26. {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} Finding Repeating Patterns: Tree-based Approach • Length 2

  27. {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} Finding Repeating Patterns: Tree-based Approach • Length 4

  28. {ABCDEFGH, 2, (1, 9)} {ABC, 3, (1, 9, 19)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} Finding Repeating Patterns: Tree-based Approach • Generate all patterns of lengths  2n, n  0, 1, ... order-1 string-joinAB1BC = ABC

  29. Finding Repeating Patterns: Tree-based Approach • Prune all trivial patterns {ABCDEFGH, 2, (1, 9)} {ABC, 3, (1, 9, 19)}

  30. Feature Indexing and Matching • 1D-List • PAT-Tree • L-Tree • Augmented Suffix Tree • Grid-Twin Suffix Tree

  31. Feature Indexing and Matching: 1D-List • There are two music objects M1 and M2 • M1: ”sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol-sol” • M2: ”do-mi-sol-sol-re-mi-fa-fa-do-re-re-mi” • The melody string of the music query • Q: ”do-re-mi” • Problem: to find whether M1 and M2 contain the melody string Q

  32. Feature Indexing and Matching: PAT-Tree • Example, songs in chord strings • Song1 : Am F2 Dm Am • Song2 : C C F C • Song3 : G E1 C D • Song4 : E1 G Am Bm

  33. Feature Indexing and Matching: PAT-Tree

  34. Prototype 1

  35. Prototype 1

  36. Prototype 2

  37. Prototype 2

  38. Prototype 2

  39. References (http://db.nthu.edu.tw) • Chen, A.L.P., M. Chang, J. Chen, J.L. Hsu, C.H. Hsu, and S.Y.S. Hua, “Query by Music Segments:An Efficient Approach for Song Retrieval,” in Proc. of IEEE Intl. Conference on Multimedia and Expo, 2000. • Chen, J.C.C. and A.L.P. Chen, “Query by Rhythm:An Approach for Song Retrieval in Music Database,” in Proc. of IEEE Intl. Workshop on Research Issues in Data Engineering, 1998. • Chou, T.C., A.L.P. Chen, and C.C. Liu, “Music Databases: Indexing Techniques and Implementation,” in Proc. of IEEE Intl. Workshop on Multimedia Data Base Management System, 1996. • Hsu, J.L., C.C. Liu, and A.L.P. Chen, “Efficient Repeating Pattern Finding in Music Databases,” in Proc. of ACM Intl. Conference on Information and Knowledge Management, 1998.

  40. References (http://db.nthu.edu.tw) • Lee, W and A.L.P. Chen, “Efficient Multi-Feature Index Structure for Music Data Retrieval,” in Proc. of SPIE Conference on Storage and Retrieval for Image and Video Databases, 2000. • Liu, C.C., J.L. Hsu, and A.L.P. Chen, “An Approximate String Matching Algorithm for Content-Based Music Data Retrieval,” in Proc. of IEEE Intl.Conference on Multimedia Computing and Systems, 1999. • Liu, C.C., J.L. Hsu, and A.L.P. Chen, “Efficient Theme and Non-Trivial Repeating Pattern Discovering in Music Databases,” in Proc. of IEEE Intl. Conference on Data Engineering, 1999.

More Related