A Subsequence Matching with Gaps-Range-Tolerances Framework: A Query-By-Humming Application

1. A Subsequence Matching withGaps-Range-Tolerances Framework:A Query-By-Humming Application Alexios Kotsifakos University of Texas at Arlington, University of Athens (Greece) Joint work with Panagiotis Papapetrou (Aalto University, Finland) Jaakko Hollmen (Aalto University, Finland) Dimitrios Gunopulos (University of Athens, Greece)

2. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

3. Introduction – Motivation Suppose you hear a song but cannot recall its name… • One solution: Hum a short part of it and perform a search on a music repository to find it or songs similar to it. • Query-By-Humming (QBH):Given a hummed query song search a music db for the K most similar songs. • Subsequence Matching: Given a query sequence and a db of sequences, identify the subsequence in db that best matches the query.

4. Subsequence Matching in a Database query What subsequence of any database sequence is the best match for Q ? database

5. Converting music pieces to time series • Each music piece is a sequence of notes. • Notes consist of pitch (frequency) and duration. Part of the Happy Birthday music score. The 2D time series representation of the above music score. Pitch interval : distance between two pitches. IOIR : duration ratio of two consecutive notes

6. QBH: A real example Hummedquery: “Yesterday” (Beatles) Target: “Yesterday”

7. QBH: A real example Hummed query:“Yesterday” (Beatles) Matched target subsequence

8. QBH: A noisy application domain • Slight or more serious errors due to instant or temporary key or tempo loss  error-tolerant method. • Allow skipping a bounded number of elements in both query and target sequences due to serious humming errors. • Constrain the length of the matching subsequence. • Optional constraint to the min. number of matching elements.

9. Our contributions • A subsequence matching framework that allows for • a constrained number of gaps on both query and target sequences, • variable tolerance levels in the matching (no probabilistic model), • a matching range that constrains the max. match length, • bounding the min. # of matched elements. • SMBGT: Given a query Q and a target sequence X (with |Q| << |X|) finds the subsequence of X that best matches Q, considering the framework. • Extensive comparative evaluation on QBH of several DB-based methods and a probabilistic model-based method.

10. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

11. Related Work • DP-based methods • Whole sequence matching: DTW and variants (cDTW, EDR, ERP) robust to misalignments and warps, LCSS allows for gaps, edit distance, Iliopoulos et al., etc. • Subsequence matching: SPRING finds the subsequences of evolving numerical streams closest to a query, DTWc and DTWs account for local adjustments of tempo, Han et al. (2007)performed uniform segmentation and sliding windows for near exact matching where user selects segments’ length. • n-gram methods for near exact matching • All account for 1D + fail to handle noise imposed by users. • Probabilistic-based methods (HMMs)

12. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

13. Problem Setting: Representing Musical Pieces • Expressing pitch • Absolute pitch: frequency of the note. • Pitch interval: frequency difference between two adjacent notes. • Encoding duration • Inter-Onset-Interval (IOI): difference in time onsets of two adjacent notes. • IOI Ratio (IOIR): ratio of IOIs of two adjacent notes. • Log IOI Ratio (LogIOIR): logarithm of IOIR. • To save computational time we deal with note transitions: <pitch interval, IOIR>, <pitch interval, LogIOIR> • Performed quantizations on pitch interval with modulo 12, and on LogIOIR to the closest integer or closest value in [-2, 2].

14. Problem Setting: Definitions • X = {x1, …,xn} representing music piece of length |X|, pair of real values for pitch and duration, DB={X1, …,XN }music db, X[ts : te]={xts , …,xte} subsequence of X (not always continuous), Q = {q1, …,qn }. • Definition 1 (Variable error-tolerant match):

15. Problem Setting: Definitions • Definition 2 (Common bounded-gapped subsequence): SMBGT: error-tolerant matching denoted as ε-match.

16. Problem Setting:Example and Problem Formulation • Example • α=2, β=1, r = 6, ε = 1 (absolute tol.) • Q = {6, 3, 10, 5, 3, 2, 9}, X = {1, 1, 3, 4, 6, 9, 2, 3, 1} • Q[2:6] with GQ = {2, 4, 5, 6} and X[3:8] with GX = {3, 4, 7, 8} Q = {6, 3, 10, 5, 3, 2, 9} X = {1, 1, 3, 4, 6, 9, 2, 3, 1} Problem: Given a database DB with N sequences of arbitrary lengths, a query sequence Q, and positive integers δ and r, find set of the top-K subsequences with SMBGT(Q, X) = {Q[2:6], X[3:8]}

17. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

18. SMGT • SMGT:α=β= inf. • Array a (|Q+1|)*(|X|+1). For all i in {1,…,|Q|}, j in {1,…,|X|} • Matrix s keeps in si,j for each ai,jthe start point ofits best alignment. • Online computation: two 1D arrays prev, cur, track scores and start points (prev.value, cur.value, prev.start, cur.start) of j-1 and j columns of a. • Best solution (SMGT) in best = (bestvalue, beststart, bestend) accounting for δ. • Check subsequences with length r (Reset()). Update values: • Match  ai,j = 1 • Mismatch  left cell can be inherited if length < r-1, top cell always.

19. SMBGT • Mismatch (propagation()): store the largest # of matched elements that can be propagated vertically or horizontally, not violating αand β. • Astart, Bstart : store the latest match positions in X and Q • Additional propagation check in Reset().

20. SMBGT: Example - Initialization Q = {0, -4, 1, 2, -2}, X = {0, 0, -4, 3, 0, 2, -3, 1} α = 2, β = 1, δ = 3,r = |Q| = 5, ε = 0 Alignment array α

21. SMBGT: Example - Computation

22. SMBGT: Example - Computation

23. SMBGT: Example - Computation

24. SMBGT: Example - Computation

25. SMBGT: Example - Computation

26. SMBGT: Example - Computation

27. SMBGT: Example - Computation

28. SMBGT: Example - Computation

29. SMBGT: Example - Computation

30. SMBGT: Example - Computation

31. SMBGT: Example - Computation

32. SMBGT: Example - Solution Q = 0 -4 1 2 -2 skipped X = 0 0 -4 30 2 -3 1 bestvalue = 3, beststart=2, bestend=6

33. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

34. Experimental Setup: Data & Evaluation • Music DB: 5643 freely available MIDI files. • 6 synthetic query sets, 100 queries/set of lengths 13-137 • Q0: exact segments of the db. • Q.10 – Q.50: randomly modified 10-50% elements (both dimensions) of each query in Q0, with at most 3 consecutive elements changed. • 100 hummed queries of lengths 14-76. • DP and Model-based competitor methods • DTW variants: DTWs, DTWc, SPRING modified to allow r. • Edit: Version for music modified for LogIOIR + quantizations • Iliopoulos et al.: Elastic version for subsequence matching supporting constant and variable tolerance, modified to allow r. • HMMs: Each db sequence modeled by an HMM.

35. Experimental Setup: Evaluation measures • Recall: % of queries for which correct answer is in top-K. • MRR: mean inverse rank of queries in their top-K results. • AR: average rank of all queries in the set, where rank is the # of sequences with score at least as good as that of the correct match.

36. Experimental Setup: Variable Tolerances • Pitch • Absolute and relative tolerances tested, with and for relative tolerance two cases considered • Duration • IOIR: • LogIOIR:

37. Experimental Results: Synthetic Queries (1/4) • Parameters • r = |Q|, δ = 0.9, 0.7. 0.6, 0.5, 0.35, 0.3 for Q0 - Q.50 , α = β = 3, tested both tolerance schemes, K = 20. • Best accuracy • SMBGT: variable abs. tolerance, t = 0.2. • SMGT: variable abs. tolerance, t = 0.25. • Il. et al.: constant rel. tolerance with εp = 1, εr = 4. • HMM: #states = 5, Gaussian observation distribution of the states. • Representation • ↑ noise: high recall for representations subset of lower noise levels’ • DP methods: <mod12, IOIR>, <pitch interval, IOIR> • HMM: <mod12, LogIOIR in [-2,2]>  smallest alphabet

38. Experimental Results: Synthetic Queries (2/4) • Q0 • DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5. • DTWs: 96% recall for K = 250. • HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.

39. Experimental Results: Synthetic Queries (3/4) • Q0 • DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5. • DTWs: 96% recall for K = 250. • HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.

40. Experimental Results: Synthetic Queries (4/4)

41. Experimental Results: Hummed Queries Any tol. scheme. • Tolerance • Small constant and variable t better than greater values. • Variable t better than constant. • Absolute better than relative. • SMBGT and SMGT: absolute, t = 0.2 and 0.25.

42. Outline • Motivation • Related Work • Problem Setting • SMBGT: The proposed method • Experimental Evaluation • Summary and Conclusions

43. Summary • Lack of flexibility for existing subsequence matching methods. Edit most promising, but 30 times lower recall than SM(B)GT for K = 50, α, βimprove accuracy. • Variable ε-tolerance better than constant tolerance. • Small rangefor α, βtested. Other applications  cross validation. • δ tuned according to hummers’ skills, r not exceed 1.2*|Q|. • Simple representation favor DP-based methods. • DP-based methods have similar retrieval time complexity.

44. Conclusions and Future Work • We proposed a novel subsequence matching framework. • We showed that it can be applied to noisy application domains, such as QBH. • We performed an extensive experimental evaluation on a large collection of music songs using hummed queries. • Directions for future work include • Testing the proposed method on other noisy domains. • Developing a software application based on SMBGT. • Indexing.

45. Thank you!!!

46. Questions ???

47. Appendix

48. SMBGT • Theorem: SMBGT is not metric, as it does not follow the triangle inequality. Consider: • 1D time series X1 = {3, 2}, X2 = {4}, X3 = {2, 2, 1} • ε-tolerance = 1, r = inf, δ = 0, α = β = 1 • SMBGT(X1, X2) = 1, SMBGT(X2, X3) = 0, SMBGT(X1, X3) = 2, but SMBGT(X1, X2) + SMBGT(X2, X3) < SMBGT(X1, X3).

49. SMBGT: Pseudocode

50. DP Methods – Edit distance • Latest version, extended for LogIOIR and quantizations. • PitchRange, DurationRange : max pitch interval, and LogIOIR range in DB. • Reports minj{α|Q|,j}.