Using Relevance Feedback in Multimedia Databases

1. Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7th International Conference on VISual Information Systems at 10th International Conference on Distributed Multimedia Systems September 9, 2004

2. Roadmap • Time series in multimedia databases and their similarity measures • Euclidean distance and its limitation • Dynamic time warping (DTW) • Global constraints and R-K Band • Relevance Feedback and Query Refinement • Experimental Evaluation • Conclusions and future work

3. What are Time Series • A collection of observations made sequentially in time. • People measure things… and things…change over time… • Their blood pressure • George Bush's popularity rating • The annual rainfall in San Francisco • The value of their Google stock

4. Time Series in Multimedia Databases Image data may best be thought of as time series…

5. Image to Time Series

6. Video to Time Series Steady pointing Hand moving to shoulder level Hand moving down to grasp gun Hand moving above holster Hand at rest

7. Time Series in Multimedia Databases Video George Washington’s Manuscript

8. Class A Class B Which class does belong to? Classification in Time Series Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.

9. Euclidean Distance Metric Given 2 time series Q = q1, …, qn and C = c1, …, cn their Euclidean distance is defined as Q C

10. Limitations of Euclidean Metric Very sensitive to some distortion in the data Training data consists of 10 instances from each of the 3 classes Perform a 1-nearest neighbor algorithm, with “leaving-one-out” evaluation, averaged over 100 runs. Euclidean distance Error rate: 29.77% DTW Error rate: 3.33 %

11. Dynamic Time Warping (DTW) Euclidean Distance One-to-one alignments Time Warping Distance Non-linear alignments are allowed

12. Q C How Is DTW Calculated? (I) Warping path w

13. (i-1, j) (i, j) (i-1, j-1) (i, j-1) How Is DTW Calculated? (II) Each warping path w can be found using dynamic programming to evaluate the following recurrence: where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimum cumulative distance among the adjacent cells.

14. Global Constraints (I) Prevent any unreasonable warping Sakoe-Chiba Band Itakura Parallelogram

15. Global Constraints (II) A Global Constraint for a sequence of size m is defined by R, where Ri = d 0  d  m, 1  i  m. Ri defines a freedom of warping above and to the right of the diagonal at any given point i in the sequence. Ri Itakura Parallelogram Sakoe-Chiba Band

16. Ratanamahatana-Keogh Band (R-K Band) Solution: we create an arbitrary shape and size of the band that is appropriate for the data we want to classify.

17. How Do We Create an R-K Band? First Attempt: We could look at the data and manually create the shape of the bands. (then we need to adjust the width of each band as well until we get a good result) 100 % Accuracy!

18. Learning an R-K Band Automatically Our heuristic search algorithm automatically learns the bands from the data. (sometimes, we can even get an unintuitive shape that give a good result.) 100 % Accuracy as well!

19. R-K Band Learning With Heuristic Search

20. R-K Band Learning in Action!

21. Classification Examples with R-K Bands

22. Face Classification

23. Relevance Feedback • A well-known and effective method in improving the query performance, especially in text-mining domains. • Refining the query based on user’s reaction • Only relatively little research has been done on relevance feedback in images or multimedia data.

24. Query Refinement Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.

25. Experiment: Datasets • Gun Problem • Leaf Dataset • Handwritten Word Spotting data

26. Experimental Design Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval. • Show the 10 best matches (k-nearest neighbors). • User ranks each result. • Accumulatively build the training set. • Learn an R-K band according to the current training data. • Generate a new query (query refinement), and repeat.

28. Results: Gun

29. Results: Leaf

30. Results: Wordspotting

31. Conclusions • Different shapes and widths of the band contributes to the classification accuracy / precision. • We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.

32. Future Work • Investigate other choices that may make envelope learning more accurate. • Heuristic functions • Search algorithm (refining the search) • Is there a way to always guarantee an optimal solution? • Examine the best way to deal with multi-variate time series for more complex data. • Explore other utilities of R-K Band and relevance feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.

33. Thank You Questions? Contact: ratana@cs.ucr.edu eamonn@cs.ucr.edu Homepage: http://www.cs.ucr.edu/~ratana All datasets are publicly available at: UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA