Coached Active Learning for Interactive Video Search

1. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

2. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

3. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

4. Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University, China

6. Search “car on road” with Google

7. Search “car on road” with Google

9. Similar to the Automatic Search task in TRECVID

10. Interactive Search Relevance Feedback

11. Interactive Search Query Modeling, i.e., to figure out what the searcher wants? Relevance Feedback

12. Interactive Search Querying Strategy, i.e., how to keep the searcher’s patience on labeling? Query Modeling, i.e., what the searcher wants? Relevance Feedback

13. Query Modeling is indeed a task of Space Exploration

14. Search & Space Exploration

15. Search & Space Exploration

16. Feature Space with Unlabeled Instances

17. Feature Space with Unlabeled Instances

21. Space Exploration for Query Modeling How to explore effectively? Query Distribution Querying Strategy: which instances to query next (i.e., being presented to the searcher for labeling)?

22. The Goal of Query Modeling in Terms of Space Exploration • A reasonable querying strategy • To find the Query Distribution (i.e., the distribution of the relevant instances) ASAP • To satisfy Searchers with Relevant Samples ASAP

23. Active Learning with Uncertainty Sampling – A Popular and Effective Strategy

24. Active Learning with Uncertainty Sampling The 1st Round – Training Classifier

25. Active Learning with Uncertainty Sampling The 1st Round – Query the Searcher

26. Active Learning with Uncertainty Sampling The 2nd Round – Training Classifier

27. Active Learning with Uncertainty Sampling The 2nd Round – Query the Searcher

28. Active Learning with Uncertainty Sampling The 3rd Round

29. Active Learning with Uncertainty Sampling The 4th Round

30. Active Learning with Uncertainty Sampling The 5th Round

31. Active Learning with Uncertainty Sampling

32. Active Learning with Uncertainty Sampling may not be a “kindred soul” with Video Search

33. Active Learning with Uncertainty Sampling may not be a “kindred soul” with Video Search

34. Active Learning with Uncertainty Sampling may not be a “kindred soul” with Video Search The spirit of Active Learning(to reduce labeling efforts) The goal of Query Modeling (to harvest relevant instances) Also found by A.G. Hauptmann and et al. [MM’06]

35. The Dilemma of Exploration and Exploitation • Exploration: to exploremore unknown area and get more about the Query Distribution and to boost future gains • Exploitation: to harvestmore relevant instances and to obtain immediate rewards

36. Our Idea – The Query Distribution, even Unknown, is Predictable

37. The Idea of Coached Active Learning • Estimate the Query Distribution after each round to avoid the risk of learning on a completely unknown distribution

38. The Idea of Coached Active Learning • Estimate the Query Distribution pdf of the underlying Query Distribution

39. The Idea of Coached Active Learning • Estimate the Query Distribution • Query users with instances picked from both dense and uncertain areas of the distribution • Balance the proportion of the two types of instances

40. The Idea of Coached Active Learning Modeling the Query Distribution Balancing the Exploration and Exploitation

41. Modeling the Query Distribution

42. Modeling the Query Distribution ? ? ? ? ? ? ? ? ? L+: An incomplete sampling of the Query Distribution

43. Modeling the Query Distribution • An indirect way of estimating the pdf of the Query Distribution • Find training examples which are statistically from the same distribution as L+ • Use the pdf(s) of those training examples to estimate that of the query ? ? ? ? ? ? ? ? ?

44. Modeling the Query Distribution • A practical implementation • Training: organize training examples into nonexclusive semantic groups • Testing: check L+with each group, see whether they are statistically from the same distribution using two-sample Hotelling T-Square Test • Testing: estimate Query Distribution with GMM ? ? ? ? ? ? ? ? ?

45. Modeling the Query Distribution • Rationale of the implementation • Samples from the same distribution may carry similarsemantics • Semantic groups (which passed the test) then reflect the searcher’s query intention

46. Modeling the Query Distribution

47. Modeling the Query Distribution Distribution of the semantic group “road”

48. Modeling the Query Distribution Distribution of the semantic group “road” Distribution of the semantic group “car” Distribution of the query “car on road”

49. Balancing the Exploration and Exploitation

50. Balancing the Exploration and Exploitation • The priority of selecting an instance x to query next (i.e., present to the searcher for labeling) Balancing Factor Exploitative Priority Explorative Priority