Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval

1. Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi* , Michael R. Lyu *, Rong Jin # *Department of Computer Science & Engineering The Chinese University of Hong Kong Shatin, N.T., Hong Kong SAR # Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824, USA The 1st IEEE EMMA Workshop in conjunction with 21st IEEE ICDE, Japan, April, 2005.

2. Outline • Introduction • Background • Log-based Relevance Feedback • Coupled Support Vector Machine • Support Vector Machine • Formulation • Alternating Optimization • A Practical Algorithm • Experimental Results • Conclusion

3. Introduction • Content-based Image Retrieval (CBIR) • An important component in visual information retrieval • QBE: query-by-example based on low-level visual features • Semantic gap: low-level features, high-level concepts QBE

4. Introduction • Relevance Feedback (RF) • A powerful tool to attack the semantic gap problem • Interactive mechanism to solicit users’ feedbacks • Boost the retrieval performance of CBIR greatly • Many existing techniques already… • Problems • Regular relevance feedback needs too many rounds of interactions for achieving satisfactory results.

5. Introduction • Motivation Relevance Feedback ? User Feedback Log Can user feedback log be used to improve the regular relevance feedback? Problem

6. Background • Log-based Relevance Feedback (LRF) • Relevance Matrix: R • RF round / Log session: Nl images are marked • Elements: relevant (1), irrelevant (-1), unknown (0) Image samples -1 -1 1 1 1 -1 -1 0 1 -1 1 -1 1 -1 -1 -1 -1 -1 0 -1 -1 1 Log Sessions

7. Background • Learning Problem for LRF • Low-level image content: • User feedback log: • Multi-Modal Learning Problem

8. Coupled Support Vector Machine • Motivation • How to attack the learning problem on the two modalities? • Low-level Image content: X • User relevance feedback log: R • Support Vector Machines: superior classification performance • A Straightforward Solution: • Learn an SVM classifier on each modality respectively • For image content X, we learn an optimal weighting vector w; • For log content R, we learn an optimal weighting vector u; • Combine their results together linearly

9. Coupled Support Vector Machine • A Straightforward Solution • For the image content modality: wTx • For the user feedback log modality: uTr

10. Coupled Support Vector Machine • Disadvantages of the straightforward solution • Linear combination • Modality Consistence • Our better solution: Coupled SVM • Learn the two modalities in a unified formulation • Enforce the prediction on the two types of information to be consistent.

11. Coupled Support Vector Machine • Formulation: Coupled SVM

12. Coupled Support Vector Machine • Optimization of Coupled SVM • Hard to be solved directly • Alternating Optimization (AO) • AO: two-step optimization • Fix Y’, try to find (u, b_u), and (w, b_w) • Fix (u, b_u) and (w, b_w), try to find Y’

13. Coupled Support Vector Machine • Alternating Optimization • Fix Y’, the primal optimization is equivalent to solving the two optimization subproblems:

14. Coupled Support Vector Machine • Alternating Optimization (AO) • By introducing non-negative Lagrange multipliers, the above two subproblems can be solved

15. Coupled Support Vector Machine • Alternating Optimization (AO) • After solving (u, b_u) and (w, b_w), fixing them, the optimal Y’ can be found to fit the data as follows:

16. Coupled Support Vector Machine • Summary of AO procedure • 1) Beginning with a small value of • 2) Performing the two-step AO procedure • 3) Repeating 2) by increasing until it achieves the setting threshold • Comments on the Coupled SVM • Can be a general approach for multi-modal learning problems • Need to investigate the convergence issue of Alternating Optimization • Need to study better methods for solving the optimization problem • Require to take some practical considerations when fitting for specific problems.

17. Coupled Support Vector Machine • A Practical Algorithm • Practical considerations • Cannot engage all unlabeled samples due to response requirement for relevance feedback • Strategy for choosing unlabeled samples • Closest to the decision boundary of SVM: most informative according to active learning • Closest to the labeled samples: to avoid too much effort in learning the label information • Introducing a parameter to control the error for label correction to avoid overlarge change in the labeled set

18. Coupled Support Vector Machine • A Practical Algorithm (cont’d)

19. Coupled Support Vector Machine • A Practical Algorithm (cont’d)

20. Experimental Results • Dataset • Images selected from COREL image CDs • Two ground-truth datasets • 20-Category: each category contains 100 images, totally 2,000 • 50-Category: each category contains 100 images, totally 5,000

21. Experimental Results (cont’d) • Low-level Image Representation • Color Moment • 9-dimension • Edge Direction Histogram • 18-dimension • Canny detector, 18 bins of 20 degrees each • Wavelet-based texture • 9-dimension • Daubechies-4 wavelet, 3-level DWT • Entropies of 9 subimages are generated for the texture feature

22. Experimental Results (cont’d) • Collection of User Log Data • Log format • A log session (LS) corresponds a relevance feedback round • Each log session contains 20 images labeled by users • Log data • On 20-Category: 161 log sessions • On 50-Category: 184 log sessions

23. Experimental Results (cont’d) • CBIR GUI for collecting feedback data

24. Experimental Results (cont’d) • Performance Evaluation • Measurement Metric • Average Precision = # relevant images / # returned images • Experimental Setting • 100 queries • 20 initially labeled images • SVM: RBF kernel, parameters set via training data • Comparison Schemes • RF-SVM • traditional relevance feedback by SVM • LRF-2SVM • log-based relevance feedback by learning two SVMs respectively • LRF-CSVM • log-based relevance feedback by Coupled SVM

25. Experimental Results (cont’d) • Performance Evaluation: on 20-Category Dataset

26. Experimental Results (cont’d) • Performance Evaluation: on 50-Category Dataset

27. Experimental Results (cont’d)

28. Experimental Results (cont’d)

29. Conclusion • A log-based relevance feedback scheme was studied by integrating user feedback log into the content learning of low-level visual features in content-based image retrieval. • A general multimodal learning technique, i.e. Coupled Support Vector Machine, was proposed for studying the data with multiple modalities. • A practical algorithm by Coupled SVM was presented to attack the log-based relevance feedback problem in CBIR. • Experimental results show our proposed scheme is effective for the log-based relevance feedback problem.

30. Q&A

31. References • Chu-Hong Hoi and Michael R. Lyu, A Novel Log-based Relevance Feedback Technique in Content-based Image Retrieval, in Proc. ACM Multimedia, New York, USA, 10-16 October, pp. 24-31, 2004 • S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proc. ACM Multimedia, pages 107--118, 2001.