Query Adaptive Late Fusion for Image Search and Person Re-identification

1. Query Adaptive Late Fusion for Image Search and Person Re-identification Liang Zheng, Shengjin Wang, Lu Tian, Fei He, Ziqiong Liu, and Qi Tian 2015-7-29

2. Outline Introduction Problem and Motivation Related Work Method Experiments and Comparisons Conclusions and Future Work

3. Introduction query Image Search query

4. query Introduction Cam-a database Cam-b Person Re-identification

5. Introduction • Some statements: • Color is a good feature • Statistically, color is a good feature on a certain dataset • Color is a good feature for a certain image • We want to use good features for every query image. × √ √

6. Introduction Rank list 1 Feature 1 Late Fusion Cosine distance with database images Pipeline Final rank list Query Feature 2 Rank list 2 …… …… Feature K Rank list K

7. Problem × Color √ Color √ For a given query image, we do not know whether features 1, 2, …, K are effective or not. Shape × Shape

8. Related Work Combining local and global features on index-level Co-indexing (Zhang et al, ICCV 2013, PAMI 2015) Globally similar images All features have fixed weights Inverted Index

9. Related Work Combine different features on the rank level Graph Fusion (Zhang et al, ECCV 2012, PAMI 2014) Feature 1 Feature 2 Feature 1 + Feature 2 All features have fixed weights

10. Method - Similarity Function For images q and d, their similarity score under feature i is. Their similarity function after fusion Kfeatures is, where wq(i) is the query-adaptive weight of feature for query .

11. Method - Motivation Good feature: L-shaped score curve Bad Feature: Flat score curve

12. Method Final score curve Original score curve Normalized score curve Step 1. Normalize the original score curves Step 2. Calculate the area under the normalized score curve. Small area -> good feature, vice versa. Step 3. Merge all the original score curves with Eq. 1. The query-adaptive weights negatively correlates with the area under the normalized score curve.

13. Method - Normalization Why normalization? For some features, the score curve may have a “high tail”, reflecting intrinsic property of a feature.

14. Method - Normalization Offline step a randomly selected query How to normalize? 1 million arbitrarily retrieved images from Flickr. Randomly select Q images as query. Search the database with these queries and K features. Each feature will have Q score curves Reference Set

15. Method - Normalization Online step Reference Set Given a query image, we have K original score curves For each original score curve Remember We have Q reference score curves We find the nearest reference score curve to the tail of the original score curve. We subtract the selected reference curve from the original curve. We linearly normalize the residual: max = 1, min = 0. After normalization, the area under the score curve is negatively related to the weight of the feature.

16. Method – similarity function • For images and , their similarity score under feature is . The similarity function after fusion all features is, • Where is the area under the normalized score curve for feature

17. Method - Normalization The impact of normalization We eliminate the influence of the tail For some features, the score curve may have a “high tail”

18. Method The impact of normalization

19. Experiments Datasets Features Good Features Bad Features

20. Experiments and Comparisons Good Features：BoW, HSV, CNN Bad Features：GIST, Random Normalizationvs No normalization

21. Experiments and Comparisons Good Features：BoW, HSV, CNN Bad Features：GIST, Random Ours vs Co-indexing (ICCV’13, PAMI’15)

22. Experiments and Comparisons We add as many as 20 random projection features to the system. Robustness to Bad Features

23. Experiments and Comparisons Time cost Comparison with state-of-the-arts

24. Experiments and Comparisons • Experiment settings on VIPeR dataset • We use a BoW representation – 5600 dim • For each local patch, we extract • 20-dim HS histogram • 11-dim Color Names (CN) • LBP • HOG Feature 1: HS BoW Feature 2: CN BoW Feature 3: LBP BoW Feature 4: HOG BoW Feature 5: eSDC (Zhao et al, CVPR 2013) Zheng et al, Person re-identification meets image search. arXiv preprint:1502.02171 (2015).

25. Experiments and Comparisons Results on VIPeR dataset

26. Conclusions A query-adaptive fusion strategy is proposed Bad features are down-weighted Good features are up-weighted Reference set is independent on the test image database Does not require extensive offline steps Works well when database changes Our method is resistant to bad features, so it enables “safe” search

27. Thank you!