Local Invariant Feature Descriptors

1. Local Invariant Feature Descriptors Bin Fan National Laboratory of Pattern Recognition (NLPR) Institute of Automation, Chinese Academy of Sciences

4. 局部图像特征描述 —— 应用 • Wide-Baseline Image Matching • Structure from Motion, Image-based Localization, Image Stitch • Object/Instance/Scene Recognition • Object Detection • Image Retrieval

5. Structure From Motion

6. Object Recognition Database …

7. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors

8. Developments: Handcrafted Descriptors • 1999, SIFT [Citation: 23819] • 2003, Shape Context • 2006, SURF [Citation: 4093] • 2008, SMD, DAISY • 2009, OSID, CS-LBP • 2010, BRIEF, HRI-CSLTP, BiCE • 2011, ORB, BRISK, LIOP, MROGH • 2012, FREAK, KAZE, SYM • 2013, Line Context

9. Developments: Data-driven Descriptors • 2004, PCA-SIFT • 2007, LDE, Learning descriptor[Brown et al.] • 2009, Best DAISY • 2012, D-BRIEF, Learning descriptor by convex optimization[Simonyan et al.], BGM/LBGM, LDAHash • 2013, BinBoost, SQ-SIFT/DAISY

10. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors • Encode information: • Gradient-based Descriptors • Intensity-based Descriptors • Descriptor-based Descriptors

11. Gradient-Based • SIFT、DAISY、BiCE、MROGH、BGM、LBGM、BinBoost、Learning Descriptor[Brown et al., Simonyan et al.] • Intensity-Based • CS-LBP、OSID、BRIEF、ORB、BRISK、FREAK、LDE、D-BREIF、LIOP • Descriptor-Based • LDAHash, LDP[Cai et al.,PAMI’11]

12. Categories of Descriptors • Design method: • Handcrafted Descriptors • Data-driven Descriptors • Encode information: • Gradient-based Descriptors • Intensity-based Descriptors • Descriptor-based Descriptors • Data type: • Floating-point Descriptors • Binary Descriptors

13. Floating-point Descriptors • SIFT、SURF、DAISY、CS-LBP、OSID、MROGH、LIOP、LBGM、LDE… • Binary Descriptors • BiCE、BRIEF、ORB、FREAK、BRISK、BGM、BinBoost、LDAHash、D-BRIEF…

15. Handcrafted Descriptors - SIFT SIFT Descriptor [Lowe’99] • Binning of Spatial Coordinates and Gradient Orientations • Soft Assignment of Binning • 4x4 spatial grids, 8 gradient orientations, 128 dim SIFT • Normalization

16. Handcrafted Descriptors - DAISY DAISY Descriptor [Tola et al.’08] • Log-polar grid arrangement • Gaussian pooling of histograms of gradient orientations • Efficient for dense computation, but not for sparse keypoints!

17. Descriptor Learning – Data Driven Methods Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12] Learning Normalized Patch Low-level feature extraction Smooth Spatial pooling Post process Projection Descriptor

18. Descriptor Learning – Data Driven Methods Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12] • Pre-defined low level features: gradient-based, filter bank based • Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like • Optimized combination of low level feature + spatial pooling • Projection: PCA, LDE … 1st: DAISY-like spatial pooling + filter bank [high Dim] 2nd: DAISY-like spatial pooling + gradient [moderate Dim] PCA is better than LDE for projecting descriptor

19. Descriptor Learning – Data Driven Methods Simonyan et al.’s method [ECCV’12] Learning Normalized Patch Gradient map calculation Smooth Spatial pooling Projection Descriptor • Spatial pooling is constrained to rings • Using L1 regularization to select pooling rings from a large pool • Max-Margin based objective function [convex] • Best reported results in the Brown et al.’s dataset

20. Handcrafted Binary Descriptors Pioneering work: LBP

21. Handcrafted Binary Descriptors BRIEF [ECCV’10, PAMI’12] Construct descriptor by binary tests: Binary tests: Pre-defined positions for binary tests:

22. Handcrafted Binary Descriptors - BRIEF Low memory, Fast to compute and match Limited performance

23. Handcrafted Binary Descriptors FREAK [CVPR’12] Organizing sampling points analogous to retina structure

24. Learning Binary Descriptors D-BRIEF [ECCV’12] • Linear representation of projection matrix by Box/Gaussian/Rect filters • Approximate projection by filter responses • Efficient computation of Box/Gaussian/Rect filter responses • Binarization after discriminative projection • Extremely compact [only 32bits = 4 bytes]

25. Learning Binary Descriptors BGM [NIPS’12] (P1(1), P2(1),c(1)) … (P1(2), P2(2),c(2)) • Explore gradient orientation maps as weak learners • Each bit is construct by one weak learner • Select discriminative gradient orientation maps by boosting (P1(n), P2(n),c(n))

26. Learning Binary Descriptors BinBoost [CVPR’13] • Each bit as a linear combination of many gradient orientation maps • Optimization based on boosting • Very compact [64 bits = 8 bytes]

27. Dataset and Evaluation • Different contexts • Image Matching • Object/Instance Recognition • Image Retrieval

28. Dataset and Evaluation: Matching Oxford dataset [2D scenes]: popular benchmark http://www.robots.ox.ac.uk/~vgg/research/affine/index.html K. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. PAMI’05 …

29. Dataset and Evaluation: Matching Oxford dataset [2D scenes]: popular benchmark Evaluation protocol: recall vs. 1-precision

30. Dataset and Evaluation: Matching Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors http://www.cs.ubc.ca/~mbrown/patchdata/patchdata.html M. Brown, G. Hua and S. Winder,  Discriminant Learning of Local Image Descriptors. PAMI’12 Three different subsets, each of which has more than 400k patch pairs Liberty Yosemite Notre Dame

31. Dataset and Evaluation: Matching Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors Evaluation protocol: False Positive Rate(FPR) vs. Recall

32. Dataset and Evaluation: Recognition • Dataset: Ukbench, ZuBuD, … • Evaluation Protocol: Recognition rate, recall

33. Dataset and Evaluation: Retrieval • Dataset: Oxford/Paris Building, Holidays • Evaluation Protocol: mAP, Precision vs. Recall AP(Average Precision): Precision across all recalls mAP: mean AP of all queries

34. Resources • OpenCV: http://opencv.org/ • SIFT, SURF, BRISK, BRIEF, ORB, FREAK • VLFeat: http://www.vlfeat.org/ • SIFT, LIOP, Covariant Feature Detectors • Oxford VGG: http://www.robots.ox.ac.uk/~vgg/research/affine/index.html • Authors’ pages…

35. Published Evaluations: Matching • K. Mikolajczyk and C. Schmid,  A Performance Evaluation of Local Descriptors. PAMI’05 • P. Moreels and P. Perona,   Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07 • Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11 • O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12 • J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12

36. Published Evaluations: Classification/Recognition • K. Mikolajczyk et al.,  Local Features for Object Class Recognition. ICCV’05 • E. Seemann et al.,   An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05 • M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07 • J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07 • K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10

37. Our Work Feature Description by Intensity Order Pooling Local Intensity Order Pattern Joint work with Zhenhua Wang

38. Feature Description by Intensity Order Pooling

39. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP …

40. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error

41. Match vs. Orientation error 64% 36%

42. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness

43. Category of handcrafted descriptors 0 255/2π r 0 Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminability

44. Category of handcrafted descriptors • Distinctiveness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

45. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

46. Category of handcrafted descriptors With a reference orientation: SIFT, SURF, DAISY, CS-LBP … • Distinctiveness +: encode spatial information, high discriminability -: sensitive to orientation estimation error • Distinctiveness • Robustness • Robustness Without a reference orientation: RIFT, Spin image +: inherently rotation invariance, robust to orientation estimation error -: discard some spatial information, limited discriminablity

47. Our Solution Construct a local coordinate for low-level feature computation Gradient orientation maps [SIFT] Center-symmetrical binary pattern [CS-LBP]

48. Our Solution Pool low-level features by intensity orders … … …… …… …

49. Our Solution Using multiple support regions

50. Using multiple support regions