Efficient Algorithms for Matching

1. Efficient Algorithms for Matching Pedro Felzenszwalb Trevor Darrell Yann LeCun Alex Berg

2. Efficient Algorithms for Matching Polynomial & exact Pedro Felzenszwalb Trevor Darrell Yann LeCun Alex Berg

3. Efficient Algorithms for Matching Polynomial & exact Pedro Felzenszwalb Trevor Darrell Yann LeCun Alex Berg Multilinear & approximate

4. Efficient Algorithms for Matching Polynomial & exact Pedro Felzenszwalb Trevor Darrell Yann LeCun Alex Berg Multilinear & approximate “Fast?” but very good

5. Efficient Algorithms for Matching Polynomial & exact Pedro Felzenszwalb Trevor Darrell Yann LeCun Alex Berg Multilinear & approximate “Fast?” but very good Happy when things work

6. First Criticism • Efficiently computing the wrong solution is not so useful… First Response • Even if say, an algorithm does not solve object recognition, it can still be a useful tool…

7. Why Matching? • Ideas hatched before me • Statistical Pattern Theory (Ulf Grenander) • Deformable Templates • Fischler & Elshlager • Etc. at least by the early 1970’s • “transform” and “appearance” parameters • Matching to estimate transform Used to be continuous, now often discrete Very general, Translation / Diffeomorphism / Assignment Image / Features / “Parts” / etc.

8. Why Matching? • Ideas hatched before me • Statistical Pattern Theory (Ulf Grenander) • Deformable Templates • Fischler & Elschlager • Etc. at least by the early 1970’s • “transform” and “appearance” parameters • Matching to estimate transform TRANSFORM MODEL IMAGE Used to be continuous, now often discrete Very general, Translation / Diffeomorphism / Assignment Image / Features / “Parts” / etc.

9. Why Matching? • Ideas hatched before me • Statistical Pattern Theory (Ulf Grenander) • Deformable Templates • Fischler & Elschlager • Etc. at least by the early 1970’s • “transform” and “appearance” parameters • Matching to estimate transform TRANSFORM MODEL IMAGE Used to be continuous, now often discrete Very general, Translation / Diffeomorphism / Assignment Image / Features / “Parts” / etc.

10. Why Matching? • Ideas hatched before me • Statistical Pattern Theory (Ulf Grenander) • Deformable Templates • Fischler & Elschlager • Etc. at least by the early 1970’s • “transform” and “appearance” parameters • Matching to estimate transform • Searching over diffeomorphisms difficult • Searching over discrete assignments easier? TRANSFORM MODEL IMAGE Used to be continuous, now often discrete Very general, Translation / Diffeomorphism / Assignment Image / Features / “Parts” / etc.

11. Search for a Transformation Image Model of Car ?

12. Find Transformation Using Correspondence Image Model of Car Sometimes… • Search through a discrete set of possible point correspondences • Objective function should be close to cost of the original model • Use the discrete correspondences to obtain a continuous transformation if needed

13. Find Transformation Using Correspondence Image Model of Car Why it works… Sometimes we can measure consistency of model appearance locally Inspired by branch and bound: “If local appearance is inconsistent, any alignment with that appearance is bad.” My preferred way of motivating local features…

14. Find Transformation Using Correspondence Image Model of Car Sometimes local appearance is not enough, so we model some version of spatial constraints. Do not make the problem harder than it was…

15. Linear Assignmente.g. Hungarian AlgorithmJust Features, no Geometry Individual feature matches provide most of the solution.

16. Quadratic Assignment(Adding Geometric Constraints) Individual feature matches provide most of the solution. Geometric consistency only has to clean things up a little. In this case we formulate the matching as an Integer Quadratic Programming problem and look for an approximate solution…

17. Second Meta-Comment • Even if a problem can have very difficult instances, the effective complexity of certain instances might be quite low. This can be quite difficult to verify formally.

18. Back to the alignment… Use Alignment to Compare Model of Car • Given alignment evaluate the model • Note: we might have been done already • Grauman et al, Zhang et al • Actually do some alignment and check • the quality of the fit.

19. That’s all fine, but Humans can be very efficient • Simon Thorpe: animal or not in <=150 ms • “Feed forward” process • Difficult to retrain (familiarization does not make a difference) • Salient parts of images are actually processed more rapidly. • Support for some styles of current algorithms • Neurophysiological evidence for some mid-level vision (illusory contours, figure ground, etc.) von der Heydt et al

20. What is an Object? • Apple yes, mist no • A rule of thumb is that objects have some definite spatial support… • Image/Scene? Some context models treat scenes or images as objects • Face, eyes, nose, eye-lashes • We can build SPT models for all of these…

21. Heuristics • Take advantage of the data • Sometimes a single feature is enough • For efficiency need to weigh this against how often that feature is found • Many (?) object recognition datasets allow easy discrimination between categories with only very simple features extracted from the whole image, eg Pascal and Caltech 101. • Segmentation or Figure/Ground -- might as well see if there is an object there before trying to recognize it…

22. An Approach • Extract features from an image • Look features up in a large database • Approximate Nearest Neighbor algorithms can make this sub-linear. • Each entry tells us about which hypotheses [Object,position,pose,...] might be present. • Use a “short list” of these to check in more detail using a matching framework • Simple matching can actually be indexed. Indyk et al, Grauman et al. • Finally use a matching to align models and apply more expensive processing