Shape Matching and Object Recognition using Low Distortion Correspondence

1. Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley

2. Object Category Recognition We address the problem of recognizing object categories as opposed to individual objects. For instance in the upper right we see images of airplanes. We would like to recognize that there is an airplane in an image without having seen that specific airplane before, or on the lower left find a cell phone in the image without having seen that particular cell phone before. So, how are we going to do this?... wide variety of object categories large intraclass variation small number of training examples recognize object not image shape, not color etc. We address the problem of recognizing object categories as opposed to individual objects. For instance in the upper right we see images of airplanes. We would like to recognize that there is an airplane in an image without having seen that specific airplane before, or on the lower left find a cell phone in the image without having seen that particular cell phone before. So, how are we going to do this?... wide variety of object categories large intraclass variation small number of training examples recognize object not image shape, not color etc.

3. Deformable Template Matching with Exemplars for Recognition Use exemplars as deformable templates Find a correspondence between the query image and each template Our strategy is based on deformable template matching. We have a database of templates, and each template is matched to a query image. The best matching template is used to identify the object in the query image. Our strategy is based on deformable template matching. We have a database of templates, and each template is matched to a query image. The best matching template is used to identify the object in the query image.

4. Deformable Template Matching with Exemplars for Recognition Use exemplars as deformable templates Find a correspondence between the query image and each template Here for instance the best matching template was of a helicopter. Note that we get more than just the type of object, we have a correspondence between the template and the query. One consequence of this is shown in light red in the image above, the spatial support of the template has been mapped onto the query using the correspondence, so we can see where the object is located. The main subroutine here is correspondenceHere for instance the best matching template was of a helicopter. Note that we get more than just the type of object, we have a correspondence between the template and the query. One consequence of this is shown in light red in the image above, the spatial support of the template has been mapped onto the query using the correspondence, so we can see where the object is located. The main subroutine here is correspondence

5. Correspondence for Deformable Template Matching Evaluate correspondence based on: Similarity of appearance near feature points Similarity in configuration of the feature points % The idea of deformable templates is not new. In % the early 1970's at least three groups were % working on similar ideas: % Grenander in statistical pattern theory % von der Malsburg in neural networks % and Fischler and Eschlager in computer vision On the left we have the model template of a helicopter. We characterize the template by a set of feature points sampled from edges in the image. These are shown as the approximately 40 colored dots in the left image. On the right we show a query image. Here feature points are extracted along edges in the entire image. Many of these will be on background or clutter, but some are on the object of interest. Our problem is to find a correspondence between the model points at left and the correct subset of feature points on the right.% The idea of deformable templates is not new. In % the early 1970's at least three groups were % working on similar ideas: % Grenander in statistical pattern theory % von der Malsburg in neural networks % and Fischler and Eschlager in computer vision On the left we have the model template of a helicopter. We characterize the template by a set of feature points sampled from edges in the image. These are shown as the approximately 40 colored dots in the left image. On the right we show a query image. Here feature points are extracted along edges in the entire image. Many of these will be on background or clutter, but some are on the object of interest. Our problem is to find a correspondence between the model points at left and the correct subset of feature points on the right.

6. Correspondence for Deformable Template Matching Evaluate correspondence based on: Similarity of appearance near feature points Similarity in configuration of the feature points A good correspondence should result in similar appearance of corresponding points, and should preserve the configuration of the model points. For instance the point on the tail of the model template at left maps to the tail of the query at right. The local appearance is roughly similar. And similarly for the point on the middle of he helicopters A good correspondence should result in similar appearance of corresponding points, and should preserve the configuration of the model points. For instance the point on the tail of the model template at left maps to the tail of the query at right. The local appearance is roughly similar. And similarly for the point on the middle of he helicopters

7. Correspondence for Deformable Template Matching Evaluate correspondence based on: Similarity of appearance near feature points Similarity in configuration of the feature points The yellow circles give an approximate idea of the scale of the descriptors we use to compare local appearance. The yellow circles give an approximate idea of the scale of the descriptors we use to compare local appearance.

8. Correspondence for Deformable Template Matching Evaluate correspondence based on: Similarity of appearance near feature points Similarity in configuration of the feature points We also want to find a correspondence that maintains the relative position of the model points. Although the shapes of the helicopters are indeed different, hence the need for deformable template maching, the relative position of the two model points is similar to the relative position of the points they correspond to. A good correspondence for all forty of these model points might look something like: We also want to find a correspondence that maintains the relative position of the model points. Although the shapes of the helicopters are indeed different, hence the need for deformable template maching, the relative position of the two model points is similar to the relative position of the points they correspond to. A good correspondence for all forty of these model points might look something like:

9. Correspondence Result this, corresponding points are the same color. note the correct matching for the tail point. Things are a little easier to see when we use thin plate splines to interpolate the mapping for a denser set of pointsthis, corresponding points are the same color. note the correct matching for the tail point. Things are a little easier to see when we use thin plate splines to interpolate the mapping for a denser set of points

10. Interpolated CorrespondenceUsing Thin Plate Splines note that the correspondence is roughly correct, the tail is shortened the main body is shorter and thicker, the landing gear is not well matched, nor is the main rotor, of course it is more or less invisible in both images... now let's get into more detail about how we approac the correspondence problem. note that the correspondence is roughly correct, the tail is shortened the main body is shorter and thicker, the landing gear is not well matched, nor is the main rotor, of course it is more or less invisible in both images... now let's get into more detail about how we approac the correspondence problem.

11. Correspondence for Deformable Template Matching

12. Geometric Blur(Local Appearance Descriptor)

13. Geometric Blur(Local Appearance Similarity)

14. Are Features Enough?

15. Measuring Distortion(Similarity in Configuration)

16. Cost Function as IQP

17. Optimization Integer Quadratic Programming is NP hard The instances we generate seem easy Using a linear bound to initialize gradient descent provides good results (In fact better than the guarantee of Goemans & Williamson's randomized algorithm) Varying the linear constraints on x allows one-one, one-many, or fixed number of outliers, etc.

18. Correspondence Result

19. Interpolated CorrespondenceUsing Thin Plate Splines

20. Quadratic Assignment(Using IQP)

21. Linear Assignment(e.g. Hungarian)

22. Correspondence Examples (Shape Matching)

23. Correspondence Examples(Shape Matching)

24. Correspondence Examples(Shape Matching)

25. Correspondence Examples(Shape Matching)

26. Correspondence Examples(Shape Matching)

27. Correspondence Examples(Shape Matching)

28. Correspondence Examples(Shape Matching)

29. Correspondence Examples(Shape Matching)

30. Correspondence Examples(Shape Matching)

31. Correspondence Examples(Shape Matching)

32. Correspondence Examples(Shape Matching)

33. Application to RecognitionCaltech 101 101 classes of objects + “background” Large Scale Roughly aligned Large intra-class variation Fei-Fei, Fergus, Perona '04

34. Caltech 101 Recognition Results

35. Model Building for Segmentation Here is an example of learning a model. Of course it will only be to slightly modify the exemplar, but it is a step in the right directions...Here is an example of learning a model. Of course it will only be to slightly modify the exemplar, but it is a step in the right directions...

36. Automatic vs Hand Segmentation

37. Application to RecognitionFaces Face dataset from Berg et al '03 Medium to large scale faces AP News photos 20 face exemplars Same methodology as Caltech 101, but multiple objects / image After one face is identified its features are removed and the search continues Compared to a detector from Mikolajczyk based on Schneiderman & Kanade, that is quite successful on this dataset

38. Application to RecognitionFaces

39. Conclusion Use rich descriptors that are insensitive to typical transformations Geometric Blur Enforce relationship constraints among corresponding features Integer Quadratic Programming Estimate smooth transform Thin Plate Splines

40. Thank You