Algebraic Functions of Views for 3D Object Recognition

1. Algebraic Functions of Views for 3D Object Recognition CS773C Advanced Machine Intelligence Applications Spring 2008: Object Recognition

2. Object Appearance • The appearance of an object can have a large range of variation due to: • Photometric effects • Scene clutter • Changes in shape (e.g., non-rigid objects) • Viewpoint changes

3. Algebraic Functions of Views (AFoVs) • A powerful mathematical foundation for investigating variations in the geometrical appearance of an object due to viewpoint changes. • “the variety of of 2D views depicting the geometrical appearance of a 3D object can be expressed as a combination of a small number of 2D views of the object” S. Ullman and R. Basri, "Recognition by Linear Combinations of Models", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 10, pp. 992-1006, 1991.

4. Orthographic Projection • Case of • 3D rigid • transformations • (3 ref. views)

5. Orthographic Projection • Case of 3D linear transformations (2 ref views)

6. More Results … • Perspective projection (2 ref. views, obtained under orthographic projection) • Objects with smooth surfaces and non-rigid objects • More reference views are required. A. Shashua, “Algebraic functions for recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp. 779-789, 1995.

7. A Word of Caution! • Only common features in the reference views can be predicted in a novel view. reference view reference view novel view

8. Recognition Framework Using AFoVs • “novel 2D views of a 3D object can be recognized by matching them to combinations of a small number of known 2D views of the object”

9. Representation and Matching using AFoVs • Representation • Objects are represented by a small number of views. • Each view is represented by some geometric features (e.g., points) • Matching • Predict the geometric appearance of an object in a novel view by combining a small number of reference views of the object.

10. Advantages of the Method • No 3D models or camera calibration are required. • Only a small number of 2D views are required. • Novel views can be different from the stored ones. • Simpler verification scheme. • More general framework (“family” of methods). • Evidence that the human visual system works similarly.

11. Main Challenges • Which model views to combine to predict a novel view? • How to establish the correspondences between novel and reference views? • How to find the coefficients of the combination?. • How to handle occlusions? • How to choose the reference views? Integrate AFoVs with Indexing!

12. Overview of the Method (cont’d)

13. Which Model Groups to Choose? • Cluster geometric features into higher level descriptions. • Consider properties that are unlikely to occur at random. • Property used in our work: convexity

14. Which Model Groups to Choose? (cont’d)

15. How to generate the appearances of a group? • Estimate each parameter’s range of values • Sample the space of parameter values • Generate a new appearance for each sample of values

16. Estimate the Range of Values of the Parameters or and Using SVD: and

17. Estimate the Range of Values of the Parameters (cont’d) • Assume normalized coordinates: • Use Interval Arithmetic(Moore, 1966) • (note that the solutions will be identical)

18. Example

19. Preconditioning the Reference Views • : • Transform the original views to new views effect of the condition number of P on the intervals such that has the best possible condition.

20. Preconditioning the Reference Views (cont’d) • Choosing: • This implies: • Thus:

21. Example (preconditioned views)

22. Decouple Image Coordinates • Same transformation generates the x- and y-coordinates: • Represent only the x-coordinates in the index table. • For each group, store the following entry:

23. Hypothesis Generation and Verification 1.take intersection of hypotheses 2. apply constraints to reject invalid hypotheses model

24. How to Choose the Scene Groups? • Using convex grouping to extract salient scene groups.

25. Implementation Issues • Space requirements • select salient groups • reject groups giving rise to bad conditioned matrices • coarse sampling of parameters • Index computation and table size

26. Important Implementation Issues (cont’d) • Sampling step (i.e., parameters of AFoVs) • Noise tolerance actual: predicted: make additional entries in a neighborhood around the indexed location

27. Experiments and Results model objects and reference views used in our experiments

28. Experiments and Results (cont’d) novel view novel view reference views reference views

29. Experiments and Results (cont’d) novel view novel view reference views

30. Experiments and Results (cont’d) novel view novel view reference views reference views

31. Criticism of the Method • Relies heavily on feature extraction • It has high memory requirements. • The index table might represent unrealistic model appearances. • Indexing based on hashing is not very efficient. • No explicit ranking of hypotheses.

32. Improving AFoVs Recognition Framework • Reject unrealistic appearances • Reduce storage requirements and improve speed • Develop a probabilistic hypothesis generation scheme • Learn shape appearance • Rank hypotheses • Represent object appearance more efficiently using improved indexing schemes and probabilistic models. W. Li, G. Bebis, and N. Bourbakis, "Integrating Algebraic Functions of Views with Indexing and Learning for 3D Object Recognition",IEEE Workshop on Learning in Computer Vision and Patter Recognition (in conjunction with CVPR04), Washington DC, June 28, 2004.

33. Combine Indexing with Learning • Sample the space of appearances sparsely and represent the samples in a K-d tree • Sample the space of views densely and represent the samples using probabilistic models. • Given a novel view: (1) Use K-d tree to retrieve a small number of candidate models (2) For each candidate model, compute the probability that it might have produced the novel view (3) Verify most likely hypotheses first

34. Combine Indexing with Learning (cont’d) • The first stage provides hypothetical matches fast. • The second stage evaluates the feasibility of hypothetical matches fast, without having to apply verification explicitly. • Only “highly likely” hypotheses are verified explicitly.

35. Improved Framework TRAINING PHASE RECOGNITION PHASE Reference views New image Extract image groups Extract model groups Access Using SVD & IA Retrieve Estimate the range of AFoVs parameters K-d Tree Hypothetical matches Sampling AFoVs parameter space Rank hypotheses dense coarse dense Validate views Estimate AFoVs parameters coarse Random Projection Low-dimensional representation Verify hypotheses Manifold learning using EM Recognition results

36. Eliminate Unrealistic Model Appearances • Under the assumption of linear transformations, many unrealistic views could be generated. • Impose rigidity constraints to eliminate them. • Storage requirements can be reduced significantly. • Recognition becomes faster and more efficient.

37. Eliminate Unrealistic Model Appearances Unrealistic Views (without constraints) Realistic Views (with constraints)

38. Indexing Appearances • Sample the space of views “coarsely” and represent the samples in an index table. • Hashing might not very well in this case ... • Need an improved indexing scheme.

39. Range Search vs Nearest Neighbor Search • Range search is not appropriate when storing a sparse number of views. • K-d trees perform a nearest-neighbor search. Nearest Neighbor Search Range Search

40. P4 P3 P4 P1 P2 P3 P2 P1 K-d Trees for Indexing • K-d trees perform a nearest-neighbor search.

41. Learning Geometric Appearance • We can pre-compute the views that an object can produce off-line. • These views form a manifold in lower dimensional space. • Model object appearance using a pdf. • Sample the space of appearances. • Fit a parametric model (e.g., mixtures of Gaussians using EM). • Use mutual information theory to choose the number of components. • EM has problems when the dimensionality of the data is high. • Apply “Random Projection” first, then run EM algorithm.

42. Manifolds of Real Objects: An Example • Need to store a small number of parameters only for each model

43. Hypothesis Ranking • Each hypothesis generated by the K-d tree is ranked by computing its probability using mixture models. • For each test group, we compute two probabilities, one from x coordinates, and the other from y coordinates. • The overall probability for a particular hypothesis is computed according to the following equation: where

44. Reference Views 1st Reference view 2nd Reference view

45. Reference Views (cont’d) 1st Reference view 2nd Reference view

46. Test Views (a) (b) (c) (d) (f) (e)

47. Test Views (cont’d) Hypothesis rejected Hypothesis rejected

48. Integrate Geometric Appearance with Intensity Appearance • Using geometrical information only does not provide enough discrimination for objects having similar “geometric” appearance but probably different “intensity” appearance. • Integrating geometric and intensity apperance during hypothesis verification to improve discrimination power and robustness. W. Li, G. Bebis, and N. Bourbakis, "3D Object Recognition Using 2D Views", IEEE Transactions on Image Processing (under revision).

49. Dense Correspondences • For each group of corresponding points, apply triangulation recursively to get denser correspondences. • Divide triangles into four sub-triangles by considering the middle point of each side of each triangle.

50. Refine AFoVs parameters (before refinement) (after refinement)