Automatic 3D modeling from range images

1. Automatic 3D modeling from range images Daniel Huber Carnegie Mellon University Robotics Institute

2. Outline • What is 3D modeling and why automate it? • Proposed approach • Automatic modeling algorithms • Basic modeling operations • Automatic modeling example

3. 6 2 8 1 15 What is 3D modeling? conversionto meshes pair-wise registration multi-viewregistration original object Applications • Reverse engineering • Multimedia content creation • Preservation of art and architecture • Terrain and environment modeling • Model-based vision systems integration final model

4. 3D modeling issues and current solutions Registration issues • Which views overlap? • What are the relative poses? • What are the absolute poses? Mechanical solutions • Measure absolute poses • Measure relative poses Manual solutions • Mark correspondences • Hand register • Specify overlapping views and verify results

5. Example application – handheld modeling • User holds object to be modeled • Range images are captured from various viewpoints • Software determines pose relations and outputs the 3D model

6. Benefits of automatic modeling • Enables new applications • Handheld modeling • Large-scale environment modeling • Saves effort • Simplifies data collection • Eliminates tedious hand registration • Provides an alternative to mechanical solutions • Less cost & complexity • Fewer limits on domain

7. Some existing modeling systems Modeling from range images • Mechanical – Rioux (interiors), Miller (terrain), Levoy, Wheeler (objects) • Manual – Neugebauer, Ikeuchi, Pulli (statues), Johnson (objects and interiors) Modeling from intensity images • Video streams – Tomasi & Kanade, Zisserman (man-made objects) • Photographs – Debevec, Zisserman (buildings), Sullivan & Ponce, Kutulakos & Seitz (objects)

8. The automatic modeling problem • Don’t assume: • Relative poses are known • Overlapping views are known • Structured scene – planes, corners, etc. • Do assume: • Views will overlap • Views may be noisy or spurious • Static scene Given a set of range images of an unstructured scene, automatically construct a consistent 3D model.

9. Overview of automatic modeling approach • Select view pairs that are likely to register correctly • Perform pair-wise registration and locally verify • Robustly combine matches in a network of views (model graph) • Infer new links based on topology • Verify global consistency • Improve quality by simultaneously registering all views

10. Key difficulties • Don’t know what the object looks like • Don’t know approximate poses or even which views overlap • Pair-wise registration fails sometimes: • Incorrect matches • Missed matches • Multiple matches (best match may be wrong) • Pair-wise registration takes time – need to choose views to register • Local verification cannot always detect errors – use global verification

11. pair-wise registration gives two equally good matches 5 views of a room 1 3 1 4 5 2 4 5 4 5 4 1 2 5 2 3 4 1 5 2 3 2 1 the resulting model hypotheses overhead view global consistency checks resolve the ambiguity Example of a locally unresolvable ambiguity

12. Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Topological inference • Conflict detection • Multi-view registration 2 5 7 1 15 7 5 15 2 2 - 5 1 - 15 2 - 7 2 - 5 Working set • Pair-wise registration results • Inferred matches Inputviews Modelhypotheses Proposed automatic modeling framework selectedviews inferred matches registrationresults selectedmatches outputmodel

13. 10 12 11 2 1 3 6 4 7 9 5 8 The model graph Encode model topology • Nodes are views • Arcs are transforms between overlapping views • Spanning tree is the minimal complete model Perform model construction • Start with no arcs • Basic operations add, modify, or delete arcs • Encode a priori information as initial arcs T1,2 correct matches incorrect matches

14. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

15. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

16. Automatic modeling algorithms Model construction difficulties • One bad match corrupts the model • Noisy data and spurious views • Incomplete information – missed or untried matches Solutions • Combinatorial optimization – deterministic and non-deterministic • Study simpler problem first • Exhaustive registration – try all pairs up front • Selective registration – enable view selection

17. Exhaustive registration • Sequential removal – start with all matches and remove • Sequential addition – start with seed view and add • Merging • Repeatedly merge partial models • Track multiple hypotheses • Non-deterministic methods • RANSAC – spanning tree + verification • MCMC or annealing – search for maximum likelihood model

18. Selective registration View selection • Use order of data collection • Cluster similarly shaped views and register in clusters • Sort views by shape similarity and register in order • Choose self-constraining or unique views • Modify exhaustive methods to include view selection

19. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

20. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

21. Uninformed pair-wise registration • Align two views with no initial estimate • Based on Johnson’s surface matching engine • Uses local shape signatures – spin images • Searches for similarly shaped regions • Fails dramatically sometimes

22. slow sliding fast sliding point-pointcorrespondences point-planecorrespondences Pair-wise registration refinement • Align two views given a close initial estimate • Iterative closest point algorithm (Besl & McKay, Zhang) • Repeatedly minimize distance between closest points • Incorrect correspondences slow convergence • Minimize point to tangent plane distance instead (Chen & Medioni)

23. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

24. Views 1 and 2: 48%, 2.1 mm Views 1 and 9: 15%, 3.1 mm Local verification – overlap distance • Registration requires significant overlap for success (~ 30%) • Correctly registered views should be close wherever they overlap • Verify matches based on overlap amount and distance • Use as a classifier or as a measure of match quality

25. S1 S2 surfaces are close S2 blocks S1 S1 is not observed S1 S2 S1 S2 C1 C2 free spaceviolation C1 C2 C1 C2 occupied spaceviolation consistentsurfaces Local verification – visibility consistency • Consider expected observations from viewpoint of sensor C1 • Consistent – S2 close to S1 wherever both observed • Free space violation – S2 blocks visibility of S1 • Occupied space violation – S1 unobserved, even though it should be (sensor model required)

26. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

27. Global verification Topological inference (positive evidence) • Coarse test – bounding box intersection • Fine test – local verification procedures • Pair-wise register if possible/necessary Conflict detection (negative evidence) • Coarse test – frustum or view volume intersection • Fine test – visibility consistency

28. Automatic modeling framework • Automatic modeling algorithms Basic modeling operations Surface matching process • View selection • Pair-wise registration • Local verification Model construction process • Global verification • Topological inference • Conflict detection • Multi-view registration

29. T45 T34 top view T56 T23 4 T61 3 T12 5 6 2 1 Close-up of tail - the red, green and blue surfaces should be aligned with the cyan one 1 Motivation for multi-view registration • Pair-wise registration gives relative pose with small error • Errors accumulate with compounding operation, leading to gaps or seams on the final model • Multi-view registration distributes the accumulated error in a principled way T12 * T23 * T34 * T45 * T56 * T61 I

30. Multi-view registration • Simultaneously align two or more surfaces • Multi-view ICP • Minimize squared distance between corresponding points (Benjemaa & Schmitt) • Requires initial pair-wise estimates • Suffers from same “sliding” problem as ICP • Minimize point-plane distances, instead (Neugebauer)

31. pair-wiseregistrationonly point-planecorrespondences point-pointcorrespondences Comparison of multi-view registration methods Synchronized cross-section

32. 14 15 13 10 9 3 6 6 12 11 top view 5 5 1 8 8 10 9 13 11 15 2 3 2 4 4 7 7 1 12 14 horizontal slice Automatically modeling the angel initial model(bad matches in red) connectivity afterlocal verification

33. 4 4 6 6 7 7 5 5 8 8 10 10 9 9 13 13 11 11 15 15 3 3 2 2 1 1 12 12 14 14 Inferring new pose relations 30% overlap 1% overlap

34. top view horizontal slice The final angel model

35. Research schedule

36. Conclusion • Expected contributions • Framework for automatic modeling – model graph, basic operations • Algorithms for exhaustive and selective modeling • Demonstration in several domains, including handheld modeling application • See the proposal for • Range image to mesh conversion • Registration uncertainty measure • Planned experiments • Details of related work