Extracting Objects from Range and Radiance Images

1. Extracting Objects from Range and Radiance Images Computer Science Division University of California at Berkeley Yizhou Yu Andras Ferencz Jitendra Malik

2. Image-based Modeling and Rendering Recover Models of Real World Scenes and Make Possible Various Visual Interactions • Vary viewpoint • Vary lighting • Vary scene configuration

3. Image-based Modeling and Rendering • 1st Generation---- vary viewpoint but not lighting • Recover geometry ( explicit or implicit ) • Acquire photographs • Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, Concentric Mosaics, Light Field Rendering etc.

4. Image-based Modeling and Rendering • 2nd Generation---- vary viewpoint and lighting • Recover geometry & reflectance properties • Render using light transport simulation or local shading • [ Yu, Debevec, Malik & Hawkins 99 ], [ Yu & Malik 98 ], [ Sato, Wheeler & Ikeuchi 97 ] Original Lighting & Viewpoint Novel Lighting & Viewpoint

5. Image-based Modeling and Rendering • 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting Novel Viewpoint Novel Viewpoint & Configuration

6. Input Multiple range scans of a scene Multiple photographs of the same scene Output Geometric meshes of each object in the scene Registered texture maps for objects Our Framework

7. Overview & Video Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

8. Outline • Overview • Range Data Segmentation • Radiance Image Registration • Meshing and Texture-Mapping • Results • Future Work

9. Overview Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

10. Range Image Registration • Registration with Calibration Objects • Cyra Technologies, Inc. • Registration without Calibration Objects • [ Besl & McKay 92 ], [ Pulli 99 ]

11. Overview Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

12. Previous Work on Range Image Segmentation • Local region growing with surface primitives • [ Hoffman & Jain 87 ], [ Besl & Jain 88 ], [ Newman Flynn & Jain 93 ], [ Leonardis, Gupta & Bajcsy 95 ], [ Hoover et. al. 96 ] • They don’t address general free-form shapes and albedo variations. • Local region growing is suboptimal in finding object boundaries.

13. Normalized Cut Framework [ Shi & Malik 97 ], [ Malik, Belongie, Shi & Leung 99 ] Recursive Binary Graph Partition by Minimizing Approximate Solution

14. From 2D Image To 3D Point Cloud • Neighborhood • 3D spherical region • Complexity • E.g. nineteen 800 by 800 scans • Clustering

15. Point Cloud Segmentation: Cues Normal Orientation Returned Laser Intensity Proximity

16. Point Cloud Segmentation: Graph Setup Nodes: Clusters Edges: Local + Random Long-range Connections Weights:

17. Point Cloud Segmentation: Criterion • Use Normalized Cut Criterion to Propose Cut • Use Normalized Weighted Average Cut to Accept Cut

18. Point Cloud Segmentation: Algorithm • Coarse Segmentation • Clustering • Cluster Segmentation • Recursive segmentation based on normal continuity and proximity • Recursive segmentation based on continuity in laser intensity and proximity • Fine Segmentation • re-clustering and segmentation on each group from coarse segmentation

19. Segmentation Results

20. Overview Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

21. Previous Work on Pose Estimation • Mathematical Background • from 3 or more points [Hung et.al. ‘85],[Haralick et.al. ‘89], Overview [Haralick et.al. ‘94] • from points, lines and ellipse-circle pairs [Qiang et.al. ‘99] • Automatic Techniques • Combinatorial search on automatically detected features [Huttenlocher and Ulman ‘90],[Wunsch and Hirzinger ‘96] • Hybrid approach (user initialization with finding silhouettes) [Neugebauer and Klein ‘99]

22. Radiance Image Registration • Requirements • Automatic: many (50 - 200) images need to be registered • Accurate: Registration must be accurate to within one or two pixels for texture mapping • General purpose: scene may be complicated, possibly without easily and uniquely identifiable features • Solution • Place registration targets in scene

23. Finding Targets in Images • Pattern matching followed by ellipse fitting

24. Combinatorial Search • Brute force • 4 correspondences each image, O(n4) time n targets • Alternative • Use fitted ellipse parameters in addition to position to estimate pose from 2 target matches, O(n2) time • 20 seconds for scene with 100 targets 3D Targets

25. Registration Summary • Place registration targets in scene before acquiring data • Automatically detect targets in data • Perform combinatorial search to match image targets to corresponding target in geometry • Find camera pose from matched points • Remove targets from images and fill in holes

26. Camera Pose Results • Accuracy: consistently within 2 pixels • Correctness: correct pose for 58 out of 62 images

27. Overview Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

28. Mesh Reconstruction and Simplification • Meshing • Use the “crust” algorithm, [ Amenta, Bern & Kamvysselis 98 ], for coarse geometry • Use nearest-neighbor connections for detailed geometry • Possible to use volumetric techniques, [ Curless & Levoy 96 ], to merge meshes • Simplification • Use quadric error metric, [ Garland & Heckbert 97 ]

29. Reconstructed Mesh with Camera Poses and Calibration Targets

30. Models of Individual Objects

31. Overview Range Images Point Cloud Point Groups Simplified Meshes Meshes Registration Segmentation Reconstruction Pose Estimation Texture Map Synthesis Radiance Images Calibrated Images Texture Maps Objects

32. Texture Map Synthesis • Regular Texture-Mapping with Texture Coordinates • Compose a triangular texture patch for each triangle. • Allocate space from texture maps for each texture patch to assign texture coordinates. • The size of each texture patch is determined by the amount of variations in photographs. • The derivative of the Gaussian is used as a metric for variations. Photograph 3D Triangle Texture Map

33. Texture-Mapping and Object Manipulation

34. Video

35. Future Work • Create watertight geometric models from sparse or incomplete data • Improve mesh simplification techniques • Texture Compression

36. Acknowledgments • Thanks to Ben Kacyra, Mark Wheeler, Daniel Chudak, Jonathan Kung at Cyra Technologies, Inc., and Jianbo Shi at CMU. • Supported by ONR BMDO, the California MICRO program, and Microsoft Graduate Fellowship.