Segmentation Based Environment Modeling Using a Single Image

1. Segmentation Based Environment ModelingUsing a Single Image SeungTaek Ryoo Department of Software, HanShin University, Korea stryoo@hs.ac.kr 2004. 9. 29

2. 3D Reconstruction Planar Objects Single View Metrology (Criminisi1999) Piecewise Planar Objects from Single Images (Sturm1999) Automatic 3D Modeling of Architecture (Dick2000) Image Based Modeling and Photo-Editing (Oh2001) Previous Works Criminis1999 Sturm1999 Oh2001 Dick2000

3. Input Image (Regular Image, OCC Map) Environment Modeling Segmented Image Environment Model Image Segmentation Depth Calculation Environment Mapping Depth Image Desired Image Objective • Objective • Realistic & Real-time View Rendering with Full-View Panorama • Image-Segmentation Based Environment Modeling The Structure of Segmentation Based Modeling

4. Image Segmentation • Segmentation • indoor scene consists of floor and ceiling. • outdoor scene consists of ground and sky. • the environment map is divided into • a floor (ground), ceiling (sky), and surrounding objects.

5. Depth Calculation (1/4) • Assumption • View direction of the camera must be parallel to the ground • The floor (ground) and the ceiling must consist of a plane that is parallel to the XZ plane in the world coordinate. • The sky must consist of a hemi-sphere with an infinite radius • The surrounding objects must be perpendicular to the ground Image plane o WC m Z R

6. Depth Calculation (2/4) • Extracting the depth value from Regular Image • The depth value can be obtained by testing for the intersection points between the floor and the ray, and between the Image plane and the ray • The equation can be obtained with the law of the triangle

7. R R Y = d Y = d Z d d r b b r Z XZ Cylinder XY Cylinder Depth Calculation (3/4) • Extracting the depth value from OCC map • depth value of the ceiling (floor) by testing the relationship b/w XZ cylinder and ray, and b/w XY cylinder and ray

8. o U V Depth Calculation (4/4) • How to get the depth value of the surrounding objects • Set the depth value of the floor(ceiling) close to the surrounding objects. • Some other assumptions should be made. • the relationship between the surrounding objects. • the relationship between the floor(ceiling) and the surrounding objects. • the relationship between the image plane and the surrounding objects. Color Image Segmented Image Depth Image

9. Triangulation Feature Points Environment Modeling Rectangular Grid

10. Texture mapping & Point sample rendering Texture Mapping Point Sample Rendering Wireframe • Texture mapping : the stretching effect of the 3D model occurs by the occlusion • Point Sample Rendering : holes occurs by the lack of the information

11. Regular Image Regular Image Desired Image Segmentation Environment Mapping Depth Calculation Environment Modeling

12. Regular Image Gyeongbokgung (Royal Palaces) Seokguram (Stone Cave)

13. Environment Map (Octahedral Map) Orthogonal Cross Cylinder Segmentation Environment Mapping Depth Calculation Environment Modeling OCC Map (Indoor Scene)

14. Orthogonal Cross Cylinder Segmentation Segmentation Based Environment Model Depth Calculation Desired Image OCC Map (Outdoor Scene)

15. Result (Art Gallery) Environment Map (sidestep) Segmented Based Environment Model (sidestep)

16. Conclusion & Future Works • Conclusion • can be used for environment modeling by extracting the depth value by the image segmentation • navigate through the virtual environment when the viewpoint has been changed • easy to implement on the environment map • Future Works • Multiple Resolution • the problem of the blurring-image effect caused by the fixed resolution • Multiple Images (Registration & Integration) • the problem of the stretching effect of the 3D model caused by the occlusion • Generalization to apply any type of images • only images whose viewpoint and the floor are perpendicular may be used.