Constructing immersive virtual space for HAI with photos

1. Constructing immersive virtual space for HAI with photos Shingo Mori YoshimasaOhmoto Toyoaki Nishida Graduate School of Informatics Kyoto University GrC2011 2011/11/09

2. Abstract • We automatically construct immersivevirtual spaces for human agent interaction • Scenes are drawn by outside photo images • Depth maps are reconstructed to express occlusion • Rough 3D models are added for agent • Processing time is about 4.7 days to reconstruct a 20m×20mvirtualspace

3. Introduction • We want to observe HHI using HAI in a virtual space • For example, sightseeing task: • we can select faraway place such as foreign country • we can easily prepare environment to observe • Our Goal: creating system to construct environment to use such a task

4. Introduction • To do sightseeing task and observe interaction, environment should be looked like the real world • virtual spaces should be immersive • outside scenes created by real world photos is needed • spatial relationship between agent and object should be correct • users walk freely on some level • How to construct such a virtual space?

5. Related Work • Model Based Rendering (MBR) • Reconstruct 3D models • Weak at trees or texture-less surfaces • [1-3] are nice methods but, • [1] can’t use outside because of a constraint of axis-aligned surface • [2,3] use high expensive equipments or use a lot of time and effort [1] Furukawa et al. 2010, Reconstructing build-ing interiors from images [2] Pollefeys et al. 2008,Detailed real-time urban 3d reconstruction from video [3] Ikeuchi et al,2004, Bayon digital archival project

6. Related Work • Image Based Rendering (IBR) • draw clearly complex structure such as natural object • Weak at occlusion • [4-5] have good image quality but, • They don’t consider agents • Movable space is restricted [4]Google Street View [5] Ibuki , 2009, Reduction of Unnatural Feeling in Free-viewpoint Rendering Using View-Dependent Deformable 3-D Mesh Model (Japanese)

7. Our Method • To make up immersive environment, we use IBR • because MBR has hole and low resolution for a task • use panorama images and omnidirectional display to show environment

8. Our Method • To collect photo images • divide a space in into a 1-2m grid • shoot about 18 photos in each grid • We use interpolation when move from one shooting point to another obstacle 1-2 meter shooting direction shooting point

9. Our Method • 3D geometry is needed for agents • use Structure from Motion and stereo method in a similar way [1,5] • create depth map for occlusion between objects and agents • This information is used for better IBR • camera position & rotation • 3D position of a point cloud

10. System Pipeline Photos ：Input ：Process System of Constructing Virtual Space ：Output Structure from Motion camera parameter Segmentation segmented image Use previous work Multi view Stereo CMVS Patches Rough 3D model Tackle in this research Creating Depth Map Creating Panorama depth map panorama image panorama depth map Show a Immersive Virtual Space Interpolation interpolated image

11. Structure from Motion (SfM) • Estimate camera parameter (translate matrix) from multi photos • we use Bundler[6] • it is robust and accurate camera position points clout and camera position camera position photos [6]Snavely et al. 2006, Photo tourism: exploring photo collections in 3D

12. Multi view Stereo • Reconstruct 3D geometry • we use CMVS[7] and Poisson Surface Reconstruction[8] • get a point cloud (patches) and rough 3D model patches and rough 3D model photos and translate matrix [7] Furukawaet al. 2010, Towards internet-scale multi-view stereo [8] Kazhdan et al. 2006, Poisson surface reconstruction

13. Create Depth Map • Deal with holes and outliers • Using an assumption that the real world is constructed by a planar surface • assume two points as same planar surface if they are segmented to same area • reconstruct surface from projected patches project patches raw image segmented image depth map

14. Create Panorama Image • To show a scene to an omnidirectional display, we create panorama images • we use Microsft ICE[9] • canonicalizedirection of panorama image from camera rotation [9]MicrosoftCorporation, Microsoft image composite editor http://research.microsoft.com/en-us/um/redmond/groups/ ivm/ice.html. panorama image and depth map

15. Interpolation • To move freely, we create interpolated image between near panorama images • correct move direction and distance about object two raw panorama images about 1-2m away from each other project patches to use as feature point find corresponding point interpolate by morphing(medium point between raw images)

16. Demo

17. Processing time • We experimented with 3 spaces • Most of processing time is SfM • We can drastically improveif we use [10] • Each shooting times are about one hour [10]Agarwal et al.2009, Building rome in a day

18. Conclusion • Conclusion • create a system to automatically construct virtual spaces for HAI • unify various methods to create a system • Future work • expand virtual spaces • research how natural and useful it for HAI