CSCE 641 Computer Graphics: Image-based Modeling

1. CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai

2. Image-based modeling • Estimating 3D structure • Estimating motion, e.g., camera motion • Estimating lighting • Estimating surface model

3. Traditional modeling and rendering Geometry Reflectance Light source Camera model rendering modeling User inputTexture map survey data Images For photorealism: - Modeling is hard - Rendering is slow

4. Can we model and render this? What do we want to do for this model?

5. Image based modeling and rendering Image-based modeling Image-based rendering Imagesuser input range scans Model Images

6. Spectrum of IBMR Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

7. Spectrum of IBMR Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

8. Spectrum of IBMR Model Panoroma Image-based rendering Image based modeling Images + Depth Geometry+ Images Camera + geometry Imagesuser input range scans Images Geometry+ Materials Light field Kinematics Dynamics Etc.

9. Stereo reconstruction • Given two or more images of the same scene or object, compute a representation of its shape • What are some possible applications? known camera viewpoints

10. 3D modeling • From one stereo pair to a 3D head model • [Frederic Deverney, INRIA]

11. 3D modeling The Digital Michelangelo Project, Levoy et al.

12. Optical mocap Vicon mocap system

13. Z-keying: mix live and synthetic • Takeo Kanade, CMU (Stereo Machine)

14. Virtualized RealityTM • [Takeo Kanade et al., CMU] • collect video from 50+ stream • reconstruct 3D model sequences • steerable version used forSuperBowl XXV “eye vision” • http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html

15. View interpolation • input depth image novel view • [Szeliski & Kang ‘95]

16. View morphing • Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96]

17. Image warping

18. Video view interpolation

19. Performance Interface • Microsoft Natal project

20. Additional applications? • Real-time people tracking (systems from Pt. Gray Research and SRI) • “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93] • Other ideas?

21. Stereo matching • Given two or more images of the same scene or object, compute a representation of its shape • What are some possible representations for shapes? • depth maps • volumetric models • 3D surface models • planar (or offset) layers

22. Outline • Stereo matching • - Traditional stereo • - Multi-baseline stereo • - Active stereo • Volumetric stereo • - Visual hull • - Voxel coloring • - Space carving

23. Papers • Stereo matching • Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363. • D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002. • Visual-hull reconstruction • Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32. • Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”, Proc. SIGGRAPH 2000, pp. 369-374. • Photo-hull reconstruction • Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl. Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173. • Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of Computer Vision, 2000, 38(3), pp. 199-218.

24. Stereo scene point image plane optical center

25. Stereo • Basic Principle: Triangulation • Gives reconstruction as intersection of two rays • Requires • calibration • point correspondence

26. Camera calibration • From world coordinate to image coordinate Perspective projection View transformation Viewport projection u sx a u0 v 0 -sy v0 1 0 0 1 2D projections 3D points Camera parameters

27. epipolar plane epipolar line Stereo correspondence • Determine Pixel Correspondence • Pairs of points that correspond to same scene point epipolar line • Epipolar Constraint • Reduces correspondence problem to 1D search along conjugateepipolar lines • Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

28. Stereo image rectification

29. Stereo image rectification • reproject image planes onto a common • plane parallel to the line between optical centers • pixel motion is horizontal after this transformation • two homographies (3x3 transform), one for each input image reprojection • C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

30. Rectification Original image pairs Rectified image pairs

31. Stereo matching algorithms • Match Pixels in Conjugate Epipolar Lines • Assume brightness constancy • This is a tough problem • Numerous approaches • A good survey and evaluation: http://www.middlebury.edu/stereo/

32. For each epipolar line For each pixel in the left image • Improvement: match windows • This should look familiar.. • Can use Lukas-Kanade or discrete search (latter more common) Your basic stereo algorithm • compare with every pixel on same epipolar line in right image • pick pixel with minimum matching cost

33. W = 3 W = 20 Window size • Smaller window - • Larger window - • Effect of window size

34. Stereo results • Data from University of Tsukuba • Similar results on other images without ground truth Scene Ground truth

35. Results with window search Window-based matching (best window size) Ground truth

36. Better methods exist... • State of the art method • Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, • International Conference on Computer Vision, September 1999. Ground truth

37. Stereo reconstruction pipeline • Steps • Calibrate cameras • Rectify images • Compute disparity • Estimate depth

38. Stereo reconstruction pipeline • Steps • Calibrate cameras • Rectify images • Compute disparity • Estimate depth • Camera calibration errors • Poor image resolution • Occlusions • Violations of brightness constancy (specular reflections) • Large motions • Low-contrast image regions • What will cause errors?

39. Outline • Stereo matching • - Traditional stereo • - Multi-baseline stereo • - Active stereo • Volumetric stereo • - Visual hull • - Voxel coloring • - Space carving

40. disparity map 3D rendering [Szeliski & Kang ‘95] Depth from disparity input image (1 of 2) X z x x’ f f baseline C C’

41. Choosing the stereo baseline • What’s the optimal baseline? • Too small: large depth error • Too large: difficult search problem all of these points project to the same pair of pixels width of a pixel Large Baseline Small Baseline

42. The effect of baseline on depth estimation

43. 1/z width of a pixel width of a pixel pixel matching score 1/z

45. Multi-baseline stereo • Basic Approach • Choose a reference view • Use your favorite stereo algorithm BUT • replace two-view SSD with SSD over all baselines • Limitations • Must choose a reference view (bad) • Visibility! • CMU’s 3D Room Video

46. Outline • Stereo matching • - Traditional stereo • - Multi-baseline stereo • - Active stereo • Volumetric stereo • - Visual hull • - Voxel coloring • - Space carving

47. camera 1 camera 1 projector projector camera 2 Active stereo with structured light • Project “structured” light patterns onto the object • simplifies the correspondence problem Li Zhang’s one-shot stereo

48. Active stereo with structured light

49. Laser scanning • Optical triangulation • Project a single stripe of laser light • Scan it across the surface of the object • This is a very precise version of structured light scanning • Digital Michelangelo Project • http://graphics.stanford.edu/projects/mich/

50. Laser scanned models The Digital Michelangelo Project, Levoy et al.