1 / 49

3D Modeling using Multi-camera and Multi-lighting (MVML) Dome

3D Modeling using Multi-camera and Multi-lighting (MVML) Dome. Broadband Network and Digital Media Lab (BBNC), Automation Department, Tsinghua University Yebin Liu http://media.au.tsinghua.edu.cn/liuyebin.jsp. Introduction. 宽带网数字媒体技术实验室. Broadband Network & Digital Media Lab.

presta
Download Presentation

3D Modeling using Multi-camera and Multi-lighting (MVML) Dome

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D Modeling using Multi-camera and Multi-lighting (MVML) Dome Broadband Network and Digital Media Lab (BBNC), Automation Department, Tsinghua University Yebin Liu http://media.au.tsinghua.edu.cn/liuyebin.jsp 3D Modeling Using MVML Dome

  2. Introduction 宽带网数字媒体技术实验室 Broadband Network & Digital Media Lab 10-year history 1999~2009 1. finished or undergoing more than 10 national major R/D projects 2. Awards for National Scientific and Technological Progress 3D Modeling Using MVML Dome

  3. Introduction 宽带网数字媒体技术实验室 Broadband Network & Digital Media Lab Homepage://media.au.tsinghua.edu.cn Head: Prof. Qionghai Dai • Born in Dec,1964. Ph.D, IEEE Senior Member • Distinguished Young Scholar of NSF China • Second Prize for National Scientific and Technological Progress 2008 • The chief scientist of national 973 Program 16 PHD Students and 23 Master students Most of the PhDs are junior, and interest in Vision and Graphics I spent 7 years in this lab. My working experiences: video streaming and coding (02’) light field and 3D display (05’) MVML dome (07’) 3D Modeling Using MVML Dome

  4. Contents 3D Modeling Using MVML Dome

  5. Contents 3D Modeling Using MVML Dome

  6. Background • 3D Studio • IBMR • FVV and performance capture • Light Stage • IBL • Performance relighting • Light Field • IBR • Computational photography 3D Modeling Using MVML Dome

  7. Multi-camera Multi-light Dome Diameter6m 40 flea2 camera A Combination of 3D Studio, Light Stage and Light Field LEDs for lighting Lamp for color calibration screen for chrome-keying 3D Modeling Using MVML Dome

  8. Topology and the Control Module 3D Modeling Using MVML Dome

  9. Multi-camera Multi-light Dome 3D Modeling Using MVML Dome

  10. Multi-camera Multi-light Dome 10 3D Modeling Using MVML Dome 2014/9/14

  11. Multi-camera Multi-light Dome

  12. Multi-camera Multi-light Dome • Resolution: 1024 by 768, frame rate: 25fps • FVV data and MVML data are available on http://media.au.tsinghua.edu.cn/fvv.jsp http://media.au.tsinghua.edu.cn/mvml.jsp 3D Modeling Using MVML Dome

  13. Contents 3D Modeling Using MVML Dome

  14. Background Multi-view Stereo (MVS): reconstruct 3D model from multiple calibrated photographs of a realistic object Available MVS methods (according to Middlebury Benchmark): 1. Volumetric based MVS 2. Surface evolution MVS 3. Depth Map merging MVS (our work is in this kind) 4. Feature propagation MVS MVS in category 3,4 are accurate but not robust 3D Modeling Using MVML Dome

  15. Former work: PCMVS • A Point Cloud (Depth Map) based FVV System1 Furukawa’07. Depth map based MVS achieve both high accuracy and robustness! FVV1 Our work. FVV2 1. Y.Liu, Q.Dai, W.Xu. A Point cloud based Multi-view stereo for Free-viewpoint video . accepted in TVCG. 3D Modeling Using MVML Dome

  16. Former work: PCMVS PCMVS-BINO PCMVS-10 PCMVS-20 • Under sparse array mode: • Binocular is much better • Stereo matching is crucial in MVS 3D Modeling Using MVML Dome

  17. Why Stereo Matching Hard? 3D Surface Epipolar line Matching Error! Target view Referenceview 3D Modeling Using MVML Dome

  18. Popular Matching Techniques Mview Stereo Optical flow Stereo cost define unit Optimization mechanism Method 3D Modeling Using MVML Dome

  19. Proposed Pipeline • We Propose： Variational 3D Modeling Using MVML Dome

  20. Variation Goes for MVS • 1.Continuous and dense Capture • 2.Rotation invariant matching • 3.VH andepipolar reduce the probability of local minima 3D Modeling Using MVML Dome

  21. Multiple Multi-scale Iteration Multiple candidates by starting the iteration from different scales Multi-scale coarse-to-fine iteration technique* Depthmapv2 Depthmapv1 Depthmapv3 Depthmapv4 Multi-scale image pyramid *. T.Brox, A.Bruhn, N.Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. ECCV’04. 3D Modeling Using MVML Dome

  22. Multiple Candidates Higher starting scale Different versions generate different well reconstruction areas well reconstruct bad reconstruct 3D Modeling Using MVML Dome

  23. Refined Depth Map Synthesis Candidate patches Choose the best patch from all candidates using patch based NCC measurement Accurate Matching Target view Reference view 3D Modeling Using MVML Dome

  24. Multi-view Depth Maps Merging View 1 before View 5 before View 9 before Combine before View 1 after View 5 after View 9 after Combine after 3D Modeling Using MVML Dome

  25. Reconstruction Performance Measured by accuracy and completeness MStereo Benchmark by Middleburycollege 3D Modeling Using MVML Dome

  26. Furukawa CVPR07 Bradley CVPR07 High performance on detail reconstruction Ground Truth Liu CVPR09 3D Modeling Using MVML Dome

  27. Experimental Results 3D Modeling Using MVML Dome

  28. CMVS Summary Pros: Continuous Depth Estimation Detail reconstruction Multiple candidates + NCC metric on Continuous surface Robust reconstruction Cons: Requires an initial depth map for each view Visual hull may not necessary Complexity: MSS optical flow is the bottleneck (half an hour for DinoSparseRing) Optimization the OF algorithm is important 3D Modeling Using MVML Dome

  29. Contents 3D Modeling Using MVML Dome

  30. Background Multi-view Stereo Multi-view Single-light Cons: Highlights, Textureless, Cycle-texture 3D Reconstruction Cons: Continuous assumption Only obtain shape, not geometry Photometric Stereo Multi-light Single-view Multi-view Multi-light 3D Reconstruction Multi-light Multi-view 3D Modeling Using MVML Dome

  31. Multi-view Photometric Stereo Views Lights Directional Illumination Multiplex Illumination 3D Modeling Using MVML Dome

  32. Multi-view Photometric Stereo • No Illumination Constraints －Illuminations is represented by low order spherical harmonics* －No constraints on location, intensity and assumptions of the lights *. R.Basri, D.W.Jacobs, "Lambertian Reflectance and Linear Subspaces", TPAMI'03 3D Modeling Using MVML Dome

  33. Multi-view Photometric Stereo • No Manual Interactions • －Space-light MVS provides initial normal information －Alternating optimization (normal and light)# • Watertight and Robust Reconstruction －Multi-view images provide watertight reconstruction －Suitable for objects with small portion of non-Lambert. surface • Key Model Recon. for Performance Capture Systems* －Bypass laser scan and model registration －Standstill for 1 second # R.Basri, D.Jacobs, I.Kemelmacher, "Photometric Stereo with General, Unknown Lighting", IJCV'06 * E.d.Aguiar, C.Stoll, C.Theobalt, N.Ahmed, H.P.Seidel, S.Thrun, "Performance Capture from Sparse Multi-view Video", SIGGRAPH'08 * D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08 3D Modeling Using MVML Dome

  34. Experimental Results 3D Modeling Using MVML Dome

  35. Experimental Results 3D Modeling Using MVML Dome

  36. Experimental Results 3D Modeling Using MVML Dome

  37. Results: Impacts of Lights 3D Modeling Using MVML Dome

  38. Results: Synthetic Datasets D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08 3D Modeling Using MVML Dome

  39. Results: Relighting 3D Modeling Using MVML Dome

  40. Results: Relighting 3D Modeling Using MVML Dome

  41. MPS Summary Pros: No illumination constraints (Unknown lights) No manual interactions Watertight and robust reconstruction Practical in performance capture, animation, relighting, etc. 1. How to extend to Motion Actors? 2. Extend to Outdoor Scenes (Complex illumination) 3. MPS under Sparse Sampling Future works: light: light: √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ × √ √ √ × × √ × × × √ √ √ × √ × √ √ √ × √ view view 3D Modeling Using MVML Dome

  42. Contents 3D Modeling Using MVML Dome

  43. The Vision Field MVML dome is a sampling of the visual field Vision Vision Field: the full space (1d View+1d Light+1d Time) on where vision problems define “Matching” in lower subspace vs. “Fusion” in higher subspace and full space 3D Modeling Using MVML Dome

  44. Vision Field: Future Works • For free-motion object, It is impossible to capture the full Time-Light subspace. • Only a few works have addressed this problem* • Can be adopted in MPS and relighting motion objects. light light light Matching and Completion time time time Impossible mode 1d Sparse sample Reconstructed subspace √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ *. B.D.Decker, J.Kautz, T.Mertens, P.Bekaert, "Capturing Multiple Illumination Conditions using Time and Color Multiplexing", CVPR'09 3D Modeling Using MVML Dome

  45. Vision Field: Future Works Reconstruction of full space vision field using surface matching Texture Mapping N Lights N Lights 1/30 S 1/30 S 1/30 S 1/30 S Target Model Tracking Model Surface Matching Surface animation techniques: 3D Modeling Using MVML Dome

  46. Vision Field: Future Works • The MVML dome achieves the samplings of the vision field. • It is a foundation platform for the research on sampling theory of the vision field. • We hope to extend results obtain in MVML dome to outdoor sconces and other practical applications. Un-synchronized Un-calibrated Un- known 3D Modeling Using MVML Dome

  47. Conclusion • Continuous depth map based MVS is prospective for its accuracy, robustness and flexibility • The Vision Field and the MVML dome has still lots of interesting problems haven't been investigated 3D Modeling Using MVML Dome

  48. Acknowledgement • Bennett Wilbrum • Moshe ben-ezra 3D Modeling Using MVML Dome

  49. 宽带网数字媒体技术实验室 Broadband Network & Digital Media Lab We hope for corporative researches on related topics Welcome to visit BBNC and Tsinghua University! Thank you! 3D Modeling Using MVML Dome

More Related