Peter Sturm

1. An overview of multi-view stereo and other topics in3D modeling from 2D images Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) Currently: sabbatical at TUM (Humboldt-fellow)

2. Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

3. Overview Before about 1995 [Mohr et al.] Currently [Furukawa et al.]

4. Overview Ingredients and factors of improvement: • Geometry • Image matching • Statistics • Numerical optimization • Better data • Multi-view stereo Other approaches and related subjects: • Other 3D modeling approaches • Camera and object tracking • Omnidirectional vision • Sensor design • ... Current research directions

5. Before about 1995 3D modeling from multiple images: • image matching often done by hand • main focus: given the matching, how to estimate camera positions, camera calibration, 3D point positions? • 1990’s: golden age of multi-view geometry [Mohr et al.] Exceptions, concerning manual matching: • binocular stereo, aerial imagery • close-range photogrammetry, mo-cap • 3D modeling from silhouettes

6. 1996 The first fully automatic approaches • automatic matching • systematic use of robust statistics • going towards surface models (here, meshes) [Beardsley et al.]

7. 1998 Handling longer image sequences • better numerical optimization methods • system integration • better data: image quality, calibration [Fitzgibbon et al.]

8. Around 1998 Matchmoving applications (tracking of camera motion) in AR [Microsoft]

9. Around 1998 Matchmoving applications (tracking of camera motion) in AR VIDEO… [2d3]

10. At about the same time Polyhedral models from many georeferenced images [MIT City Scanning Project]

11. From 1998 onwards Going from “hacks” to mathematically founded surface representations and evolutions: level sets (now, increasedusage of meshes) [Keriven et al.]

12. About 2003 Towards truly dense 3D models: • probabilistic occlusion handling • point-based surface representations [Strecha et al.]

13. About 2007 Current state of the art • patch-based surface representation [Furukawa et al.]

14. About 2007 Current state of the art • patch-based surface representation [Furukawa et al.]

15. Multi-View Stereo Optimization: [Gargallo et al.]

16. Quick Summary Ingredients of multi-view stereo: • surface representation: point-based, patch-based, mesh, implicit surface, voxel sets, ... • quality measure: “photo-consistency”, invariance to lighting differences, ... • prior assumptions (regularization): smoothness, piecewise planarity, class-specific priors (e.g. faces) • optimization approach, resolution strategy [Vetter et al.]

17. Overview Ingredients and factors of improvement: • Geometry • Image matching • Statistics • Numerical optimization • Better data • Multi-view stereo Other approaches and related subjects: • Other 3D modeling approaches • Camera and object tracking • Omnidirectional vision • Sensor design • ... Current research directions

18. Other approaches and subjects Interactive 3D modeling based on user annotations • single-image 3D modeling [Criminisi et al.]

19. Other approaches and subjects Interactive 3D modeling based on user annotations • multi-image 3D modeling [Debevec et al.]

20. Other approaches and subjects Exploitation of other depth cues than stereo • shading • focus / defocus

21. Other approaches and subjects Active approaches using controlled illumination: • Photometric stereo • Flash photography • Structured light • ...

22. Other approaches and subjects 3D Modeling from silhouettes (“visual hull”) • Possible in real-time [Hernandez et al.]

23. Other approaches and subjects VIDEO…

24. Other approaches and subjects Real-time camera tracking: • Model-based object tracking [Fua et al.]

25. Other approaches and subjects Real-time camera tracking: • Model-based object tracking • SLAM (simultaneous localization and mapping) VIDEO… [Lhuillier et al.]

26. Other approaches and subjects Omnidirectional vision: • wide angle cameras • panoramic imaging • surveillance • navigation

27. Other approaches and subjects Some works on sensor design: • Filters for HDR mosaics • Lens arrays • Mirror design [Nayar et al.]

28. Other approaches and subjects Some works on sensor design: • Filters for HDR mosaics • Lens arrays, e.g. for computational photography [Georgiev et al.]

29. Overview Ingredients and factors of improvement: • Geometry • Image matching • Statistics • Numerical optimization • Better data • Multi-view stereo Other approaches and related subjects Current research directions: • Global optimization methods for geometry computations • Use of community image collections • Modeling surface appearance • Modeling deformable / articulated objects • ...

30. Current research directions Exploration of community image collections: • PhotoSynth / PhotoTourism [Snavely et al.]

31. Current research directions Detailed 3D modeling of large scenes • Interior of buildings, cities [Furukawa et al.] VIDEO…

32. Current research directions Modeling surface appearance: • Shiny objects [Lensch et al.]

33. Current research directions Modeling surface appearance: • Shiny objects [Birkbeck et al.]

34. Current research directions Modeling surface appearance: • “The Light Stage” [Debevec et al.]

35. Current research directions Modeling surface appearance: • Tanslucid, refractive, reflective surfaces, smoke, fire, ...

36. Current research directions Modeling complex geometries: [Paris et al.] [Quan et al.]

37. Current research directions Modeling deformable / articulated objects: [Fua et al.]

38. An overview of multi-view stereo and other topics in3D modeling from 2D images Peter Sturm INRIA Grenoble – Rhône-Alpes (Institut National de Recherche en Informatique et Automatique) Currently: sabbatical at TUM (Humboldt-fellow)