240 likes | 257 Views
Recovering Geometric, Photometric and Kinematic Properties from Images. Jitendra Malik Computer Science Division University of California at Berkeley Work supported by ONR, Interval Research, Rockwell, MICRO, NSF, JSEP. Physics of Image Formation. Lighting BRDFs Shape and Spatial layout
E N D
Recovering Geometric, Photometric and Kinematic Properties from Images Jitendra Malik Computer Science Division University of California at Berkeley Work supported by ONR, Interval Research, Rockwell, MICRO, NSF, JSEP
Physics of Image Formation • Lighting • BRDFs • Shape and Spatial layout • Internal DOFs Images
Solving inverse problems requires models • Define suitable parametric models for geometry, lighting, BRDFs, and kinematics. • Recover parameters using optimization techniques. • Humans better at selecting models; computers at recovering parameters.
But there will always be unmodeled detail….. • Models are always approximate. • Adding more parameters doesn’t help; data will be insufficient to recover these parameters.
Hybrid Approaches are best! • ANALYSIS • use images to recover a subset of object parameters. These are chosen judiciously so that they can be recovered robustly • SYNTHESIS • render using appropriately selected images or subimages, transformed using the model.
Talk Outline • Geometry • Debevec, Taylor and Malik, SIGGRAPH 96 • Photometry • Yu and Malik, SIGGRAPH 98 • Debevec and Malik, SIGGRAPH 97 • Kinematics • Bregler and Malik, CVPR 98
Modeling and Rendering Architecture from Photographs George Borshukov Yizhou Yu Paul Debevec Camillo Taylor Jitendra Malik Computer Vision Group Computer Science Division University of California at Berkeley
Overview • Photogrammetric Modeling • Allows the user to construct a parametric model of the scene directly from photographs • Model-Based Stereo • Recovers additional geometric detail through stereo correspondence • View-Dependent Texture-Mapping • Renders each polygon of the recovered model using a linear combination of three nearest views
Our Modeling Method: • The userrepresents the scene as a collection of blocks • The computersolves for the sizes and positions of the blocks according to user-supplied edge correspondences
Block Model User-Marked Edges Recovered Model
Arc de Triomphe Modeled from five photographs by George Borshukov
Surfaces of Revolution Taj Mahal modeled from one photograph by G. Borshukov
Synthetic View Photograph Recovered Model
Recovering Additional Detailwith Model-Based Stereo • Scenes will have geometric detail not captured in the model • This detail can be recovered automatically through model-based stereo
Scene with Geometric Detail Approximate Block Model
Model-Based Stereo • Given a key and an offset image, • Project the offset image onto the model • View the model through the key cameraWarped offset image • Stereo becomes feasible between key and warped offset images because: • Disparities are small • Foreshortening is greatly reduced
Key Image Warped Offset Image Offset Image Disparity Map
Synthetic Views ofRefined Model Four images composited with View-Dependent Texture Mapping
Rendering with View-DependentTexture Mapping • Triangulate the view hemisphere • For each polygon, determine which images viewed it from which angles • Label each triangle vertex according to best viewed image 2 5 1 4 3 view hemisphere
Rendering with View-DependentTexture Mapping • To render, determine to which triangle the viewpoint belongs • Compute Barycentric weights for the triangle vertices • Render the polygon with a weighted average of the three vertex images 2 5 1 4 3 view hemisphere
The Campanile (Debevec et al) • 20 photographs used • approx. 1-2 weeks of modeling time. • Real time rendering
Recovered Campus Model Campanile + 40 Buildings