270 likes | 281 Views
Interactive Virtual Relighting and Remodelling of Real Scenes. C. Loscos 1 , MC. Frasson 1,2 ,G. Drettakis 1 , B. Walter 1 , X. Granier 1 , P. Poulin 2 (1) iMAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes * iMAGIS is a joint project of CNRS/INRIA/UJF/INPG
E N D
Interactive Virtual Relighting and Remodelling of Real Scenes C. Loscos1, MC. Frasson1,2,G. Drettakis1, B. Walter1, X. Granier1, P. Poulin2 (1) iMAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes * iMAGIS is a joint project of CNRS/INRIA/UJF/INPG (2) Département d ’informatique et de recherche opérationnelle, Université de Montréal
Motivation Interior design Changes in lighting Geometric modification
Motivation • Goal: interactive system • simple capture process • interactive ( ~ 1 sec. per frame) • modification of lighting • modification of geometry
Motivation • We have to: • create a simple model of the real scene • geometry • approximate reflectance • represent real global illumination • develop interactive methods for modifications • Goal is to be convincing, not highly accurate
Previous Work • Geometric reconstruction • vision methods [Faugeras et al. 97, ...] (Realise) • constraint-based systems [Debevec et al. 96, Poulin et al. 98] • software: Photomodeler, etc. • Reflectance recovery • e.g., [Sato et al. 97, Ward92, Debevec98, Yu et al. 98, etc].
Previous Work • Real-time direct shadows • real point light source [State et al. 96] • Common global illumination • non-interactive • [Nakamae et al. 86, Fournier et al. 93, Jancène et al. 95, Debevec 98, Yu et al. 98, Yu et al. 99] • interactive • [Drettakis et al. 97, Loscos et al. 98]
Algorithm Overview • Input • Pre-process • Interactive modification
Algorithm Overview - Assumptions • Single viewpoint • Diffuse assumption • Lighting: • direct lighting: ray casting • indirect lighting: hierarchical radiosity radiosity = reflectance x ( direct light + indirect light )
Simple Input Process • Geometric reconstruction • several (4-5) images from different viewpoints • geometric modelling using “Rekon” [Poulin et al. 98] • Reflectance reconstruction • several (5-7) images from a single viewpoint • different lighting conditions: single light source at different positions • “radiance images”
Input • Radiance images from single viewpoint • combining multiple images reduces artefacts of estimation different lighting conditions
Pre-process • Computation of approximate diffuse reflectance pixel by pixel • compute individual reflectance images • merge reflectance images using confidence values • Initialise lighting system • data structure • hierarchical radiosity system
Reflectance Computation • For each radiance image reflectance=radiosity/(directlight+indirectlight) photograph reflectance
Confidence Images • Estimate confidence • confidence ~ quality of reflectance estimate • create a confidence image per light source position • Begin with confidence = Visibility • low in shadow regions • Filtering process to remove unwanted effects • low for outliers (specular effects, light tripod)
x avg. x Merged Reflectance Computation reflectance confidence merged reflectance
Reflectance Direct lighting Indirect lighting pixel Interactive Modification: Shadow Reprojection • Direct illumination: pixel by pixel • Indirect illumination: optimised radiosity solution a
Shadow Re-projection simulated photograph
original object insertion Add/move/remove object (virtual or real) • Visible surface changes: pixel by pixel local update • project bounding box of dynamic object • localise directly affected pixels
original object insertion Add/move/remove object (virtual or real) • Direct lighting updates: shaft structure • localisation of visibility changes (shadows) • accelerate visibility computation (blocker lists)
Add/move/remove object (virtual or real) • Indirect illumination computed by a radiosity solution (optimised by the shaft structure) • Example: moving object Position 1 Position 2
Removing Real Objects • Use of the reflectance image (lighting effects removed) to generate new textures reflectance images
Light Source Modification • Insertion of a virtual light source • computation for every pixel • new form-factors • new visibility • Indirect illumination: radiosity solution
Lighting Modification Original virtual lighting Insertion of virtual light
Conclusion • Input • data simple to acquire • Pre-process • data structures optimised for fast updates • Interactive modification • add and move virtual objects • remove real objects • relighting
Future Work • Improve reflectance computation • use of high dynamic range images (instead of RGB) • better control of indirect illumination • Allow motion of real objects • Faster: parallel computation
Future Work • Remove restrictions • diffuse reflectance [Yu et al. 99] • fixed view-point