Low-cost Photometric Calibration for Interactive Relighting

1. Low-cost Photometric Calibration for Interactive Relighting Céline Loscos and George Drettakis • iMAGIS*-GRAVIR/IMAG-INRIA • Computer Sciences Department - University College of London

2. Context: augmented reality • Mix real and virtual worlds • Applications • entertainment (virtual studio) • cinema (special effects) • medical • etc. • Richer virtual experience • real world references

3. Common illumination Common illumination [Breen et al. 96]

4. Motivation • Applications in augmented reality • need of interactive systems • simulation of realistic illumination needs time • Find the right balance between • the capture process speed • the response speed of the system • the quality of the lighting simulation convincing results

5. Overview • State of the art • relighting and remodelling for several known lighting conditions [Loscos et al. 99] • Photometric calibration • Conclusion

6. Realistic lighting simulation • Global illumination • direct (from light sources) + indirect light (inter-reflections) • Lighting simulation • Input • scene geometry + reflectance and emittance properties of surfaces • Output • lit scene • Reflectance: describes the portion of light reflected • Classical methods: ray casting or radiosity

7. Inverse illumination • Goal • find radiometric properties (reflectance, light source exitance) • real scene known independently of the original lighting conditions allows relighting • Inverse illumination [Sato et al. 97, Yu et al. 99, etc] • input • lit scene • output • reflectance estimation

8. Our relighting method [Loscos et al. 1999] • Interactive relighting of real scenes • Realistic common illumination • consistency of lighting between real and virtual • Simple capture process • few photos • low-cost equipment

9. Assumptions • Relighting from a single viewpoint • Diffuse scene • Direct lighting: ray tracing • Indirect lighting: hierarchical radiosity

10. Input data: reflectance estimate • Radiance images from a single viewpoint • a single light source per image Different lighting conditions

11. Reflectance estimate pixel per pixel • For each radiance image • Indirect approximated by an ambient term reflectance=radiosity/(directlight+indirect light) Original photograph Estimated reflectance

12. Merged reflectance reflectance confidence Merged reflectance x avg. x

13. Results of the method [Loscos et al. 1999] Video

14. Limitations in the reflectance estimate • Colours transformed by the camera • loss of information: saturation, etc. • Inaccuracy of the reflectance estimate reflectance Reflectance pixels

15. Solution: High-Dynamic Range images • Radiance images [Debevec et al. 97] • Input • several pictures from the same point of view at different shutter speeds • RGB values within integer range [0-255] • Output • camera’s response function • high-dynamic range of colours • Remark: need to control the shutter speed

16. Adaptation: low-cost HDR images • New solution for a semi-automatic digital camera Kodac DC260 • No direct control of the shutter speed • Use of the EV parameter provided by the camera

17. Adaptation: low-cost HDR images • 9 EV values [-2..2] = 9 different exposure times • EV = 0 : automatically chosen shutter speed • Use of the conversion typically used in photography • Fix to an arbitrary value ( EV = 0) • Results in • better range of colours and less saturation

18. Limitations in the reflectance estimate • Problems • several lighting conditions • exposure time automatically selected by the camera • inconsistent radiance values • Make radiance images consistent • based on radiosity equation • least squares solution

19. Make radiance images consistent • Algorithm • choose a reference radiance image • compute a reference reflectance for the reference image (only for directly lit areas) • compute an error factor for each radiance image • apply this factor to get a consistent image

20. Limitations in the reflectance estimate • Incorrect illumination estimate • incorrect estimate in shadow areas

21. (indirect = ambient) Initial reflectance Indirect lighting iteration New reflectance Iterative algorithm for reflectance estimate • For each pixel: • convergence of reflectance values

22. Calibration results Reflectance RGB Initial radiance After iterations

23. Calibration results Reflectance for a scanline (RGB) reflectance pixels

24. Calibration results Reflectance for a scanline (initial radiance) reflectance pixels

25. Calibration results Reflectance for a scanline (after iterations) reflectance pixels

26. Calibration results Reflectance (single exposure time) RGB Initial radiance After iterations

27. Improvements due to calibration Reflectance (single exposure time) RGB Initial radiance After iterations

28. Conclusion • Photometric calibration • improvement of the reflectance estimate quality • respects the restrictions to the low-cost computation and equipment price

29. Future work • Improve the final display • apply the response function of the camera • apply a tone mapping • Simplify the capture process • General perspectives • specular effects • moving viewpoint • outdoor scenes • toward real time