Computational Photography: Expanding Camera Capabilities

1. Course 3: Computational Photography Ramesh RaskarMitsubishi Electric Research Labs Jack TumblinNorthwestern University Course WebPage : http://www.merl.com/people/raskar/photo

2. Course 3: Computational Photography Course WebPage http://www.merl.com/people/raskar/photo Course Evaluation http://www.siggraph.org/courses_evaluation

3. Welcome • Understanding Film-like Photography • Parameters, Nonlinearities, Ray-based concepts • Image Processing and Reconstruction Tools • Multi-image Fusion, Gradient domain, Graph Cuts • Improving Camera Performance • Better dynamic range, focus, frame rate, resolution • Future Directions • HDR cameras, Gradient sensing, Smart optics/lighting

4. Speaker: Jack Tumblin Jack Tumblin is an Assistant Professor of Computer Science at Northwestern University. His interests include novel photographic sensors to assist museum curators in historical preservation, computer graphics and visual appearance, and image-based modeling and rendering. During his doctoral studies at Georgia Tech and post-doc at Cornell, he investigated tone-mapping methods to depict high-contrast scenes. His MS in Electrical Engineering (December 1990) and BSEE (1978), also from Georgia Tech, bracketed his work as co-founder of IVEX Corp., (>45 people as of 1990) where his flight simulator design work was granted 5 US Patents. He is an Associate Editor of ACM Transactions on Graphics, was a member of the SIGGRAPH Papers Committee (2003, 2004), and in 2001 was a Guest Editor of IEEE Computer Graphics and Applications. http://www.cs.northwestern.edu/~jet

5. Speaker: Ramesh Raskar Ramesh Raskar is a Senior Research Scientist at MERL. His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002, 2003 and 2004. He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005. He is a member of the ACM and IEEE. http://www.merl.com/people/raskar/raskar.html

6. Opportunities • Unlocking Photography • How to expand camera capabilities • Digital photography that goes beyond film-like photography • Opportunities • Computing corrects for lens, sensor and lighting limitations • Computing merges results from multiple images • Computing reconstructs from coded image samples • Cameras benefit from computerized light sources • Think beyond post-capture image processing • Computation well before image processing and editing • Learn how to build your own camera-toys

7. Traditional Photography Detector Lens Pixels Image

8. Picture Computational Photography: Optics, Sensors and Computations GeneralizedSensor Generalized Optics Computations Ray Reconstruction 4D Ray Bender Upto 4D Ray Sampler

9. Computational Photography Novel Cameras GeneralizedSensor Generalized Optics Processing

10. Computational Photography Novel Illumination Light Sources Novel Cameras GeneralizedSensor Generalized Optics Processing

11. Computational Photography Novel Illumination Light Sources Novel Cameras GeneralizedSensor Generalized Optics Processing Scene: 8D Ray Modulator

12. Computational Photography Novel Illumination Light Sources Novel Cameras GeneralizedSensor Generalized Optics Processing Display Scene: 8D Ray Modulator Recreate 4D Lightfield

13. Computational Photography Novel Illumination Light Sources Modulators Novel Cameras Generalized Optics GeneralizedSensor Generalized Optics Processing 4D Incident Lighting Ray Reconstruction 4D Ray Bender Upto 4D Ray Sampler 4D Light Field Display Scene: 8D Ray Modulator Recreate 4D Lightfield

14. A Teaser: Dual Photography Projector Photocell Scene

15. A Teaser: Dual Photography Projector Photocell Scene

16. A Teaser: Dual Photography Projector Photocell Scene

17. A Teaser: Dual Photography Projector Photocell Camera Scene

18. The 4D transport matrix:Contribution of each projector pixel to each camera pixel projector camera scene

19. The 4D transport matrix:Contribution of each projector pixel to each camera pixel projector camera scene Sen et al, Siggraph 2005

20. The 4D transport matrix:Which projector pixel contribute to each camera pixel projector camera ? scene Sen et al, Siggraph 2005

21. Dual photographyfrom diffuse reflections the camera’s view Sen et al, Siggraph 2005

22. 1558 A Brief History of Images Camera Obscura, Gemma Frisius, 1558

23. A Brief History of Images 1558 1568 Lens Based Camera Obscura, 1568

24. A Brief History of Images 1558 1568 1837 Still Life, Louis Jaques Mande Daguerre, 1837

25. 1558 A Brief History of Images 1568 1837 Silicon Image Detector, 1970 1970

26. 1558 A Brief History of Images 1568 1837 1970 Digital Cameras 1994

27. Dream of A New Photography Old New • People and Time ~Cheap Precious • Each photo Precious Free • Lighting Critical Automated* • External Sensors No Yes • ‘Stills / Video’ Disjoint Merged • Exposure Settings Pre-select Post-Process • Exposure Time Pre-select Post-Process • Resolution/noise Pre-select Post-Process • ‘HDR’ range Pre-select Post-Process

28. Survey • How many of you are photographers ? • How many of you are photo-artists ? • How many of you are involved in camera design/development ? • How many do active programming ? • Field of work: Academics? Industry ? Research ? Art ?

29. Schedule • 8:30 Introduction (Raskar) • 8:40 Photographic Signal & Film-like Photography (Tumblin) • 9:10 Image Processing Tools (Raskar) • 9:40 Improving Film-like Photography (Tumblin) • 10:15 Break • 10:30 Image Reconstruction Techniques (Raskar) • 11:15 Smart Lights and Beyond Photography (Tumblin) • 11:45 Smart Optics and Sensors (Raskar) • 12:05 Discussion Course Page : http://www.merl.com/people/raskar/photo