Sebastian Enrique Columbia University senrique@cs.columbia.edu

1. COMS 6160 – Real-Time High Quality Rendering Nov 3rd, 2004 Relighting Framework Sebastian Enrique Columbia University senrique@cs.columbia.edu

2. COMS 6160 Relighting Framework Nov 3rd, 2004 + = RELIGHTED SCENE IMAGES LIGHTS About Relighting What is Relighting? “Given a set of different illuminated images from a scene, relighting is the process of producing new images of the same scene with new lighting conditions, composing in some way the original data.” Why it is useful? Photorealistic real-time rendering of complex scenes with complex illumination is an open problem. One IBR approach is to capture or render a set of original images from a scene, and then relight it in real time to produce the same scene but with novel illumination! Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 2

3. COMS 6160 Relighting Framework Nov 3rd, 2004 ? IMAGES COMPRESSED IMAGES About Relighting (cont.) What is one of the most challenging parts of it? High quantity of images is needed to produce good results, the challenge is to find/use a good compression technique in order to relight fast and using as less memory as possible. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 3

4. COMS 6160 Relighting Framework Nov 3rd, 2004 Some Previous Work • Doorsey et al. used a progressive radiosity method where they pre-rendered synthetic scenes to simulate lighting conditions –superimposing single light source images- in opera stages in 1991. • Many (Hallinan ‘94, Epstein ‘95, etc.) have pre-acquired real images changing the lighting direction. • Debevec compressed the pre-acquired images in JPG and processed in the compressed domain in 2000. • Sloan uses low-frequency spherical harmonics on geometry in 2002. • In 2003 Sloan uses clustered or VQPCA on spherical harmonic coefficients. • Ng compressed data using wavelets in 2003. • Sloan (Local Deformable PRT) and Ramamoorthi (Triple Product Wavelets) are the most recent related techniques, 2004. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 4

5. COMS 6160 Relighting Framework Nov 3rd, 2004 LSD Relighting Approach • The Lighting Sensitive Display (LSD) was developed by Shree Nayar, Peter Belhumeur, and Terry Boult. • Basically, it is a monitor with an attached camera that captures the lighting conditions of the environment. The monitor shows an scene, which changes (is relighted) as the illumination in the environment changes. • They adopted an image-based approach, using a large set of images. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 5

6. COMS 6160 Relighting Framework Nov 3rd, 2004 LSD Relighting Approach (cont.) • They have developed a novel algorithm (using two stages of Principal Component Analysis or PCA) that compresses that large set of images and allows the relighting in real-time with complex lighting conditions. • They have reached compression ratios of 476:1 for colored images. • The algorithm simultaneously exploits correlations over the lighting domain as well as coherences over the spatial domain of the image. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 6

7. COMS 6160 Relighting Framework Nov 3rd, 2004 LSD Relighting Approach (cont.) • We based our Relighting Framework on the LSD approach. • In the following slides we will explain: • How the input images for the LSD algorithm should be taken. • First compression stage of the algorithm. • Second compression stage of the algorithm. • Real-Time rendering. • Then, we will get on: • Extending the LSD approach with the use of cubemaps. • Problems found. • Future directions. • Discussion and Videos Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 7

8. COMS 6160 Relighting Framework Nov 3rd, 2004 Setting Up Images • To take the images, a grid of light source positions is generated in the front face of an imaginary cube (a plane parallel to the image plane). • For each position on the grid, place the distant light with maximum intensity and generate (render or capture) an image. Scene to Render / Capture Distant Light Source For Single Image Fixed Viewpoint Grid of Positions for Distant Light Sources, it’s in the front face of an imaginary cube. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 8

9. COMS 6160 Relighting Framework Nov 3rd, 2004 First Stage SVD • The first SVD (Singular Value Decomposition) exploits the fact that locally (within small regions of the images) the variation due to changes in illumination can be approximated by a small number of bases. • The image is divided into m square blocks, each containing p pixels, and bases are computed for each. • Each image Ii is an image of the scene illuminated by a single distant point light source. Image divided in m square blocks Each block has p pixels Image Ii IMAGES Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 9

10. COMS 6160 Relighting Framework Nov 3rd, 2004 First Stage SVD (cont.) • Iij denote the jth block of the ith image. • For each block in the scene a low-dimensional approximation is computed as follows: • Ij is a p x n matrix representing a collection of image blocks. The I column of Ij is formed by p pixels of the jth block. • Using SVD, a rank b approximation to Ij is found as: • Ej is a p x b column-orthogonal matrix, called block bases. • Sj is a b x b diagonal matrix. • Cj is an n x b column-orthogonal matrix. • The singular values from Sj are absorbed into CjT, getting a b x n matrix, Lj, called block lighting coefficients. Each Ij represents the pixels for the same block on all images Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 10

11. COMS 6160 Relighting Framework Nov 3rd, 2004 • Applying Rank b: Ij  • Abosorving Sj values into CiT: x x p x n Ej Sj CjT Ij  p x b b x b b x n x p x n Ej Lj p x b b x n First Stage SVD (cont.) • SVD: Ij Ej Sj = x x p x n p x p p x n CjT diagonal matrix (singular values) n x n Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 11

12. COMS 6160 Relighting Framework Nov 3rd, 2004 L E (m * b) x n (m * p) x b First Stage SVD (cont.) • Then, stacking all of the m Ljblock matrices, we get the lighting coefficient matrixL. • The image bases m Ej are also stacked in the matrix E. • The collection of submatrices within E and L contain all the information needed to approximate the collection of images corresponding to the n lighting directions. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 12

13. COMS 6160 Relighting Framework Nov 3rd, 2004 First Stage SVD (cont.) Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10 Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 13

14. COMS 6160 Relighting Framework Nov 3rd, 2004 L  x (m * b) x n U V (m * b) x q q x n Second Stage SVD • The fact that there is much coherence in image blocks is exploited using a second SVD. • It is applied to the light coefficient matrix L: • U is an (m x b) x q column-orthogonal matrix, called lighting coefficient bases. • V is a q x n matrix, called compressed coefficient matrix. • Rank q denotes the number of linear bases kept to approximate L. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 14

15. COMS 6160 Relighting Framework Nov 3rd, 2004 Second Stage SVD (cont.) Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10; q=200 Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 15

16. COMS 6160 Relighting Framework Nov 3rd, 2004 Note block discontinuities! Examples Bad Rank Election (showing only red color component) Relighted Image SVD First Stage Rank 3 Original Image Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 16

17. COMS 6160 Relighting Framework Nov 3rd, 2004 Examples (cont.) Higher Ranks Only small differences in brightness are noticeable, but there are no visible block discontinuities. Relighted Image SVD First Stage: Rank 10 SVD Second Stage: Rank 20 Original Image (in fact, two combined original images) Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 17

18. COMS 6160 Relighting Framework Nov 3rd, 2004 Real-Time Rendering • Real-Time illumination is represented by the illumination field vectors. • Each element si corresponds to a point light, and its value represents the intensity that that point light is contributing with to the scene. • To render the relighted scene in real-time, preprocessed matrices E, U, and V should be used: • Compute a compressed coefficient vector as the product (q elements) • Compute a lighting coefficient vectorI as (m*b elements) • To render the j block, a subvector Ij from I must be multiplied with the corresponding stacked Ej matrix. This must be done each frame for the m blocks. (b elements) (p elements) Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 18

19. COMS 6160 Relighting Framework Nov 3rd, 2004 Real-Time Rendering (cont.) Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 19

20. COMS 6160 Relighting Framework Nov 3rd, 2004 Demo: Point Lights / David Image: 480x640; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20 Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 20

21. COMS 6160 Relighting Framework Nov 3rd, 2004 Grid of Positions for Distant Light Sources in the Full Cube Distant Light Source For Single Image Fixed Viewpoint Extending LSD: Cube Lighting • In place of using only the front face of the cube to capture images with different lighting condition, use all of the 6 faces. • The parameters that change are the quantity of input images and the length of the s light field vector (both now 6 * n). • In this way, the scene can be relighted with illumination coming for all around. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 21

22. COMS 6160 Relighting Framework Nov 3rd, 2004 High-Dynamic Range Cubic Environment Map • Having the relighting scheme for the full cube, we can add cubic environment maps to relight the scene. • To have more accuracy, High-Dynamic Range (HDR) cubic environment maps are used (floating point values in place of 0..255). • The CubicMap could be rotated, and each element in the s light field vector is contributed by the value of the corresponding texel, weighted by the solid angle. • HDR(i) is the corresponding texel value; N is the corresponding cube face normal; R is the vector from the origin to the center of the element i in the cube grid; m is the face cube grid resolution (squared is how many images per face exist); each cube face has a size of 2 x 2. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 22

23. COMS 6160 Relighting Framework Nov 3rd, 2004 Demo: CubeMap & Point Lights / Nicole Image: 512x512; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20 Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 23

24. COMS 6160 Relighting Framework Nov 3rd, 2004 Eb = […] Ub = […] Vb = […] PREPROCESS (SVD 1 & 2) IMAGES MATRICES Eb = […] Ub = […] Vb = […] REAL-TIME RELIGHTING FRAMEWORK MATRICES RELIGHTED SCENE HDR Framework Summary Generate Scene-Matrices For Images Set Of Each Scene to Relight Relight in Real-Time the Given Scene-Matrices with HDR CubeMap and Point Lights Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 24

25. COMS 6160 Relighting Framework Nov 3rd, 2004 Problems Found • Preprocess takes TOO MUCH TIME. • Preprocess uses TOO MUCH MEMORY. • Few things could be done in current graphics hardware (HDR cubemap processing specially). Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 25

26. COMS 6160 Relighting Framework Nov 3rd, 2004 Future Directions • Optimize preprocessing stages (distributed computations?) • Use error metrics to automatically select adequate ranks. • Extend user interface to allow the relighting using lower ranks than that given in the input matrices. • Allow viewpoint changes. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 26

27. COMS 6160 Relighting Framework Nov 3rd, 2004 End of Talk • Ready to hear… • Comments • Suggestions • Discussions • Questions • More VIDEOS to show while chatting… Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 27