Photorealistic Animation Rendering with Energy Redistribution

1. Photorealistic AnimationRendering with Energy Redistribution Yu-Chi Lai 賴祐吉 University of Wisconsin - Madison

2. Agenda Introduction Physically-based rendering methods Population Monte Carlo energy redistribution Future works

3. Goal and Applications Goal: generate realistic animations Applications Movie Interactive entertainments Computer games Virtual reality walk-throughs Light Engineering Etc. From Day After Tomorrow

4. Applications Animation: Kristensen et al. Grand Theft Auto 4 IGNEntertainment

5. Modeling and Simulating Appearance

6. Agenda Introduction Background Population Monte Carlo energy redistribution Future works

7. Rendering Equation • Reflection Equation • Energy Balance Equation • Difficulty: Lr appear in both sides of equation => Fredholm equation of the second type Incoming light Sum BRDF Light reflected incoming light reflected at the point

8. Physically-based Rendering Render images according to physical principles Radiosity: finite element Ray-tracing: based on Monte Carlo integrations Unbiased: path tracing, bidirectional path tracing, metropolis light transport, energy redistribution path tracing and so on. Biased: irradiance caching, photon mapping, and so on. From Jenson et al.

9. Path Integral for General Integration Measurement equation: Path Integral: • : a path • Ω: path space • : area product measurement • : contribution of a path Difficulty: high dimensions, computation grow exponentially when using deterministic integrations

10. We can estimate the integral by generating a set of samples: Different ways to generate samples: path tracing, light tracing, bidirectional, … Monte Carlo Algorithms

11. Issues with MC methods • Computationally expensive • Variance reduced slowly with number of samples • Reuse path samples From PBRT Book

12. Frame-by-Frame Photorealistic Animation Rendering Takes a long time to generate Several hours per frame is industry standard Temporal noise Flickering Shimmering From Max-Planck Institute, German

13. Physically-based Animation Rendering Right now the research is mainly from German’s MPI, USA’s UCSD, Standford, and Cornell. Parallel independently rendering => temporal coherence Use multiple processors to do independent ray tracing [Warld’01] Coherently trace rays: Kdtree, Grids, and bounding volume hierarchy (BVH) [Wald’01b, Wald’07a] Efficient update schemes of acceleration structure [Popov’06, Lauterbach’06, Yoon’07] Implement global illumination in GPU architecture including radiosity, photonmapping, ray tracing, …[Purrel’02, Purell’03] Copy the samples across the frames => validity Irradiance Reuse.[Martin’99, Tawara’02, Tawara’04, Smyk’05] Photon shooting (next slide)

14. Photon Shooting • Photon shooting algorithm [Myszkowski’01, Dmitriev’02, Weber’04] • Instant Radiosity [Keller’97, Wald0’2a, Laine’07] • Light cut algorithm: [Walter’05, Hansan’07, Hansan’08] • Create a set of point light sources in the entire animation • Cluster the light sources into a tree according to the perceptual metrics in temporal and spatial domain Animation by Hansanet al

15. Reuse Path Samples • Camera is allowed to move [Briere’96, Murakami’89, Jevans’92, Sequin’89, Bala’99, Fernandez’00, Havran’03, Mendez’06] • Light source is allowed to move or change properties: [Sbert’04a, Sbert’04b , Sbert’04C, Ghosh’06] Animation by Havranet al Animation by Ghoshet al

16. Grand Challenge Efficiently render entire animation Do parallel computation in multi-processors or GPU. Do coherent ray tracing. Efficiently update the acceleration structure. Reuse samples from previous computation Reduce the temporal artifacts Explore the temporal coherence among paths Reuse the computation results. Perceptual evaluation of the rendering results. Evaluate the strength of each algorithm. Allow us to distribute computation to perceptual important features.

17. Agenda Introduction Background Population Monte Carlo energy redistribution Future works

18. Challenges • How to adjust the sampling parameter according to path or scene properties? • How to concentrate more computation on paths that are important without introducing bias? • How to reuse paths temporally? • How to handle the huge computation for animation rendering?

19. Markov Chain Monte Carlo Write the integrand as: : the sensor measurement for pixel j of frame k : represents all other factors : represents the radiant energy passing through the image sweep • Create the distribution of paths in animation proportional to the contribution

20. Metropolis and Energy Redistribution Path Tracing • Generate a sequence of paths: where path is generated according to

21. Overview of Population Monte Carlo Energy Redistribution Animation Rendering System

22. PMC-ER in Each Frame Process

23. Detail • Preprocess: collect the information for the following computation • Energy redistribution: distribute the energy to similar paths by using spatial and temporal perturbations. • Resampling: • Eliminate paths from the population. • Generate replaced paths to determine the area of exploration. • Adjust the rendering parameters

24. Adjust Sampling Parameters (EGSR07) • Different regions have different details • We would like to adjust sampling parameters accordingly • Low detailed regions: large distribution radius • High detailed regions: smaller distribution radius

25. Population Monte Carlo Algorithm To estimate integral At t iterations, a PMC estimator of the integral is given by

27. Kernel Function and Adaptation • Kernel Function • Adapt values: to choose proper perturbation radius • Initialize them to constant values when a path is generated • After each successful perturbation, the acceptability is labeled with the perturbation radius, and the path, i • By using

28. Adaptation Results • Color represents perturbation radius • Red: 5, Green: 10, Blue: 50

29. Cornell Box ERPT PMC-ER

30. Room Scene ERPT PMC-ER

31. Concentrate More Computation on Certain areas (ISVC) • Stratified exploration of the image plane • The importance of regions on the image are not perceptually the same. • Some types of paths are visually more important and harder to find. PMC-ER 4SPPs PMC-ER 8SPPs

32. Regeneration (I) • Perceptually distributed pixel positions according to which is the radiance sample variance in each pixel • Weighting

33. Regeneration (II) • Use light tracing to generate a valid light paths • Link each surface vertex to the camera to form a set of valid paths • Evaluate whether it is a caustics path • Weighting

34. Results 4SPPs+Reg 4SPPs 8SPPs

35. Results 6SPPs+Reg 6SPPs 12SPPs 18SPPs

36. Problem in Frame-by-Frame Rendering • Each frame takes long time. • Parallel rendering with condor system • Temporal artifacts: temporal perturbation

37. Temporal Perturbation • Update the position of diffuse vertices • Reconstruct the specular sub-path • Check the validity of the path

38. Cornell Box Frame-By-Frame With Temporal Perturbation

39. Chess Body Frame-By-Frame With Temporal Perturbation

40. Room Scene Frame-By-Frame With Temporal Perturbation

41. Chess Board Frame-By-Frame With Temporal Perturbation

42. Basement Frame-By-Frame With Temporal Perturbation

43. Contributions • A new rendering algorithm based on PMC framework • Correlatedly explore important paths • Automatically adjust energy redistribution area according to the information collected in previous iterations • Elimination-regeneration to achieve ergocity and adjust the exploring area according to paths’ remaining energy • New lens perturbation method • Increase the caustics perturbation success rate • Ease the control of caustics perturbation on the image plane • New regeneration methods • Concentrate the computation on perceptual important regions • Concentrate the computation on perceptual important types of paths. • Temporal perturbation method: exploration the temporal coherence among paths • A algorithm allows us to render a scene in parallel

44. Limitations • Human observation is the evaluation tool for animation quality. • Dark regions are hard to get the chance to be explored and thus are relatively noisy. Although it is hard to notice in single image, this becomes an issues because human perception is very sensitive to this kind of temporal inconsistency. • Temporal perturbations in each condor process will create a large set of temporal files for related frames. Transferring and updating data in condor daemon process involves a large number of disk IOs. • Our variance-sample distribution criterion is based on variance of sample radiances but this did not represent the result after energy redistribution. • We separate perturbations into two types, temporal and spatial perturbations, and this makes control harder and the initial probing samples for each perturbation is relative low. • Limit the light to area light sources and the efficiency goes down when the number of lights goes up

45. Agenda Introduction Background Population Monte Carlo Energy Redistribution Future Works

46. Future Works • Animation quality evaluation algorithms • Current available perceptual animation quality evaluation algorithm is for video compression. • Adjust the quality perceptual evaluation algorithm for Monte Carlo algorithms • Construct a temporal filter based on result of the temporal perturbations: • The result of temporal perturbation can create the relations among pixels in different frames, if we can use this relation information to create a temporal filter accordingly, we should be able to reduce the temporal artifact and iterations to generate a smooth result • Develop an perturbation which perturb in spatial domain randomly but perturb in a fixed regions in temporal domain deterministically

47. Future Works • Apply Population Monte Carlo with path tracing into animation rendering using environment map lighting. • A path has the form of L(D|S)C. • For each path of current frame, temporally trace the path with a proper temporal perturbation algorithm to enhance the temporal coherence among frames. • Spread the photon collection positions in photon splatting with Metropolis: • Generate a set of collection positions • Use temporal perturbation to correlatedly generate new photon collection positions from the previous frame. • Use the light cut or light clusters algorithm to solve the many light problem.

48. Future Research • Explore the parallel ability in GPU • Energy redistribution path tracing are naturally parallel. => transform the ERPT algorithms onto GPU • Explore the research possibility in environment map lighting and shadow generation. • Construction and application of flock tiles: • If we can find a common set of temporal and spatial boundary conditions for setting up a tile. • We can use the constraint simulation to simulate the inner agents’ motion according to flock rules • The animation tiles can be used to construct a seamless animation such as a large crowd in a city, a school of fishes, a flock of birds or the traffics in a city.

49. Publications • Computer Science • Yu-Chi Lai, Steven Chenney, Shaohua Fan “Group Motion Graphs”, Eurographics/SIGGRAPH Symposium on Computer Animation 2005, pp. 281–290. • Yu-Chi Lai, Shaohua Fan, Stephen Chenney, and Charles Dyer, “Photorealistic Image Rendering with Population Monte Carlo Energy Redistribution”, Eurographics Symposium on Rendering, 2007, pp. 287-296. • Yu-Chi Lai, Feng Liu, Li Zhang, and Charles Dyer, “Efficient Schemes for Monte Carlo Markov Chain Algorithms in Global Illumination”, Proc. 4th International Symposium on Visual Computing, 2008. • Yu-Chi Lai, Feng Liu, and Charles Dyer,“Physically-based Animation Rendering with Markov Chain Monte Carlo”, (submit to Eurographics Symposium on Rendering 2009)

50. Publications • Computer Science (Others) • Shaohua Fan, Stephen Chenney and Yu-Chi Lai“Metropolis Photon Sampling With Optical User Guidance”, Eurographics Symposium on Rendering, 2005, pp. 127-138 • Yu-Chi Lai, Stephen Chenney, Shaohua Fan, “Data-Driven Group Animation”,Technical Report, Department of Computer Sciences, University of Wisconsin-Madison, 2005 • Yu-Chi Lai, Shaohua Fan, and Charles Dyer, “Population Monte Carlo Path Tracing”, Technical Report, Department of Computer Sciences, University of Wisconsin-Madison, 2006 • Shaohua Fan, Stephen Chenney, Bo Hu, Kam-Wah Tsui and Yu-Chi Lai,“Optimum Control Variate”, Computer Graphics Forum, Vol. 25, No. 3, pp. 351-358, 2006. • Yu-Chi Lai, Shaohua Fan, Feng Liu, Brandom Smith, Stephen Chenney, Li Zhang and Charles Dyer,“Population Monte Carlo Sampler for Rendering”, Technical Report, Department of Computer Sciences, University of Wisconsin-Madison, 2009