Interactive Rendering using the Render Cache

1. Interactive Rendering using the Render Cache Bruce Walter, George Drettakis iMAGIS*-GRAVIR/IMAG-INRIA Steven Parker University of Utah *iMAGIS is a joint project of CNRS/IMAG/INRIA and UJF

2. Motivation • Goal: Interactive rendering Ray tracing Path tracing

3. Motivation • High-quality renderers • Pixel based • Ray tracing, path tracing, etc. • Wide range of lighting effects • Reflection, refraction, global illumination, etc. • Too slow for interactive use • Many seconds per image • Often used only for final images • Alternate renderers used for interactive editing

4. Motivation • Interactive rendering • Rapid feedback is paramount • High accuracy is less important • Fast consistent framerate • Eg, > 5 fps • Could use faster renderer • Eg, hardware accelerated scan conversion • Use same renderer • Need to bridge framerate gap

5. image renderer user application Visual Feedback Loop • Standard visual feedback loop • Entirely synchronous • Framerate is limited by the renderer

6. image renderer display user application Visual Feedback Loop • Modified visual feedback loop Asynchronous interface

7. Goals • Independent display process • Works with many (pixel-based) renders • Exploit frame to frame coherence • Reproject pixels from previous frames • Fast consistent framerate • Use simple, fast methods • Concentrate rendering effort • Prioritize pixel’s need for recomputation

8. Demo

9. Previous Work • Approximate or progressive approaches • Progressive radiosity or ray tracing • Frameless rendering • Reprojection or warping • Image based rendering (IBR) • Ray tracing acceleration

10. Previous Work • Parallel processing • Multiprocessors or distributed clusters • Intelligent display processes • Post-rendering warp • Holodeck system

11. Algorithm Overview Displayprocess renderer project Render cache depthcull image interpolate renderer sampling

12. Image Estimation • Projection • Project cached points onto current image • Camera transform provided by application • Z-buffer • In case multiple points map to a pixel

13. Image Estimation • Problem: visual artifacts Original view New view

14. Image Estimation • Depth cull heuristic • Problem: occluded points may be visible • Z-buffering only works within a pixel • Find pixels with locally inconsistent depths • Likely to be from different occluding surfaces Raw projection After depth cull

15. Image Estimation • Interpolation / smoothing • Problem: small gaps in point data • Interpolate pixel colors • Compute locally-weighted average colors • Currently uses 3x3 neighborhoods Raw projection After depth cull After interpolation

16. Image Estimation • Results after each stage Projection Depth cull Interpolation

17. Image Estimation • Problem: visual artifacts • Simple filter can remove many artifacts • Need new points from renderer • Previously invisible areas • Color changes due to non-diffuse shading • Eg,specular highlights • Changes due to user editing • Changes in lighting, geometry, materials

18. Sampling • Generate priority image • Based on pixel’s need for re-rendering • Priority given to pixels with older points • Render cache stores an age with each point • Empty pixels priority based on local density • Highest priority given to regions without points

19. Sampling • Choose pixels for rendering • Sampling must be sparse • Relatively few pixels are rendered per frame • Chosen using error-diffusion dither • Concentrates pixels in high priority regions • Maintains good spatial distribution • Requested pixels sent to renderer(s) • Results returned at some later frame

20. Sampling Displayed image Priority image Requested pixels

21. Optimizations • Further prioritizing sampling • Identify points that are likely to be outdated • Color change heuristic • Renderer supplied hints • Prematurely age these points • Forces sooner resampling of these points

22. Optimizations • Moving objects • Application can supply object transforms • Applied to points in the render cache • Improves tracking of moving objects • Points also aged to encourage resampling

23. Demo

24. Results • Timing: 70.5 ms or 14 fps • 256x256 image, display process only • 195 Mhz R10K processor

25. Conclusions • Greatly improved interactivity • Eg, ray tracing, path tracing • Efficient reuse of rendered pixels • Using reprojection and simple filters • Prioritized sparse sampling • Efficently uses limited rendering budget • Independent automatic display process • Can be used with many different renderers

26. Future Work • Larger images • Cost scales linearly with # of pixels • Higher render ratios • Currently work well out to about 1:64 • Anti-aliasing

27. The End

28. Overview • Cache rendered results • Stored as colored points in 3D • Estimate current image • Project points onto current image plane • Filter to reduce artifacts • Prioritize future rendering • Identify problem pixels • Sparse sampling for limited render budget