Interruptible Rendering

Interruptible Rendering Master’s Project Presentation Cliff WoolleyUniversity of Virginia Joint work with: David Luebke * Benjamin Watson† Abhinav Dayal† Originally presented at:ACM SIGGRAPH 2003Symposium on Interactive 3D Graphics * University of Virginia † Northwestern University

Motivation Interruptible Rendering

Balance complexity with interactivity • Age-old tradeoff: detail vs. frame rate • Typical approach: • Maintain a high, constant frame rate • Create a mesh that can always be rendered in time Interruptible Rendering

Improve on traditional LOD • Is a high, constant frame rate good enough? • Is fidelity ever more important than frame rate? • How can this be decided at runtime while still guaranteeing interactivity? Interruptible Rendering

Error metric unification • Inspired by [Meruvia 2000] • Combine notions of temporal and spatial error (i.e. “lateness” vs. “coarseness”) • Use to drive progressive refinement Interruptible Rendering

Unified error metrics • With interruptible rendering, we address this key question: At what point does further refinement of the current frame become pointless? Interruptible Rendering

Methodology Interruptible Rendering

Methodology • Refine a stream of continuous LODs • Monitor input frequently • Minimize dynamic visual error Interruptible Rendering

Refine a stream of continuous LODs • Render progressive refinements on top of each other until “out of time” • Ensure that we can stop refining at any time and move on to the next frame Interruptible Rendering

Refinement • Three refinement schemes implemented: • Splatting (Tech sketch, SIGGRAPH 2002) • Progressive polygonal hulls • Progressive ray casting (Abhinav Dayal, I3D 2003) • Will work with any progressive rendering scheme Interruptible Rendering

Progressive hull refinement • Start with a very coarse rendering • Progressively render refinement stream over top of the coarse image Interruptible Rendering

Progressive hulls • Simplification method by [Sander et al. 2000] • Record a sequence of constrained edge collapses and play it back in reverse • Guarantees containment V3 V2 Vn V1 V4 Interruptible Rendering

Progressive hulls V4 V1 V2 V3 Vn Interruptible Rendering

Progressive hulls Interruptible Rendering

Monitor input frequently • Ideally: input monitored continuously • Realistically: check every x ms • Allows quick reaction when sudden changes in input occur • Allows system to be self-tuning Interruptible Rendering

Minimize dynamic visual error • Always display image with least error • Sometimes: • Further refinement is pointless • Temporal error (lateness) exceeds spatial error (coarseness) for the current frame • Front buffer “out of date” • Dynamic visual error (spatial + temporal) in front is greater than in the back Interruptible Rendering

Calculating dynamic visual error • Spatial error • Screen-space size of coarsest geometry visible • Temporal error • Project bounding box of object in screen-space • Find the maximum screen-space distance any corner has moved since frame began Interruptible Rendering

Start refining a new image toward most current input in the back buffer. clear back buffer Iback = Icurrent Clear the back buffer start start clear front buffer Ifront = Icurrent Clear the front buffer Refine the current image in the back buffer. improve iback tback = Icurrent – Iback sback = Iback – iback Refine the back buffer Refine the current image in the front buffer. improve ifront tfront = Icurrent – Ifront sfront = Ifront – ifront Refine the front buffer tback > sback? no yes time time tb > sb? N Y tfront = Icurrent – Ifront efront = sfront + tfront eback = sback + tback It is pointless to continue refining. swap buffers Ifront = Iback sfront = sback Compute dynamic visual error Swap buffers tfront > sfront? efront >= eback? no yes no yes tf > sf? N Y ef >= eb? N Y Back buffer now closer to Icurrent than front is. swap buffers Ifront = Iback Swap buffers Rendering to Front Buffer Rendering to Back Buffer Rendering to Front Buffer Rendering to Back Buffer 

Evaluation Interruptible Rendering

“Gold Standard” evaluation • Compares an ideal rendering to interactive approximations • Ideal rendering: full detail, zero delay • Interactive approximations are • Unmanaged • Constant fidelity (in pixels) • Constant frame rate (fixed Hz) • Interruptible rendering Interruptible Rendering

Frame generation • Record what the user sees and when • Generate each approximation offline • Record actual frames displayed over time • Account for: • Render time • Double buffering • Frame locking • Rendering into front buffer (interruptible) Interruptible Rendering

Comparing frames • Error measures • RMS – Root Mean Square error • Lindstrom's perceptually based error Difft = Idealt - Renderedt Interruptible Rendering

Two test input streams • Rotation • Model-centered • Fixed angular velocity • User interaction • Includes both view translation and rotation • Includes both static and dynamic segments • Both input streams recorded to files, with timestamps Interruptible Rendering

Interaction sequence: ray caster 42 45 54 57 Frames: 33 36 39 48 51 60 Ideal Inter-ruptible Constant Fidelity Un-managed

Video Interruptible Rendering

Rotation input stream Interactive input stream Progressive hulls error(rms) error(rms) time(secs) time(secs) Ray casting error(rms) error(rms) time(secs) time(secs) constant frame rate constant fidelity interruptible unmanaged

Demo Interruptible Rendering

Conclusions Interruptible Rendering

Benefits • Accuracy • Balancing spatial and temporal error • Interactivity • Slow renderers, like ray caster • Large models • Self-tuning feedback system • Adapts to hardware Interruptible Rendering

Limitations • Overdraw in progressive renderer • Progressive ray caster better here • Cost of monitoring temporal error • Rendering immersive models • Requires reversing the containment criteria for polygon rendering • Not a problem for ray caster Interruptible Rendering

Future work • Temporal & spatial error • Better estimates of each (e.g. color) • Better comparisons of both • Textures, normal maps, etc. • Reducing overdraw • View-dependent refinement • Ray tracing: a promising candidate Interruptible Rendering

Acknowledgements • Peter Lindstrom for ltdiff • OpenRT Interactive Raytracing Project • Stanford 3D Scanning Repository • National Science Foundation • Awards 0092973, 0093172, and 0112937 Interruptible Rendering

Questions? Interruptible Rendering

Progressive ray casting • Progressive refinement • Coarse to fine sampling/reconstruction • Sampling • Where to shoot the ray • Adaptive • Non-adaptive • Reconstruction • Producing imagery from samples Interruptible Rendering

Sampling • Coarse to fine - quadtree approach • Sample every quadtree node's center • Breadth first traversal • Predetermined random traversal per level • Ray casting – using OpenRT • Per ray API ~ 600,000 rays per sec. • Simple phong shading • No shadows or specular highlights Interruptible Rendering

Reconstruction • Placement • Place splat at center of each quadtree node • Shading • Flat shaded quad covering the node's screen space • Alpha-textured quads (smooth reconstruction) • Quad size = twice of node's screen space • Texture: Gaussian blob • Transparent at edges • Opaque at center Interruptible Rendering

Alpha textured Flat shaded Interruptible Rendering

Spatial error Calculating dynamic visual error • Temporal error • Computed as in case of polygonal hulls • Spatial error • Diagonal length of the largest quadtree node displayed on the screen Interruptible Rendering

Interruptible Rendering