State of the Art in Realtime Ray Tracing & Interactive Global Illumination

State of the Art inRealtime Ray Tracing&Interactive Global Illumination Ingo Wald Carsten Benthin Joerg Schmittler Philipp Slusallek Saarland University Timothy J. Purcell Stanford University

Agenda • Introduction & Motivation [Wald] • Part I – Interactive Ray Tracing Architectures [Both] • Ray tracing in SW: The RTRT/OpenRT engine [Wald] • Ray tracing on GPUs [Purcell] • Special purpose ray tracing hardware: SaarCOR [Purcell] • Part II – Advanced Ray Tracing Issues [Wald] • Handling dynamic scenes • The OpenRT interactive ray tracing API • Practical applications • Part III – Interactive Global Illumination [Wald] • Instant Global Illumination • Conclusion & Outlook into the Future [Both] Realtime Ray Tracing & Interactive Global Illumination

Introduction & Motivation Computer Graphics Today: • Interactive Graphics (Games etc):  Hardware Accelerated “Triangle Rasterization” • “paint each triangles onto the screen” • High performance hardware • But: Only simple models, approximations, only local effects, etc. • High-Quality Graphics (Movies, Design, Simulation etc): Software (offline) “Ray Tracing” • “trace rays from eye backwards into the scene” • Reverse physical light transport: Can model all effects • All effects, complex scenes, physically correct, etc. • But: Historically too costly for interactivity Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene • Find objects nearby ray (traverse special data structures) • Compute ray-object intersection tests • Determine closest hitpoint Grid (2D) Octree (2D) Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) • Can recursively shoot new rays (e.g. reflections, shadow tests, etc) Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Simple Algorithm 1.) Create ray from eye through pixel 2.) Trace ray into scene 3.) Compute color of ray (“shade” the ray) 4.) Display final image Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing Advantages: • Complexity is logarithmic in scene size • Ideally suited for complex models • Allows plug-n-play shading • Simple to use and extend • Allows for exact visibility queries • Physically correct shadows, reflections, global effects, ... • Very beneficial to have at interactive rates ! Realtime Ray Tracing & Interactive Global Illumination

Part IInteractive Ray Tracing

Realizing Realtime Ray Tracing Realtime Ray Tracing offers many advantages ! Question: How to realize Realtime Ray Tracing ? Today: Three options... • Faster software ray tracing • Supercomputers: Muuss et al, Utah [Parker et al.] • Clusters: Saarland OpenRT [Wald et al.], Utah [deMarle et al.] • Ray Tracing on programmable GPUs • Carr et al., Purcell et al. • Building special purpose hardware • Schmittler et al. Realtime Ray Tracing & Interactive Global Illumination

Software Ray TracingThe RTRT/OpenRT Interactive Ray Tracing System

The OpenRT Interactive Ray Tracing Engine The OpenRT Interactive Ray Tracing Engine: • Won’t go into technical details • No details on any kernel algorithms • See the STAR... • More important: Summarize “State of the Art” • State of the Art of OpenRT Features • State of the Art Performance Data • Today’s possibilities, problems and challenges Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing Engine Features of OpenRT: • Highly efficient RT kernels [EG’01,...] • Specially optimized for modern CPUs (caches, SSE, ...) • Recently much faster than EG’01 results... • Parallelization on PC cluster [EuroPar’03] • Up to 24 dual-Athlon MP 1800+ • Usually linear scalability up to 25 fps@640x480 • Can handle dynamic scenes [Part II] • OpenGL-like API [Part II] • Supports arbitrarily programmable Shaders • Supports interactive global illumination [Part III] Realtime Ray Tracing & Interactive Global Illumination

RTRT/OpenRT State of the Art Performance Important: Much higher performance than EG’01: • Obviously: CPUs got faster.... • 2.5GHz Pentium IV vs. 866MHz Pentium III: 3x faster • Plus: Better compiler support • Intel C Compiler: ~2x faster than gcc • ICC intrinsics: much better than hand-coded assembler • Plus: Better kd tree construction algorithms • Modified “surface area heuristic” algorithm [Havran] • Roughly 2x faster than originally used kd-trees • Plus: Faster implementation...  Taken together: Much faster then EG’01 Realtime Ray Tracing & Interactive Global Illumination

RTRT/OpenRT State of the Art Performance Data State of the art performance data • On single PentiumIV 2.5GHz • In Frames per second at 1024x1024 pixels  Up to 7 million rays per second on 2.5GHz laptop ! Realtime Ray Tracing & Interactive Global Illumination

RTRT/OpenRT PerformanceImportant Lessons • Raw ray tracing performance quite high • Up to 7M rays/sec on 2.5 GHz notebook • Corresponds to 7fps@1024x1024 • Still up to 4 million rays/sec on 1.5Mtri model • But: Hard to get peak performance • Shading cost, coherent rays, ... • But even ‘average’ performance quite high ! • Important: Today’s biggest bottleneck is shading ! • Even simple shading in ERW6: 7.1Mrps  2.3 Mrps • Future research: Have to look at faster shading ! • SSE, Packet shading, ... Realtime Ray Tracing & Interactive Global Illumination

Ray Tracing on GPUs&Special Purpose Ray Tracing Hardware – The SaarCOR Architecture -

Part IIAdvanced Issues

Advanced Ray Tracing Issues Part I: Different ways to realize realtime ray tracing • Can’t (yet) say which is best... • But can say: “It is coming” • Actually, it’s available already today... Realtime Ray Tracing & Interactive Global Illumination

Advanced Ray Tracing Issues Ray Tracing is coming  New issues need to be solved: • How to handle dynamic scenes ? • Need new algorithms • How can I use it ?  API ! • What new things can I do with it ? • Build and explore new applications • Important: Those issues have to be solved anyway... • No matter wether it’s GPU’s, CPU’s, or SaarCOR, ... • All these issues are equally valid for all architectures! Realtime Ray Tracing & Interactive Global Illumination

Advanced Ray Tracing Issues:Dynamic Scenes

Handling Dynamic Scenes • Ray tracing already beneficial in static scenes • Visualizing complex models • Complex shading • High quality walkthroughs (e.g. w/ global illumination) • .... But: “Static only” is a severe limitation • Cannot really interact with the model ! • Need support for dynamically changing scenes… Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes • Fact: Ray Tracing needs acceleration structure • Too costly to rebuild each frame • Problem: Few research on this topic so far… • Just wasn’t interesting before interactive ray tracing… • Previous work: Usually on special cases • Utah ‘Hack’: Keep dynamic objects out of accel structure… • [Reinhard RW2001]: Incremental updates of Uniform Grid • Costly, not hierarchical • [Moeller, EG2001]: Only rigid-body animation Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes • Different kinds of dynamic behavior • Hierarchical, rigid-body motion vs. unstructured motion • Constrained unstructured motion (e.g. maximum displacement) • All triangles animated vs. few triangles animated • Amortized over many or over few rays • … • Actually different problems • Need specialized solutions Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes Our approach: • Offer suite of different techniques • Hierarchical animation of whole objects • Fast reconstruction of objects for unstructured motion (with sacrifices in traversal speed) • High-quality BSPs for often-used static objects (with relatively long rebuild time) • Allow to combine them in a hierarchical way  API ! Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes • Application groups geometry into ‘objects’ • Similar to building display lists (API) • Each object has its own acceleration structure • Support Unstructured Motion • Use different BSPs for different objects • Cheap BSPs for unstructured motion • High-quality BSPs for objects w/ rigid-body motion • Allow localized, fast rebuild of unstructured motion • But: Unstructured motion still costly • Too much data to send over network !!! • Tolerable for moderately complex objects (16k-64ktri) Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes • Objects are ‘instantiated’ into the scene • Just like ‘calling’ a display list ( API) • Each instance has a transformation attached to it • Hierarchical Anim: Inversely transform rays instead of objects • Changing this transformation transforms whole object • Instances are organized in additional hierarchy level • With its own acceleration structure • Only this has to be rebuilt every frame • Side Effect: Instantiation is for free • Sunflowers: 26k instances of 36k-sunflower  1 billion triangles Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes- Results Realtime Ray Tracing & Interactive Global Illumination

Handling Dynamic Scenes- Results • TopLevel BSP reconstruction tolerable • Some milliseconds even for a few thousand objects • But definitely is not for free... • Hierarchical animation is cheap • Unstructured motion still a problem • But: Sufficient for most applications so far • Used exclusively in all RTRT/OpenRT applications • Future work: • Better support for “immediate mode” rendering • Port algorithms to SaarCOR architecture Realtime Ray Tracing & Interactive Global Illumination

Advanced Ray Tracing Issues:The OpenRT API

The need for an API • So far: All technical problems have been addressed • Fast ray tracing implementations • Shading, massive models, etc, are easy... • Dynamic scenes • “Theoretically” we’re done... • Questions: How can I use it ? • Need an API ! Realtime Ray Tracing & Interactive Global Illumination

The need for an API • So far: All technical problems have been addressed • Fast ray tracing implementations • Shading, massive models, etc, is easy... • Dynamic scenes • “Theoretically” we’re done... • Questions: How can I use it ? • Need an API ! • Problem: No APIs for interactive RT available • OpenGL not suitable for ray tracing • Renderman/Rayshade/Povray inherently offline … Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API Goals for an Interactive Ray Tracing API: • As easy to learn and use as (basic) OpenGL • Leverage existing OpenGL experience • As powerful in shading as RenderMan OpenRT: Combine the best of both • Application API much like OpenGL/GLUT • Plus necessary extensions for Ray Tracing • Shaders, Objects, ... • Shader API like RenderMan Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API • Application API very OpenGL-like • Geometry/Transformations just like OpenGL Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API • Application API very OpenGL-like • Geometry/Transformations just like OpenGL rtMatrixMode(RT_MODELVIEW); rtPushMatrix(); rtRotatef(1,0,0,M_PI/2); rtScalef(1,2,1); rtBegin(RT_TRIANGLES); rtColor3f(0,1,0) rtNormal3f(1,1,1); rtVertex3f(0,0,0); ... rtEnd(); rtPopMatrix(); Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API • Application API very OpenGL-like • Geometry/Transformations just like OpenGL • Geometry Objects • Just like Display Lists (except: no side effects) • rtNewObjects(), rtBegin/EndObject(), rtInstantiate(),… • Support fully programmable shader objects • Surface, Light, and Pixel Shaders, ‘Renderer Object’ • Difference to GL: Different Rendering Semantics • “Retained-Mode” vs. “Immediate Mode” • Changes take effect only at end of frame ! • No “state machine” model Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API Shader API • Shaders loaded from DLL’s/.so’s • Declare Shader • E.g. shader parameters etc • Write callback-functions • ‘Shade()’ (for surfaces), ‘Illuminate()’ (for lights), … • Access scene data with RenderMan like API • Geometry: rtsShadingNormal(), … • Lights: rtsIlluminate(), rtsOccluded(), ... • Can shoot arbitrary secondary rays • rtsTrace(…) Realtime Ray Tracing & Interactive Global Illumination

The OpenRT Interactive Ray Tracing API • OpenRT Summary • Abstracts from underlying engine • Completely hides kernel implementation • Hides distribution infrastructure • Low-level API rather than scene graph API • OpenGL like application interface • RenderMan like flexible shader API • State of the Art • OpenRT V1.0 is out... • Used exclusively to drive Saarland RTRT/OpenRT • Used for all shown applications (except GPUs and SaarCOR) • Ongoing work: SaarCOR@OpenRT Realtime Ray Tracing & Interactive Global Illumination

Advanced Ray Tracing Issues:Applications

Advanced Ray Tracing Issues: Applications • So far: All ingredients for a complete rendering engine are available ! • Technology (Fast ray tracing, dynamic scenes) • API to use it • Availability of a new technology opens up potential for new applications • Have to explore what’s possible and what not... • Many applications might not even have been thought of ! • Here: Only brief overview of today’s possibilities Realtime Ray Tracing & Interactive Global Illumination

Application I: Classical Ray Tracing • Classical Ray Tracing with Complex Shading Effects • Local: Volume object, Lightfield, bumpmapping, procedural shaders.... • Global Effects: Shadows, Reflections, ... • All effects work together (e.g. shadows from lightfield & volume obj.) “Office” w/ complex shading effects Realtime Ray Tracing & Interactive Global Illumination

Application I: Classical Ray Tracing • Classical Ray Tracing for physically correct simulation • Industrial project w/ Hella corp • Simulate reflection&refraction behavior of car headlight • Up to 25 levels of reflection/refraction, complex glass effects “Car Headlight” – Physically correct glass simulation Realtime Ray Tracing & Interactive Global Illumination

State of the Art in Realtime Ray Tracing & Interactive Global Illumination