BACK ⇚ to the FUTURE

1. BACK⇚ to the FUTURE Image Based Rendering in 3d Games Daniel Horn and Billy Chen

2. In the past… Welcome to early 199x • No consumer graphics hardware acceleration • 3d Texture mapping too expensive • Demand for immersive 3d environments

3. Early Solutions • Image based rendering Wing Commander (1990) Wolf3d(1992), Doom (1993)

4. Early Solutions • Image based rendering Wing Commander (1990) Wolf3d(1992), Doom (1993)

5. Wing Commander: Light Fields • The actual datafiles contain a light field

6. Wing Commander: Light Fields • Simplified rendering algorithm • All pixels come from the same sub-image per object Select Subimage Rotate / Scale Rotate / Scale Select Subimage

7. Wing Commander: Light Fields (Demo) Select Subimage Rotate / Scale Rotate / Scale Select Subimage

8. Comparative Triangle-based app • Xwing (1993): Flat shaded untextured tri’s • Tie Fighter (1994) Gouraud Shading • Versus Wing Commander (1990): Sprites

9. Glory Days of 3d Games (1996-Now) • Quake WC Prophecy • Vega Strike Doom 3

10. So, What Changed Recently? • More Video RAM • 512 Megs Onboard Ram • PCI Express Access to Main Memory • Fragment Shaders • Can program access patterns to 4d lightfields • Builtin Texture Compression • Fast, random access, fixed rate

11. Why Go Back? This is now, now • More options • Can become compatible with triangle rendering • Captures living creatures well • Captures highly detailed geometry well • Better antialiasing for long range viewing • Mipmaps = LOD

12. Proof Of Concept • Getting a ship lightfield into Vega Strike • Step 1 Acqusition • Step 2 Parameterization/Packing • Step 3 Integration of Light Shop into Vega Strike

13. Acquiring a spherical light field  

14. Quasi-uniform sampling • uniform sampling on sphere for camera positions yields uniform sampling of ray space • sampling uniformly in  and  results in non-uniform sampling on sphere surface • Several techniques use platonic solids to form quasi-uniform sampling on surface of the sphere [Camahort98, Yershova04]

15. Quasi-uniform sampling • enforce uniform spacing in a ring • 31 rings (4 degree increment in ) • variable number of samples in 

16. Sampling density • d = fhsin / (R – hcos) • Let p = pixel widththen: • d  p • fhsin  p(R – cos) •  p(R-h) / hf p = 0.53mm f = 125mm h = 178mm R = 1000mm •  1.12 degrees 90 hours! f d  h R

17. Acquired images • 2418 images, 256 x 256 • 2 x 2 image tiles • coaxial light field: illumination fixed to camera • background light field captured for matte extraction

18. Introducing Light Shop • A system for manipulating and rendering light fields • 3 components • Modeling interface • Ray-shading program • Renderer • Provides a uniformed framework for light field manipulation: deforming, focusing, refracting, shadowing, etc. • Applications: photography, compositing, interactive games • Already supports two-plane parameterization, add spherical parameterization

19. Rendering spherical light fields (, , s, t) [Chen05] [Shum99]

20. How Vega Strike Draws 3d Models • Models are stored as many meshes • Each mesh has but 1 texture • When a model X at location L is drawn • L is pushed onto each of X’s meshes’ draw queues • To Render • Meshes are sorted by texture • For each mesh M • Call MakeActive() on M’s texture • For each location L in M’s draw queue • Load L Matrix to OpenGL • Draw M

21. Integration with Vega Strike • Subclass Texture with Texture4D • Ship model is a single axis-aligned quad • On Texture4D::MakeActive • Bind 3d texture data • Bind Vertex Program to place quad vertices at edges of light field bounding box • Bind Light Shop Fragment Program to render light fields

22. How Fragment Program Functions • Fragment Program operates per pixel • use screen coordinates and OpenGL Projection and ModelView matrices to determine a query-ray • Use query-ray and parameterization obtain lightfield color

23. DEMO

24. Future: Integration with OpenGL • Ideally light fields and geometry • Should both be shaded in the same way • Should be able to interact • Should complement each other • Creatures and objects with high detail • Light Fields • Man Made Objects • Geometry

25. Relighting Lightfields (A) • Capture many light fields per object • One per lighting condition • Given OpenGL lights, pick interpolation of captured light fields to accurately depict lighting condition • LightShop Demo

26. Relighting Lightfields (B) • Capture Diffuse Light Field • Very diffuse ambient light • Capture Normals • Proxy geometric object • Range Scanning • Use OpenGL Shading • Shade pixels based on OpenGL lights and normals

27. Shadows • Light fields capture self-shadowing • Want to apply shadows from occluders • Other objects blocking light sources • Use proxy geometry to • Query shadow map • or Shoot rays to light • Currently we use a plane as proxy

28. Lightfields as textures • We have heightmaps, bumpmaps • Why not use lightfields as a texture for geometry • Directly competes with parallax occlusion mapping…. • But much faster—at the cost of video RAM • Just use 4d coordinates to lookup a value • An parallax occlusion map “cache”

29. Animated Lightfields • Graphics cards have fast DivX decompression • Harness this to stream over light fields • Light fields should have really good mpeg compression characteristics • Can use for animated characters • Actual walking, talking humans in 3d games • Famous actors could walk and pose to have their characters inserted into games