Lightfields and Lumigraphs

1. Lightfields and Lumigraphs Vision for GraphicsCSE 590SS, Winter 2001Richard Szeliski (with lots of slides from Michael Cohen)

2. Modeling light • How do we generate new scenes and animations from existing ones? • Classic “3D Vision + Graphics”: • take (lots of) pictures • recover camera pose • build 3D model • extract texture maps / BRDFs • synthesize new views Vision for Graphics

3. Computer Graphics Output Image Model Synthetic Camera Vision for Graphics

4. Computer Vision Output Model Real Scene Real Cameras Vision for Graphics

5. Combined Output Image Real Scene Model Synthetic Camera Real Cameras Vision for Graphics

6. But, vision technology falls short Output Image Model Real Scene Synthetic Camera Real Cameras Vision for Graphics

7. … and so does graphics. Output Image Model Real Scene Synthetic Camera Real Cameras Vision for Graphics

8. Image Based Rendering Output Image Real Scene Synthetic Camera Images+Model Real Cameras -or- Expensive Image Synthesis Vision for Graphics

9. Ray • Constant radiance • time is fixed • 5D • 3D position • 2D direction Vision for Graphics

10. All Rays • Plenoptic Function: • all possible images • too much stuff! Vision for Graphics

11. Line • Infinite line • 4D • 2D direction • 2D position • non-dispersive medium Vision for Graphics

12. Ray • Discretize, then interpolate • Distance between 2 rays • Which is closer together? Vision for Graphics

13. Image • What is an image? • All rays through a point • Panorama? Vision for Graphics

14. Image • Image plane • 2D • position Vision for Graphics

15. Object • Light leaving towards “eye” • 2D • just dual of image Vision for Graphics

16. Object • All light leaving object Vision for Graphics

17. Object • 4D • 2D position • 2D direction Vision for Graphics

18. Object • All images Vision for Graphics

19. Lumigraph / Lightfield • Outside convex space • 4D Empty Stuff Vision for Graphics

20. Lumigraph • Inside convex space • 4D Stuff Empty Vision for Graphics

21. Lumigraph • How to • organize • capture • render Vision for Graphics

22. Lumigraph - Organization • 2D position • 2D direction s q Vision for Graphics

23. Lumigraph - Organization • 2D position • 2D position • 2 plane parameterization u s Vision for Graphics

24. s,t u,v s,t u,v Lumigraph - Organization • 2D position • 2D position • 2 plane parameterization t v u s Vision for Graphics

25. Lumigraph - Organization • Hold s,t constant • Let u,v vary • An image s,t u,v Vision for Graphics

26. Lumigraph - Organization • Discretization • higher res near object • if diffuse • captures texture • lower res away • captures directions s,t u,v Vision for Graphics

27. Lumigraph - Capture • Idea 1 • Move camera carefully over s,t plane • Gantry • see Lightfield paper s,t u,v Vision for Graphics

28. Lumigraph - Capture • Idea 2 • Move camera anywhere • Rebinning • see Lumigraph paper s,t u,v Vision for Graphics

29. Lumigraph - Rendering • For each output pixel • determine s,t,u,v • either • find closest discrete RGB • interpolate near values s,t u,v Vision for Graphics

30. s u Lumigraph - Rendering • For each output pixel • determine s,t,u,v • either • use closest discrete RGB • interpolate near values Vision for Graphics

31. s u Lumigraph - Rendering • Nearest • closest s • closest u • draw it • Blend 16 nearest • quadrilinear interpolation Vision for Graphics

32. s u Lumigraph - Rendering • Depth Correction • closest s • intersection with “object” • best u • closest u Vision for Graphics

33. s u Lumigraph - Rendering • Depth Correction • quadralinear interpolation • new “closest” • like focus Vision for Graphics

34. s u Lumigraph - Rendering • Fast s,t,u,v finding • scanline interpolate • texture mapping • shear warp Vision for Graphics

35. Lumigraph - Ray Space Vision for Graphics

36. Lumigraph - Ray Space • Image effects: • parallax • occlusion • transparency • highlights Vision for Graphics

37. Lumigraph - Demo • Lumigraph • Lion, Fruit Bowl, Visible Woman, Path Tracing Vision for Graphics

38. Lightfield - Demo • Digital Michelangelo Project • Marc Levoy, Stanford University • Lightfield (“night”) assembled by Jon Shade Vision for Graphics

39. Surface Lightfields • Turn 4D parameterization around • Leverage coherence Vision for Graphics

40. Surface Lightfields • Wood et al, SIGGRAPH 2000 Vision for Graphics

41. 3D Representations • Image is 2D • Lumigraph is 4D • What happened to 3D? • 3D Lumigraph subset • Concentric mosaics Vision for Graphics

42. 3D Lumigraph • One row of s,t plane • i.e., hold t constant s,t u,v Vision for Graphics

43. 3D Lumigraph • One row of s,t plane • i.e., hold t constant • thus s,u,v • a “row of images” s u,v Vision for Graphics

44. Concentric Mosaics • Replace “row” with “circle” of images Vision for Graphics

45. Concentric Mosaics Vision for Graphics

46. Concentric Mosaics • From above Vision for Graphics

47. Concentric Mosaics Depth correction Vision for Graphics

48. Concentric Mosaics • Panorama Vision for Graphics

49. 2.5D Representations • Image is 2D • Lumigraph is 4D • 3D • 3D Lumigraph subset • Concentric mosaics • 2.5D • Layered Depth Images • View Dependent Surfaces Vision for Graphics

50. 2.5 D ? Layered Image Depth Layered Depth Image Vision for Graphics