CSCE 641 Computer Graphics: Image-based Rendering (cont.)

1. CSCE 641 Computer Graphics: Image-based Rendering (cont.) Jinxiang Chai

2. Outline • Light field rendering • Plenoptic sampling (light field sampling) • 3D light field (concentric mosaics) • Others

3. Review: Plenoptic Sampling • Q: How many images are needed for • anti-aliased light field rendering?

4. Review: Plenoptic Sampling • Q: How many images are needed for • anti-aliased light field rendering? • A: formulate this as high-dimensional signal (4D Plenoptic function) reconstruction and sampling problem

5. Review: Between Two Planes v Z t Z2 t v Z1 Z1 Z2

6. Review: Minimal Sampling Rate Image resolution 1/∆v Sample interval 1/∆t

7. Review: Minimal Sampling Rate Ωv 1/∆Tmax>=Ωv*(f/zmin-f/zmax)

8. Review: Minimal Sampling Rate • Minimal sampling rate depends on: • - texture of object (Ωv) • - focal length (f) • - depth complexity (zmin, zmax) 1/∆Tmax>=Ωv*(f/zmin-f/zmax)

9. 3D Plenoptic Function • Image/panorama is 2D • Light field/lumigraph is 4D • What happens to 3D?

10. 3D Plenoptic Function • Image/panorama is 2D • Light field/lumigraph is 4D • What happens to 3D? • - 3D light field subset • - Concentric mosaic [Siggraph99]

11. 3D light field • One row of s,t plane • i.e., hold t constant s,t u,v

12. 3D light field • One row of s,t plane • i.e., hold t constant • thus s,u,v • a “row of images” s u,v

13. Concentric mosaics [Shum and He] Polar coordinate system: - hold r constant - thus (θ,u,v)

14. Concentric mosaics Why concentric mosaic? - easy to capture - relatively small in storage size - inside looking out

15. Concentric mosaics From above How to capture images?

16. Concentric mosaics From above How to capture images?

17. Concentric mosaics From above How to render a new image?

18. Concentric mosaics From above How to render a new image?

19. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

20. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

21. Concentric mosaics From above How to render a new image? - for each ray, retrieval the closest captured rays

22. Concentric mosaics From above How about this ray? How to render a new image? - for each ray, retrieval the closest captured rays

23. Concentric mosaics From above object How to retrieve the closest rays?

24. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

25. Concentric mosaics From above object (s,t) interpolation plane What’s the optimal interpolation radius? How to retrieve the closest rays?

26. Concentric mosaics From above object (s,t) interpolation plane What’s the optimal interpolation radius? 2rminrmax/(rmin+rmax) How to retrieve the closest rays?

27. Concentric mosaics From above object (s,t) interpolation plane How to retrieve the closest rays?

28. Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

29. Concentric mosaics From above object (s,t) interpolation plane How to retrieval the closest rays?

30. Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays?

31. Concentric mosaics From above object (s,t) interpolation plane How to synthesize the color of rays? - bilinear interpolation

32. Concentric mosaics From above

33. Concentric mosaics From above

34. Concentric mosaics From above

35. Concentric mosaics • What are limitations?

36. Concentric mosaics • What are limitations? • - limited rendering region? • - large vertical distortion

37. Concentric mosaics • What are limitations? • - limited rendering region? • - large vertical distortion

38. 2.5 D representation • Image is 2D • Light field/lumigraph is 4D • 3D • - a subset of light field • - concentric mosaics • 2.5D • - layered depth image [Shade et al, SIGGRAPH98] • - view-dependent surfaces

39. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - image with depths

40. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: - rays with colors and depths

41. Layered depth image [Shade et al, SIGGRAPH98] Layered depth image: (r,g,b,depth) - image with depths - rays with colors and depths

42. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image

43. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image • - Incremental in X and Y • Guaranteed to be in back-to-front order • - Forward warping one pixel with depth

44. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image • - Incremental in X and Y • Guaranteed to be in back-to-front order • - Forward warping one pixel with depth

45. Layered depth image [Shade et al, SIGGRAPH98] Rendering from layered depth image How to deal with occlusion/visibility problem? • - Incremental in X and Y • Guaranteed to be in back-to-front order • - Forward warping one pixel with depth

46. How to form LDIs • Synthetic world with known geometry and texture • - from multiple depth images • - modified ray tracer • Real images • - reconstruct geometry from multiple images (e.g., • voxel coloring, stereo reconstruction) • - form LDIs using multiple images and • reconstructed geometry

47. 2.5 D representation • Image is 2D • Light field/lumigraph is 4D • 3D • - a subset of light field • - concentric mosaics • 2.5D • - layered depth image [Shade et al, SIGGRAPH98] • - view-dependent surfaces

48. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

49. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture

50. View-dependent surface representation From multiple input image - reconstruct the geometry - view-dependent texture