Quadric Transfer for Immersive Curved Screen Displays

1. Quadric Transfer for Immersive Curved Screen Displays Ramesh Raskar, Jeroen van Bar, Thomas Willwacher, Srinivas Rao Mitsubishi Electric Research Labs Cambridge, MA USA

2. Curved Screen Displays Multiple overlapping projectors Goal : • Replace single-proj • Higher resoltn • Higher brightness • Sub-pixel auto-alignment • Parametric solution Planetarium Simulators

3. Dome Projection Edge-Blended Display Spherical Warping Sub-Frames

5. OutlineSeamless Curved Display • Multi-projector methods • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

6. OutlineSeamless Curved Display • Multi-projector methods • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

7. Multi-Projector Displays Precise config = Costly setup Manual alignment = High maintenance

8. Multi-Projector Displays

9. Multi-Projector Displays

10. Multi-Projector Displays

11. Camera 1 2 2 1 3 1 4 4 2 3 3 4 1 2 10 Seconds 3 4 Planar Displaywith parametric approach R Raskar

12. Planar projective transfer: homography • Two images of 3D points on a plane • Related by a 3x3 matrix ~ x’ = A3 x 3 x M x A3 x 3 x’ x’ x Proj 1 Proj 2 Proj 1 Proj 2

13. X x’ x Parametric Image Transfer X x x’ Planar Homography Quadric Transfer

14. Parametric Approach • Calibration • Lower camera resolution • Tolerance for pixel localization errors • Faster calibration • Rendering • Efficient well-defined warping • Avoids look up tables

15. Multi-Projector Displays

16. Curved ScreensView for a Sweet-spot Projector Sweet spot

17. Calibration for a Sweet-spot Projector Camera at Sweet spot

18. Discretized non-parametric approach Projector Image p1 p6 c1 c6 Camera Image Desired Image =

19. Pre-Warped Projection Discretized Warping Software Spitz - PolyDome™

20. OutlineSeamless Curved Display • Multi-projector methods • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

21. Degenerate ruled quadrics: cone two planes Quadric classification Projectively equivalent to sphere: sphere ellipsoid paraboloid hyperboloid Ruled quadrics: hyperboloids of one sheet

22. Quadrics 4x4 symmetric matrix, Nine d.o.f X Q 9 points in 3D define quadric

23. Quadric Image Transfer [Shashua97] X If , x’ x 21 params, 4 more than necessary !

24. Simplified Quadric Image Transfer Based on.. X Homography with polar plane Projected conic x’ x 17 param warp

25. Simplified Quadric Image Transfer X 17 param warp x’ x Planar homography: 4+ corresponding pixels Quadric transfer: 9+ corresponding pixels

26. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

27. Calibration of Quadric Screens 1 Dome Screen 2 4 3

28. Approach Calibration • At each projector i , • Project structured pattern • View with stereo camera • Finding camera to projector quadric transfer, Run-time • At each projector i, • Pre-warp input image using

29. CalibrationFinding relationship between camera and projector Low-res Camera 640x480 images But each Projector 1024x768

30. Before Blending

31. After Blending

32. Intensity Correction in Overlap Projector Framebuffers

33. Projector Framebuffers

34. Projector Framebuffer Intensity Weights

35. OutlineSeamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Calibration with camera-pair • Parametric Rendering solution

36. Rendering a 3D Scene Steps at each projector (Pre-distort vertex 3D location) • For each triangle T with vertices {Mj} • For each vertex M • Find pixel m via VirtualViewProjection( M ) • Find warped pixel m’via quadricTransferof m • Replace M with m’

37. Vertex Shader for Quadric Transfer in Cg vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) { vertout OUT; float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f ); float4 m, mi ; float3 m2,mp; float scale; m = mul( modelViewProj, m1); m2.x = m.x/m.w; m2.y = m.y/m.w; m2.z = 1; scale = mul(m2, mul(E,m2)); mp = mul(A,m2) + sqrt(scale)*e; mi.x = m.w * (mp.x)/(mp.z); mi.y = m.w * (mp.y)/(mp.z); mi.zw = m.zw; OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified return OUT; } ParametricWarp

38. 3D Terrain Rendering

39. Parametric Rendering Benefits • Head tracking support • Update quadric transfer per frame • Single pass rendering • Avoid post-rasterized warp • Efficient rendering • Better image quality • Programmable hardware • Distributed rendering • Runs unmodified 3D applications

40. Head Tracked Single Pass Rendering

41. Distributed Rendering with Unmodified Application

42. Subpixel Accurate Registration

43. Convex Dome

44. Acknowledgements • Mitsubishi Electric Research Labs • Paul Beardsley, Jay Thornton • Joe Marks • Mitsubishi Electric, Japan • Masato Ogata, Hiroyuki Wada • Masatoshi Kameyama, Ashizaki

45. Seamless Curved Display • Multi-projector low cost method • Simplified Quadric Image Transfer • Complete Parametric calib+rendering solution • Head tracking support, single pass rendering www.MERL.com/Projects/Projector/

46. Details I Skipped .. • Photometric Correction [Majumder03] • Depth of field is limited • Estimating camera and projector params • Internal and External params • Issue with near-planar 3D points • Finding pixels weights for blending • Non-linear optimization • Rendering • Warping and Depth buffer issues

47. Advantages • Parametric warp • Lower camera resolution • Tolerance for pixel localization errors • Faster calibration • Efficient well-defined warping

48. What is homography ? ~ j = A3 x 3 i A3 x 3 a1 a2 a3 b1 b2 b3 c1 c2 c3 jx jy 1 ix iy 1 = k j i ~ jx = (a •i)/ (c •i) Proj 1 Proj 2 ~ jy = (b •i)/ (c •i)

49. Rendering 2D + 3D scene Concave Dome Convex Dome

50. Projector Mailing List majordomo@cs.unc.edusubscribe projector Projector bibliography www.raskar.com/Projector/