Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture

1. Improving the Speed ofVirtual Rear Projection:A GPU-Centric Architecture Matthew Flagg, Jay Summet, James M. Rehg GVU Center College of Computing Georgia Institute of Technology

2. Ubiquitous Interactive Displays Every flat surface can be an interactive display 2

3. VRP: Shadow Elimination Single Projector Case 3

4. Shadow Elimination Half power shadows Double Projector Case Passive VRP 4

5. Shadow Elimination Proportional feedback law Boosting projector outputs 5

6. Occluder Light Suppression Detecting occluded pixels 6

7. Occluder Light Suppression Detecting occluded pixels 7

8. Occluder Light Suppression Detecting occluded pixels 8

9. Occluder Light Suppression Nonlinear feedback law Detecting occluded pixels 9

10. Virtual Rear Projection Show ICCV’03 demo video 10

11. 2 Challenges for VRP • High image quality • Seams between display regions projected by different projectors • Photometric Uniformity • Fast Compensation • Avoid perception of shadows caused by system lag • Image processing required to ensure high image quality 11

12. Limitations With Previous Work • Camera view of screen must be unobstructed • Requires reference image capture before occlusion • Cannot be co-located with projector • Shadows still perceptible • Shadow detection image processing performed on CPU 12

13. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 13

14. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 IR backlit camera image 14

15. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 Turn off occluded pixels 15

16. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 Occluder Light Suppression 16

17. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Shadow Eliminator • Blinding Light Suppressor • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 Turn on occluded pixels with second projector 17

18. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 Shadow Elimination 18

19. New Approach • Detect Occluders, Not Shadows • Co-locate projector with camera • Active IR imaging • Based on work by Tan and Pausch CHI’02 • Projector Roles: • Blinding Light Suppressor • Shadow Eliminator • Image Processing on GPU • Pixel Shaders • Render-To-Texture with DirectX9.0 Shadow Elimination and Occluder Light Suppression 19

20. Fast Compensation: GPU-Centric Approach 20

21. Fast Compensation: GPU-Centric Approach 1. Warping, background subtraction 2. Median filtering and dilation for inter-frame tolerance 3. Gaussian blur for blending 4. Compositing and warping 21

22. Pixel Shader Pipeline camera render display texture texture 1 image (A) (B) (C) background render back texture texture 2 buffer 22

23. Addressing Image Quality • Luminance Attenuation Maps (LAMs) • Simple feedback-based approach to accommodate non-linearities of projector-camera • Seam Blending LAM for left projector LAM for right projector seam – with blending seam – no blending 23

24. Virtual Rear Projection Results Play Video 24

25. Virtual Rear Projection Results • Image processing speed increased from 15Hz to 110Hz (camera capture rate), placing limit on the projector (85Hz refresh rate) • Projector latency accounts for 76% of total system latency! • With occluder movement tolerance of 5cm, shadows are imperceptible up to 94 cm/sec (fast walking) 25

26. Conclusion • Presented new approach to VRP • Occluder Detection in IR spectrum • All processing moved to GPU • 2 System Challenges Met • Display Image Quality • Shadow Perception Avoidance • Shadows eliminated fast enough to accommodate walking 26

27. Future Work • Explore hardware solutions • Recent results show an LCD projector having ½ the latency of a DLP and LCOS • User Study • VRP currently used in Collaborative Design Lab in School of Aerospace Engineering • Replicate laboratory evaluation of passive VRP with new active VRP system • Improve Image Quality • Better seam blending 27