[ cvPONG ]

1. [cvPONG] A 3-D Pong Game Controlled Using Computer Vision Techniques Quan Yu and Chris Wagner

2. [Project Idea] • <> 3-D adaptation of the classic Pong game • <> Score by bouncing the ball past the opponent’s paddles • <> Avoid letting the ball go past your paddles • <> Calculate the user’s paddle positions based on the positions of the user’s hands and head • <> Total of 3 Paddles • <> Implementation: • <> Unity3D + EmguCV (a OpenCV wrapper)

3. skin detection paddle camera user

4. [Methods Used] • <> Background Subtraction - Chris • <> Differentiate user from background • <> Skin Detection – Chris, Quan • <> Find skin pixels in image • <> Erosion & Dilation - Chris • <> Clean image • <> Finding Components - Quan • <> Find head and hands

5. [Background Subtraction] • <> Use background subtraction to obtain mask region for skin detection • 1. Background Subtraction 1 • <> Compare object pixels’ intensities in capture frame with those of previously captured background image • <> How do we choose a good T?

6. [Background Subtraction] • <> Use background subtraction to obtain mask region for skin detection • 2. Mixture of Gaussians • <> Each pixel modeled by a mixture of K Gaussian distributions • <> Different Gaussians represent different colors • <> Mixture weights determined by time proportions that colors stay in scene • <> Learns probable background colors are the ones which stay longer and are more static • <> Implemented in OpenCV: BackgroundSubtractorMOG()

7. [Background Subtraction] <> Use background subtraction to obtain mask region for skin detection 2. Mixture of Gaussians An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection P. KaewTraKulPong and R. Bowden

8. [Skin Detection] • <> Apply skin detection in masked area to locate user’s hands and head • <> Elliptical Boundary Model • <> Convert RGB color space to normalized 2-D chrominance space by eliminating intensity (handles varying illumination conditions) • <> Most non-skin chrominances concentrate on a single point (gray point) • <> Want skin chrominance classification distribution to avoid overlapping this point (false positives) • <> Skin chrominance distribution fits nicely into a skewed normal distribution towards the gray point

9. [Skin Detection] <> Apply skin detection in masked area to locate user’s hands and head <> Elliptical Boundary Model Chrominance Distribution Single Gaussian Model Elliptical Boundary An Elliptical Boundary Model for Skin Color Detection Jae Y. Lee and Suk I. Yoo

10. [Finding Components] • <> Skin detection gives binary image in which we can find three largest components (2 hands, head) • 1. Erode then dilate image • <> Removes small components and noise • 2. Calculate contour boundaries of remaining components • <> Use OpenCV method, findContours • 3. Calculate bounding box around three largest contours • 4. Take center of bounding boxes as position of user’s paddles

11. [Demo]

12. [Problems] • <> Trade-off between accuracy and performance • <> Shadows/Illumination • <> Glasses! • <> Similarly colored objects in foreground • <> Speed – 8 fps in debug mode • <> Erosion/Dilation removes too many skin pixels • <> Low-fidelity between frames • <> EmguCV is not completely compatible with Unity3D • <> …

13. [Outcome and Lessons Learned] <> A controlled environment is better than a good algorithm in skin detection <> Vision-based input is a more fun experience than the keyboard/mouse <> Speed/interactivity is important in video games <> Simple background subtraction does not work well <> Simple skin detection works pretty good <> Noise is hard to remove

14. [Future Work] <> Incorporate MoG background subtraction to prune skin detection areas <> Utilize shape analysis/face detection to separate hands/head <> Smoother paddle movement between frames <> GPU programming to improve performance <> Add more game elements

15. [Thank you!] Questions, comments?