Soccer Video Analysis

1. Soccer Video Analysis EE 368: Spring 2012 Kevin Cheng

2. Goal To detect and track key features needed to interpret events in a Soccer game from a video

3. Clip

4. Overview Frame Pre-Processing Input Image Field Detector Player Detector Ball Detector

5. Field Detector Field Color Map Creator Input Image Input Image Field detector is “trained” on a set of images with known RGB values, using a 16x16x16 color map Harris Corner Detector RGB Map Assignment Convex Hull of Largest Region Hough Transform Hough Mask Field Mask

6. Field Detector Field Color Map Creator Input Image Input Image For each frame, using the field color map, each pixel is labeled as “field” or “non-field” Harris Corner Detector RGB Map Assignment Convex Hull of Largest Region Hough Transform Hough Mask Field Mask

7. Field Detector Field Color Map Creator Input Image Input Image The largest region is considered where the field lies. A mask is created from the convex hull of the region. Harris Corner Detector RGB Map Assignment Convex Hull of Largest Region Hough Transform Hough Mask Field Mask

8. Field Detector Field Color Map Creator Input Image Input Image Using a canny edge detector and hough transform, we attempt to extract the field lines to help extract information of the field location Harris Corner Detector RGB Map Assignment Convex Hull of Largest Region Hough Transform Hough Mask Field Mask

9. Player Detector Team Color Decider Input Image Input Image • Using the first frame as a keyframe, we determine the mean RGB values of the 2 teams. • This is done by taking a histogram of the field, and finding the 2 highest bins that are NOT labeled as ‘field’ by the field map. Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

10. Player Detector Team Color Decider Input Image Input Image • To find potential player positions, we use a SIFT keypoint detector to narrow down the search regions Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

11. Player Detector Team Color Decider Input Image Input Image • Based on the RGB values around the SIFT keypoints, each keypoint is assigned to a team based on distance in RGB space. • Keypoints whose calculated distance is too large are thrown out. Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

12. Player Detector Team Color Decider Input Image Input Image • Since a single player can produce multiple SIFT keypoints, we implement a minimum distance connectivity map to remove redundant keypoints. Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

13. Connectivity Map • A connectivity map is created were the (i,j) entry is the pixel distance between keypointi, and keypoint j. • This matrix is square and symmetric

14. Connectivity Map • All entries larger than 40 are tagged as duplicates. • Here we see that keypoints (2,3) should be merged, and that (5,6) are duplicates.

15. Connectivity Map • Systematically, we remove duplicates starting from the highest connected keypoint.

16. Connectivity Map • Systematically, we remove duplicates starting from the highest connected keypoint.

17. Connectivity Map • The remaining keypoints are passed along as the player positions detected in the current frame

18. Player Tracking Team Color Decider Input Image Input Image • Since it cannot be assumed that we detect all players in every single frame, we implement a tracking algorithm. • This algorithm once again utilizes a connectivity map to match current detections with history element. Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

19. Player Tracking History Now • The (i,j) entry compares the pixel distance between the ith current detection and jth history element • Moving from left to right, history elements with their closest current keypointsin a one-to-one matching

20. Player Tracking Team Color Decider Input Image Input Image • For each player, we assign a counter that keeps track of its detection history: • If the history element is matched with a current element, we increment the counter by 2 (max of 15). • If a history element does NOT find a match in the current frame, we decrement the counter by 1 (minimum of 0). • Only players whose counter is greater than 5 are displayed. • If a player’s counter drops below 0, it is removed from the history Harris Corner Detector SIFT Keypoint Detector Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions

21. Ball Detector Input Image Input Image • The ball detector is based on corner detection since a small round object should have high “cornerness”. Harris Corner Detector Harris Corner Detector Hough and White Mask Gaussian Weighting Max Value Ball Position

22. Ball Detector Input Image Input Image • To filter out all other corners in the image, we only look at corners with white RGB values. • Another source of white corners in the image are the white field lines. So we also mask out corners that lie on the detected hough transform. Harris Corner Detector Harris Corner Detector Hough and White Mask Gaussian Weighting Max Value Ball Position

23. Ball Tracking Input Image Input Image • Lastly, for ball tracking, we weight the corner metric values by a gaussian centered at the last detected spot of the ball. • After the gaussian weighting, we take the maximum remaining value to be the new location of the ball. Harris Corner Detector Harris Corner Detector Hough and White Mask Gaussian Weighting Max Value Ball Position

24. Results: Player Detector • Strengths: • Has a > 90% detection and tracking rate of • Built in robustness to false-positives and false-negatives • Weaknesses: • A little bit jittery from frame to frame due to keypoint based detection. • Will falsely assign referees onto a team.

25. Results: Player Detector

26. Results: Ball Detector • Strengths: • Maintains track of ball very well • Weaknesses: • Requires to see the ball in every frame • Partial ball obscuring could cause tracker to latch onto another local maximum.

27. Results: Ball Detector

28. Timing Analysis • 2.63 seconds per frame. • Field Detector: 46.6% (1.15 s/f) • .27 s spent in applyMap • Likely due to poor MATLAB programming. • Canny edge and Hough transform also big contributors • Player Detector: 15% (.40 s/f) • SIFT detector: .28 s/f • Ball detector: 10.9% (.27 s/f)

29. Further Work • Field Detection: • Add ability to detect where in the field you are using field lines and features. • Player Detection and Tracking: • SIFT is a large computational bottleneck. Are there more viable alternatives? • Ball Detection and Tracking: • Improve handling of cases where ball is obscured for a short amount of time. • Analysis and interpretation of detected events.