Motion Segmentation at Any Speed

1. Motion Segmentation at Any Speed Shrinivas J. Pundlik Department of Electrical and Computer Engineering, Clemson University, Clemson, SC

2. Gestalt Theory and Visual Perception • The human visual system: • Focus on well organized patterns rather than disparate parts • “Grouping” - the idea behind visual perception • Factors affecting the grouping process: proximity, similarity, closure, smoothness, symmetry, common fate and so on.

3. Gestalt Laws of Grouping Proximity Continuity Common Fate • Basis of many image and video segmentation algorithms • Work well in combinations. For example, proximity and similarity • Motion segmentation – grouping or aggregating entities with common fate

4. Applications of Motion Segmentation • Object detection and tracking • Surveillance • Robot Motion • Image and Video Compression • Video Editing/Motion Magnification • Shape Recovery

5. Existing Approaches Extraction of Motion Layers Wang and Adelson 1994, Weiss 1996 Ayer and Sawhney 1995,Xiao and Shah 2005 Ke and Kanade 2002 Detecting Motion Discontinuities Black and Fleet 1999, Birchfield 1998 Feature Point Grouping Beymer 1997, Fua 2003 Kanhere et. al. 2005 Normalized Cuts Shi and Malik 1998

6. Preview

7. Existing approaches Consider 2 frames or a spatio-temporal volume Threshold on velocities τ> (Δx/ Δt) Proposed approach An incremental approach Threshold on position Waits till enough evidence accumulates before segmenting Incremental Motion Segmentation

8. Different Approaches To Segmentation Segmentation/Grouping Agglomerative Divisive Start with single point and grow the group. Region growing or Region merging Start with entire data and split into clusters. Clustering or partitioning last step first step last step first step seed 2 seed 1

9. Representation of Motion • Why use feature points instead of optic flow? • Reduced time and complexity of computation • Reliable and repeatable • Well suited for tracking over long sequences

10. Feature Tracking Idea behind feature tracking: minimize the dissimilarity between two feature windows in the successive frames Good Features: Small image regions having high intensity variation in more than one direction Affine Consistency Check:

11. Feature Clustering • Clustering Data: Feature displacement over multiple frames • K-means clustering by fitting lines • Works better than clustering points

12. Results of Feature Clustering by K-means • Limitation: Clustering not accurate for more challenging sequences

13. Affine Partitioning • Requires prior initialization and number of groups to be found • Processing only on feature motion between two frames

14. Normalized Cuts Graph: G(V,E) Partitions: A,B Weight of an edge: w Affinity Matrix: Feature motion between two frames

15. Region Growing • Process over two frames • Select seed point • Fit affine model to neighbors • Repeat until the group does not change: • Discard all features except the one near the centroid • Grow group by including neighboring features with similar motion till it grows no further • Update the affine model

16. Finding Neighbors • Traditional way: Spatial window • Makes the algorithm sensitive to the feature locations • Alternative: Delaunay Traingulation • Simple and efficient technique

17. Finding Consistent groups • Parameters affecting region growing: grouping threshold, choice of frames and seed point • Different choice of seed points produce different grouping results • Features grouped together irrespective of the choice of seed points are consistent feature groups

18. Consistent Groups

19. Maintaining Groups Over Time • Finding new feature groups • Segmenting new objects entering the scene • Splitting existing feature groups • Split when configuration of a group changes over time • Adding new features to existing feature groups • Include new scene information over time

20. Results Segmentation results for the statue sequence

21. Results Frames 3, 4, 5, 6 of the statue sequence with threshold = 0.7 Frames 8, 64, 188, 395,6 of the fast statue sequence (generated by dropping every alternate frame)

22. Results Segmentation results for different sequences

23. Conclusions • Segmentation based on the availability of evidence • An incremental approach – able to handle long sequences