Motion Segmentation over Image Sequences Using Multiway Cuts and Affine Transformations

1. Motion Segmentation over Image Sequences Using Multiway Cuts and Affine Transformations Braga Natarajan

2. Organization Motion Segmentation Motion models, affine transformation Energy minimization, graph cuts, multiway cuts Combining multiway cuts and affine transformations Algorithms and results

3. Motion segmentation • Image segmentation – by color, texture, shape, and motion • Motion – very important cue • Divide image into regions – exhibiting relatively different coherent motion • Ideal motion segmentation

4. Why motion segmentation? • Number of applications • Robotics • Video coding/compression • Video indexing/retrieval • Object tracking, surveillance • Intermediate image processing task – output given to high level computer vision problems

5. Motion representation • Pixel motion represented by a 2D vectorEither dense(optical flow) or sparse(features) • Models: translation: 2 parameters affine: 6 parameters

6. Lucas-Kanade affine estimation • Minimize nonlinear equation: • Iterative linear solution (Newton’s method): 2x2 matrix 2x1 vector current image residue pixel location next image unknown parameters (A, d), 6x1 vector 6x6 gradient matrix 6x1 error vector [Lucas & Kanade, 1981; Shi & Tomasi, 1994]

8. Energy minimization • Motion segmentation as energy minimization • challenge: Thousands of dimensions! • Solution: Graph cut methods [Boykov, Veksler, and Zabih 1999] • accurate • fast Smoothness penalties Data penalties

9. Image Correspondence • Motion segmentation comes under the general category of image correspondence • Goal of image correspondence: assign labels to every pixel in the image • Energy functions can be devised once labels are defined and listed • What do labels mean?

10. Stereo Correspondence

11. Motion

12. Binary cut • Maximum flow-minimum cut, Ford-Fulkerson 1956 • graphs constructed - a node per pixel • source and sink terminals; binary variables 0 and 1 • t-link weight: data penalty, n-link weight: smoothness penalty • Minimum s-t cut, pixels get label 0 or 1 based on what links are cut t-link n-link cut n-link

13. Multiway cuts • More than 2 labels; typical for motion and stereo – multiway cuts • Repeated binary cuts by forming binary graphs for every pair of labels: alpha-beta swap • Repeated binary cuts by forming binary graphs for a particular label and the existing label, for all labels: alpha expansion

14. Multiway cuts

15. Parent algorithm • Multiway cut for stereo and motion with slanted surfaces, Birchfield and Tomasi 1999 • Combines multiway cuts and affine transformation • Works iteratively by progressive refinement of displacement functions of labels • Algorithm re-implemented, proposed algorithms are extensions to this paper

18. Motion segmentation over image sequences • Parent algorithm when employed on sequence of images, does not produce consistent results • Also computationally inefficient to exhaustively search over all translational displacement functions for every frame. frame1, 5 segments frame2, 6 segments

19. Changes to parent algorithm result from previous frame pair control number of loop iterations do affine merge at the end

20. frame t frame t+1 Algorithm parent algorithm, affine merge • Run parent algorithm on first frame and get correct number of segments, parameters are fixed • Set number of iteration loop for affine update of displacement functions to a constant • Initial motion segment image for next frame is predicted by affine warping of current motion segments and re-estimation of displacement functions • Final iterative affine merge step merges neighboring regions predict label image initialize parent algorithm for next frame frame t+1 frame t+2

21. Affine merging of regions – if neighbor regions within threshold, then merge; if number of segments is still more relax threshold and repeat • This step similar to over segmentation step but does not involve energy computation, hence threshold dependent

22. Results for frames 2, 10, 19 and 25 are shown. • Number of motion segments maintained • Algorithm took 71.18 seconds to run on 27 frames, parent algorithm took 97.04 seconds • Boundaries between segments are not crisp due to occlusions and lack of texture

23. Taxi sequence, algorithm works well for frames 1 to 36. Frame 5, 18 are shown. • Failure for 36 to 40 due to small motion of taxi and two components for right vehicle • Frame 40 failure of affine merge shown • Right vehicle segmentation poor due to occlusion by tree

24. Hard constraint points for stereo • Another extension to stereo correspondence • Cost functional not able to preserve small and thin long objects in depth maps • Multiway cuts smoothes out small regions

25. Normalized correlation • Normalized correlation performed, unambiguous disparity points are chosen and initialized as hard constraint points • These points initialize multiway cuts, number of iterations of affine updating is controlled sum of squared differences inside window scan line left image right image ambiguous minimum clear minimum

27. Occlusion detection • Errors of regions in between motion layers due to movement of foreground over background • Selective occlusion detection is done using estimated affine parameters • Assumption – multiway cuts labels occluded areas with the label of the foreground

28. Compute residues of region a and region b based on affine parameters of both region 1 and region 2 and pick the worst. • The occluded region has the worst residue because it has no matching region in the next frame.

30. Conclusions • Studied, analyzed and implemented multiway cuts and affine transformation techniques • All implementation in C++ from scratch, using Kolmogorov and Blepo • Two extensions to the parent algorithm – motion segmentation over image sequences and hard constraint points for stereo • Simple occlusion detection presented • Results are reasonable

31. Future work • Spatiotemporal multiway cuts for segmenting object in the video volume • Redesigning cost functionals to improve segmentation results • Integrate occlusion detection with multiway cuts for getting cleaner borders.

32. Thank You