Mean Shift For Tracking

1. Mean Shift For Tracking CS 663, AjitRajwade Based on a paper by Comaniciu, Ramesh and Meer

2. Problem statement • Ask user to mark out a region of interest (ROI) in the first frame of a video. • Track the movement of the ROI in all subsequent frames of the video automatically.

5. Target Histogram • Let us suppose the ROI (i.e. the object we want to track) is centered at location (x0,y0) in frame 0. • Letthe set {xi,yi},1<=i<=N, stand for all pixel locations inside or on the object. • We will characterize the object by its color distribution (i.e. histogram) {qu},1<=u<=M, taking the colors values at {xi,yi}. This is called the target histogram.

6. Algorithm: Object Model • The ROI obviously moves in the video across frames. We want to track the ROI. • So we search in frame 1 for the ROI in the vicinity of point (x0,y0).

7. Frame 0 (ROI in red) Frame 1 (Notice that ROI has moved) (x0,y0) (x0,y0) (x,y)

8. Naïve Algorithm • Naïve method: • Take small windows of the same shape as the ROI centered at a point (x’,y’) in the neighborhood around (x0,y0) in frame 1. • Compare its histogram with {qu},1<=u<=M, using some measure of histogram similarity. The paper uses the Bhattacharya coefficient (larger coefficient = more similar histograms): • Repeat for all other points in the neighborhood. The point with the maximum value of the Bhattacharya coefficient gives the correct location of the object in the new frame. Histogram of a region centered at (x’,y’)

9. Faster Algorithm • The paper by Comaniciu and Meer suggests a better method. • Consider the Bhattacharya coefficient between target histogram q and the histogram of the ROI (i.e. region centered at location (x0,y0)) in frame 1: • The Bhattacharya coefficient between the target histogram and the histogram of a region centered at arbitrary location (x,y) in the neighborhoodof (x0,y0) in frame 1 is:

10. Faster Algorithm • We now do a Taylor series expansion of about giving us: See homework solutions for detailed derivation http://www.cse.iitb.ac.in/~ajitvr/CS663/HW5_sol Note when we take a Taylor series expansion, there is an implicit assumption that is not too far away from Weighted histogram: locations farther away from the center are given lower weights

11. Faster Algorithm • Plugging the expression for into the expression for gives us: • We want to maximize the LHS w.r.t. (x,y). So we take its first derivative and set it to 0. The first term in the above equation vanishes.

12. Faster Algorithm • When you complete the calculations, you get the following:

13. Final Algorithm • Compute the target histogram in frame t = 0 at location (x0,y0) . • Initialize the target object to location (x0,y0) in frame t=1. • Compute the value of . • Perform mean shift as follows with initially x = x0, y = y0: • If change in x and y is small, stop; else continue with more mean-shift updates. • Repeat the process for next frame, i.e. t = 2. Set

14. Scale changes? • What do you do if the object shrinks or expands (Scale changes) from one frame to another? • Run mean-shift with different values of h. Pick the one which gave the largest value of the Bhattacharya coefficient. The size of the ROI is directly related to h (larger h = larger ROI).