Advanced Computer Vision Lec 07: Optical Flow

1. Advanced Computer VisionLec 07: Optical Flow Yuheng Wang yxw9636@rit.edu 1/4/2011

2. Motivation • Motion is a rich source of information about the world: • Moving objects detection and segmentation • Observer self-motion • Object behavior understanding • Scene dynamics understanding

3. Temporal Info Extraction • Optical Flow: the apparent motion of brightness patterns in the image (u, v) ? E(x, y, t) E(x+dx, y+dy, t+dt) framet: E(x, y, t) framet+dt: E(x+dx, y+dy, t+dt)

4. v Ex Et Optical Flow: 1D Case Brightness Constancy Assumption (BCA): No change in brightness with time

5. Tracking in the 1D case ? 0

6. Temporal derivative: Spatial derivative: Tracking in the 1D case α 0 • Assumptions: • Brightness constancy • Small motion

7. Temporal derivative at 2nd iteration The same estimate for spatial derivative Tracking in the 1D case Iteration helps refining the velocity vector 0 Spatial derivative:

8. 2D: From 1D to 2D Color/intensity constancy assumption 1D: One equation, two velocity (u,v) unknowns…

9. Formally: 2D Motion estimation • To find correspondence between frames • Given a pixel in E(x, y, t), to find nearby pixels of the same color in E(x+dx, y+dy, t+dt) • Key assumptions • Color/intensity constancy: no color/intensity values of the corresponding pixels in frame t and frame t+1 are constant • Small motion: motion for pixels a small region are small and uniform

10. Brightness constancy assumption • Brightness constancy assumption is often assumed by researchers(the basis of optical flow methods), however, it’s not always the case in real world (poor estimation of the 2D real motion field) figure by Jahne et al, 1999

11. 2D Optical flow • A pixel (x, y) at time t with intensity E(x, y, t): • Image constraint • (No illumination changes) • chain rule of differentiation • Brightness constancy • constraint equation (BCCE) where : partial derivatives of E with respect to x, y, t, respectively (can be calculated) (u, v) :velocity of optical flow (to be solved)

12. Aperture problem (1) • BCCE: 1 equation, 2 unknowns • can only calculate the component of velocity in the direction of gradient Normal flow: Can NOT determine the component perpendicular to Vn Known as aperture problem v u Constraint line

13. Aperture problem(2) • Underline conditions

14. Visual demo of aperture problem Motion along just an edge is ambiguous http://robots.stanford.edu/cs223b/index.html

15. Aperture Problem Exposed More constraint need to solve u and v http://robots.stanford.edu/cs223b/index.html

16. Aperture Problem in real life Z axis Motion field Optical flow Barber’s pole http://www.sandlotscience.com/Ambiguous/Barberpole_Illusion.htm

17. “optical flow” revisited • Optical flow: the apparent motion of brightness patterns in the image. • Generally, optical flow corresponds to the motion field, but not always(the barber's pole illusion), these are unusual cases. • For this lecture we will assume that optical flow corresponds to the motion field.

18. Solve for u and v • Add more constraints to u and v • Lucas-kanade method: • Combine local constraints of u and v of multiple pixels in a small window • Horn-Schunck Method • Regularize the variations of u and v across the flow field (assumes that the transitions of u and v are smooth across the space)

19. Lucas-Kanade method • Idea: the velocity is small and approximately constant for all pixelswithin a small window between 2 consequent frames wherep1, p2, …pnare pixels inside the window centered at p • 2 unknowns (u, v) and n equations: over-determined system • A compromised solution by weighted least squares

20. Combine local constraints v u 0 Constraint lines

21. LK solution • These equations can be written in matrix format: Know A and b, solve for V Method: pseudo inverse Window surround p: Aperture problem

22. Weighted window • Give different weights to different pixels where W is a n x n diagonal matrix containing the weights W(i,i) = wi assigned to pixel pi wi : usually Gaussian function of the distance between window center and pixel pi

23. λ1 , λ2are the 2 eigenvalues of matrixATA (λ1 > λ2) Large λ1 Large λ2 Large λ1 Small λ2 Small λ1 Small λ2 Corner pixels Edge pixels Low texture pixels

24. Implementation Frame E(k-1) Estimate (u,v) of E(k) Inverse Mapping: E’(k-1) No Error < thres? Yes Stop

25. Errors in Lucas-Kanade • What are the potential causes of errors of LK algorithm? • ATA is not invertible • There is much noise in the image • When our assumptions are violated • Brightness constancy is NOT satisfied • The motion is NOT small • A point does NOT move like its neighbors • Window size is too large • What is the ideal window size?

26. Variational methods • Function: maps an input value to an output value • Functional: maps an input function to an output value function functional f (x, y) E( f (x, y) ) Output: scalar Input: (x, y) scalar Output: scalar Input: f(x, y) vector

27. General ideas • Idea: compute functions u(x, y) and v(x, y) to minimize a proper energy functional (total error): • Data term D(u,v) penalizes deviations from constancy assumptions • Smoothness term S(u,v) penalizes deviations from smoothness of the solution • Regularization parameterα>0determines the degree of smoothness Smoothness Data fidelity

28. Horn-Schunck Method (1981) • Idea: Assume over smoothness of the flow field • Data term: penalize the deviations from Brightness Constancy Assumption (BCA) • Smoothness: penalize sum of squares of the magnitudes or Stationary cameras: Data fidelity Smoothness

29. is always full rank. HS regularization Without regularization: Solution: where , The above solution requires that ATAbe of full rank, that is, on a corner. Add a regularization term to the cost:

30. Regularized flow ( sum of squares is minimal) Optical flow Regularization • It’s a sum of squared terms (a Euclidian distance measure). • We’re putting it in the expression to be minimized • => In texture free regions, v = 0 • => On edges, points will flow to nearest points. Optical flow Regularized flow

31. Estimation of partial derivatives • Take the average of 4 pairs of difference i i +1 k +1 k j j +1

32. Weighted sum: convolution j j+1 i i+1 W2 = 0.25* W1 = 0.25* W3 = 0.25* W4 = 0.25 * i i +1 k +1 k j j +1

33. Implementation(1): computeDerivatives.m function [Ex, Ey, Et] = computeDerivatives(Im1, Im2) % Horn-Schunck original method Ex = conv2(Im1, 0.25* [-1 1; -1 1], 'same')… + conv2(Im2, 0.25*[-1 1; -1 1], 'same'); Ey = conv2(Im1, 0.25*[-1 -1; 1 1], 'same')… + conv2(Im2, 0.25*[-1 -1; 1 1], 'same'); Et = conv2(Im1, 0.25*ones(2), 'same')… + conv2(Im2, -0.25*ones(2), 'same');

34. Estimation of the Laplacians where The Laplacian: a weighted average of the values at neighboring points

35. Iterative solutions • Iterative solutions for u and v • Where αis the regularization parameter( α>0) determining the degree of smoothness • α2: prevents haphazard adjustments to the estimated flow velocity • caused by noise in the estimated derivatives(Ex and Ey) .

36. Implementation(2): hsmain.m Im1 = conv2(Im1, gaskernel, 'same'); % Use a Gaussian kernel for smoothing Im2 = conv2(Im2, gaskernel, 'same'); % Set initial value for the flow vectors u = zeros(size(Im1)); v = zeros(size(Im1)); % Estimate spatiotemporal derivatives [Ex, Ey, Et] = computeDerivatives(Im1, Im2); avgkernel = [1/12 1/6 1/12; 1/6 0 1/6; 1/12 1/6 1/12]; % Averaging kernel for k =1: numEpochs % Compute local averages of the flow vectors ubar = conv2(u, avgkernel, 'same'); vbar = conv2(v, avgkernel, 'same'); % Compute flow vectors constrained by its local average and the optical flow constraints u = ubar - (Ex .*(Ex .*ubar + Ey .*vbar + Et))./(alpha^2 + Ex.^2 + Ey.^2); v = vbar - (Ey .*(Ex .*ubar + Ey .*vbar + Et))./(alpha^2 + Ex.^2 + Ey.^2); end

37. Experiments • 3 tennis videos • Each video: 600 frames (= 30fps x 20s ) • Each frame: 240x320x3 • Convert color image to gray-scale image (240x320) • Compute optical flow of each frame • Only works for first 200 frames for each video

38. Partial derivatives: Ex , Ey Ex Ey

39. Optical flow of 1 frame

40. Tracking results • tennis1track.avi http://www.youtube.com/watch?v=SYwyoGvZ_4k (partially loses player 2) • tennis2track.avi http://www.youtube.com/watch?v=Qz0UM31e5Cs • tennis3track.avi http://www.youtube.com/watch?v=55uqkjk5sGw

41. Track of 2 players in tennis video 1http://www.youtube.com/watch?v=SYwyoGvZ_4k (partially loses player 2) Frame numver Player 1 Player 2 x coordinate y coordinate

42. Track of 2 players in tennis video 2http://www.youtube.com/watch?v=Qz0UM31e5Cs Frame numver Player 1 Player 2 x coordinate y coordinate

43. Track of 2 players in tennis video 3http://www.youtube.com/watch?v=55uqkjk5sGw Frame numver Player 1 Player 2 x coordinate y coordinate

44. Tracks of 2 players in all 3 tennis videos Frame numver Player 1 Player 2 x coordinate y coordinate

45. Discussion • Pros of optical flow method • Work with both videos by both stationary cameras and non-stationary cameras • Can compute observer self-motion • Cons of optical flow method • Accumulating tracking error: “… tracking by only optical flow will suffer from tracking drift, since error will accumulate when only velocity information is used.“ • Remedy: object detection + optical flow tracking DeCarlo and Metaxas, IJCV 2000

46. References [1] David J. Fleet and Yair Weiss (2006). "Optical Flow Estimation". Handbook of Mathematical Models in Computer Vision. Springer. [2] B. D. Lucas and T. Kanade. “An iterative image registration technique with an application to stereo vision”. Proceedings of Imaging Understanding Workshop, pp. 121—130, 1981. [3] B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial Intelligence, vol 17, pp 185-203, 1981. [4] http://robots.stanford.edu/cs223b/index.html [5] Douglas DeCarlo and Dimitris Metaxas. “Optical Flow Constraints on Deformable Models with Applications to Face Tracking”, International Journal of Computer Vision, 38(2), pp. 99-127, July 2000.