Motion Estimation using Markov Random Fields

1. Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb Summer School on Image Processing, Graz 2004

2. Overview • Introduction • Optical flow • Markov Random Fields • OF+MRF combined • Energy minimization techniques • Results

3. Introduction • Input: • Sequence of images (Video) • Problem • Extract information about motion • Applications • Detection, Segmentation, Tracking, Coding

4. Spatio-temporal spectrum f φ

5. Motion – aliasing Large area flicker f 1/t φ 1/x Loss of spatial resolution

6. Large motions - temporal aliasing f Temporal aliasing φ Great loss of spatial resolution

7. Temporalanti-aliasing f φ • No more overlaping on the f axis. • filtering (anit-aliasing) is performed after sampling, hence the blurring

8. Motion – eye tracking f φ

9. Motion estimation • Images are 2-D projections of the 3-D world. • Problem is represented as a labeling one. • Assign vector to pixel • Vector field field of movement • Low level vision • No interpretation

10. Example Ideal

11. Problems • Problem is inherently ill-posed • Solution is not unique • Aperture problem • Specific to local methods

12. Optical flow • Main assumption: Intensity of the object does not change as it moves • Often violated • First solved by Horn & Schunk • Gradient approach • Other approaches include • Frequency based • Using corresponding features

13. Image differencing

14. Gradient approach • Local by nature. Aperture problem is significant. • Image understanding is not required • Very low level

15. Horn & Schunk • Intensity stays the same in the direction of movement. I(x,y,t) •After derivation

16. Horn & Schunk • Spatial gradients Ix,Iy • e.g. Sobel operator • Temporal gradient It • Image subtraction

17. Regularization • Tikhonov regularization for ill-posed problems • Add the smoothness term • Energy function

18. Result

19. Problems of the H-S method • Assumption: There are no discontinuities in the image • Optical flow is over-smoothed. • Gradient method. Only the edges which are perpendicular to motion vector contribute • Image regions which are uniform do not contribute. • Difficulty with large motions (spatial filtering)

20. Optical flow enhancement • Optical flow can be piecewise smooth • Discontinuities can be incorporated • Solution: use the spatial context • Problem is posed as a solution of the Bayes classifier. Solution in optimization sense. Search for optimum

21. Bayes classifier • Main equation • Solution using MAP estimation

22. Markov Random Fields • Suitable: Problems posed as a visual labeling problemn with contextual constraints • Useful to encode a priori knowledge • required for bayes classifier (smoothness prior) • equvalence to Gibbs random fields (gibbs distribution, exponential like) • Neighbourhoods • Cliques • pairs,triples of neighbourhood points) • build the energy function

23. MRF • Define sites: rectangular lattice • Define labels • define neighbourhood: 4,8 point • Field is MRF: • P(f)>0 • P(fi|f{S-i})=P(fi|Ni)

24. Coupled MRF • Field F is an optical flow field • Field L is a field of discontinuities • line process • Position of the two fields.

25. Context • neighbourhoods and cliques

26. Motion estimation equations

27. Energy for MAP estimation Parameters are estimatedad hoc

28. Energy minimization • Global minimum • Simulated annealing • Genetic Algorithms • Local minimum • Iterated Conditional Modes (ICM) (steepest decent) • Highest Confidence First (HCF) • specific site visiting

29. Simulatedannealing • (1) Findthe initial temperatureof the system T. • Assign initial values of the field to random • For every pixel: • Assign random valueto f(i,j) • Calculate the difference in energy before and after If the change is better (diff>0) keep it. Else keep it with the probability exp(diff/T) • (4) Repeat (3) N1 times • (5) T = f(T) where f decreases monotono • (6) Repeat (3-5) N2 times

30. Results (Square) OF+MRF Horn-Schunk OF

31. Taxi

32. Results (Taxi)

33. Line process result (Taxi)

34. Cube

35. Results (cube)

36. Line process result (cube)

37. Q & A