Globally Optimal Wavelet-Based Motion Estimation using Interscale Edge and Occlusion Models

1. Globally Optimal Wavelet-Based Motion Estimation using Interscale Edge and Occlusion Models Levent Sendur lss29@cam.ac.uk Department of Psychiatry, University of Cambridge, UK and Onur G. Guleryuz oguleryuz@erd.epson.com Epson Palo Alto Laboratory Palo Alto, CA (Please view in full screen mode to see the animations.)

2. Overview • Problem statement and the space we are trying to fill among estimation techniques. • Properties. • Probability Model: • Model for uncovered regions (using DCTs). • Observation model. • Motion field model (in wavelet domain). • Interscale dependencies. • Dynamic programming. • Examples. • Conclusion.

3. Problem Statement Well known motion estimation techniques have two important issues • Methods that are deemed to be accurate (such as optical flow algorithms and probabilistic extensions) generate iterative estimates that are only locally optimal. • Models of uncovered regions and motion field discontinuities are incorporated as an afterthought. These models contradict regularization constraints and result in unnatural tug-of-war in the iterations. We propose a dense motion field estimation technique that is noniterative, that can be optimized globally with edges and uncovered regions built in.

4. Block based This paper … Optical flow based, Markov random fields, … Available Estimation Techniques optimization globally optimal locally optimal crude model sophisticated model sophistication

5. Properties of this Work • Per pixel, dense field (but flexible, can use blocks of pixels to reduce computation). • Stochastic and non-iterative. • Piecewise smooth motion field model that captures local smoothness. • Discontinuities and occlusions built in. • Motion field parameterized in terms of its wavelet transform coefficients. • Wavelet based model accounts for interscale dependencies over motion field edges. • Novel and robust occlusion model. • Deliverables: Globally optimal solution + Motion field segmentation.

6. Table Tennis, frames 3 and 4 frame n-1 frame n : smooth : edge : uncovered Field segmentation Field

7. Part that is explained by motion Part that is explained by uncovered regions : occlusion/uncovered region segmentation : motion field : i.i.d. noise Frame Evolution Model : current and past frames : pixel coordinates

8. Bayes Rule Stochastic Model model each conditional probability in turn Region has motion if: Cheaper to “code” with motion vectors (motion vector cost +DFD) Region if uncovered if: Cheaper to code using DCTs.

9. Establish a block around each pixel. • Assume the DCT coefficients of this block are independent Gaussian RVs, N N average coding cost in bits Uncovered Regions Given Occlusion Segmentation : model eachpixel using DCTs. (Given that this pixel is uncovered, what is the probability that it has the value …?) … x … … • Estimate mean and variance for each DCT coefficient. …

10. Observation Model (i.i.d. Gaussian noise)

11. Motion Model : Piecewise smooth Wavelet transform coefficients of the field are sparse. Most coefficients have magnitudes that are close to zero.

12. l=4 l=3 l=2 Wavelet Transform of the Motion Field l=1 y y’ w w’ x x’ z z’

13. Wavelet Transform of the Motion Field …

14. l=4 l=3 l=2 Tree Structure for Everything • All segmentations defined on nodes of our multiresolution tree data structure. • Probability at each node conditionally independent given the parent node (Multiresolutional Markov structure). l=1 y y’ w w’ x x’ z z’

15. Interscale Dependency Model I Segmentation hierarchy follows our tree data structure. l=4 l=3 • Uncovered node hierarchy: l=2 • Motion node hierarchy: or l=1 y y’ w w’ x x’ z z’

16. Motion over smooth region: • Motion over edge region: or Interscale Dependency Model II Motion node hierarchy: Why? Wavelets are sparse but not that sparse. There are interscale dependencies over wavelet coefficients over edges. l=4 l=3 l=2 l=1 y y’ w w’ x x’ z z’

17. Dynamic Programming I 1- Exhaustively consider: • all possible motion vectors on all pixels, • all possible segmentations. 2- Evaluate the log cost: 3- Find the combination that minimizes the log cost.

18. Dynamic Programming II It is dynamic programming because: 1- Tree structure defines additive costs at each node. 2- Multiresolutional Markov structure allows elimination at each node. … l=2 l=1 y y’ w w’ x x’ z z’ Can choose among all that sum up to the same (Bottom-up dynamic programming by propagating three types of costs at each node: uncovered cost, motion-smooth cost, motion-edge cost.)

19. Table Tennis, frames 3 and 4 (over 4x4 blocks) frame n-1 frame n : smooth : edge : uncovered Field segmentation Field

20. Stefan, frames 55 and 56 frame n-1 frame n : smooth : edge : uncovered

21. Stefan, frames 55 and 56 l=11 l=14 l=13 l=12 l=10 l=9 l=8 l=7 l=6 l=5 l=4 l=3 l=2 l=1 : smooth : edge : uncovered

22. Football, frames 128 and 129 frame n-1 frame n : smooth : edge : uncovered

23. Football, frames 98 and 99 frame n-1 frame n : smooth : edge : uncovered

24. Football, frames 136 and 137 frame n-1 frame n : smooth : edge foot13 : uncovered

25. Coast Guard, frames 157 and 158 frame n-1 frame n : smooth : edge : uncovered

26. Conclusion • Globally optimal, dense field. • Noniterative. • Field segmentation. • Robust uncovered region model. • Transform domain models. • Interscale dependencies accounted for. • Dynamic programming (1D) applied in 2D. • Results model (no optimization issues). • Better edge determination. • In conjunction with edges, better uncovered region determination.