Robust Video Stabilization Based on Particle Filter Tracking of Projected Camera Motion (IEEE 2009) Junlan Yang Univer

1. Robust Video Stabilization Based on Particle Filter Tracking of Projected Camera Motion (IEEE 2009)Junlan Yang University of Illinois,Chicago

2. Reference • [1]A tutorial on particle filters for online nonlinear non-Gaussian Bayesian tracking • [4]probabilistic video stabilization using kalman filtering and mosaicking • [5]Fast electronic digital image stabilization for off-road navigation • [18]condensation conditional density propagation for visual tracking

3. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

4. Introduction • Video Stabilization • Camera motion estimation • Particle filter • Tracking projected affine model of camera motion • SIFT algorithm (范博凱) • Detect feature points in both images • Removing undesired (unintended) motion • Kalman filter

5. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

6. P P ( x , y , z ) 0 0 0 ( x , y , z ) 1 1 1 Z Z X X Motion Camera Camera Y Y at time t at time t 1 0 Example of camera motion

7. Generating Camera model • Related of two vectors

8. Building 2-D affine model • Projection of P in time t0 and t1

9. Building 2-D affine model • Rewriting the related of two projected vectors • 2-D affine model

10. Building 2-D affine model Global motion estimation is to determine the six parameters for every successive frame

11. Why do she use 2-D affine model to represent camera motion? • A pure 2-D model • 2-D translation vector and one rotation angle • 3-D model • Giant complexity

12. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

13. Particle Filtering Estimation • Markov discrete-time state-space model state vector at time k observations z, andthe posterior density is

14. To approximate the posterior

15. Estimation of current state

16. Importance density q(.) • Traditionally – prior density • This paper takes into account the current observation zk. The proposed important density whose mean vector obtained from the current observation zk • Why do she use the particle filtering estimation ?

17. Advantage of particle filtering estimation • With Low error variance • Proof : In large particle numbers condition, the estimation gives lower error variance than

18. Covariance matrix of errors

19. Lemma 1: where

21. Lemma 1: Strong law of large number

24. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

25. Complete system of video stabilization • At time k

26. Getting six parameters • SIFT algorithm – Find corresponding pairs • At time k It needs three pairs to determine a unique solution Y X A

27. (a) SIFT correspondence from frame 200,201 in outdoor sequence STREET

28. Generate particles • Important density q(.) is a six-dimensional Gaussian distribution • Particles • In experience , N set to only 30 with better quality than prior distribution set N = 300

29. Quality of the particles • N particles have N proposals of transformation matrix ,and N Inverse transform to frame k have N candidate image Ai • Compare these images with k-1 frame A0

30. Similar with A0 and Ai • Mean square error • Difference of gray-scale from pixel to pixel • Feature likelihood • Distance of all corresponding feature points

31. Particle filtering for global motion estimation • Weight for each particle • Estimation of current state where

32. Accumulative motion • At time k-1 to k • At time 0 to k Where s is scaling factor , R is rotation matrix and T is translation displacement

34. Intentional Motion estimation and motion compensation • Compensate for the unwanted motion

35. Complete system of video stabilization • At time k

36. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

37. Original image , (b) Matched-feature-based motion estimation (MFME) • (c) p-norm cost function-based motion estimation (CFME) (d) proposed method (PFME)

38. Original image , (b) MFME (c) CFME (d) PFME

39. Original image , (b) MFME (c) CFME (d) PFME

40. Original video sequence (ground truth) (b) unstable video sequence (c) PFME

41. Ty? (a) Motion in horizontal direction (b) Motion in vertical direction

42. Comparison of average MSE and PSNR for stabilized output PSNR = peak signal to noise ratio Large PSNR has low distortion

43. Outline • Introduction • Camera Model • Particle Filtering Estimation • Complete System of Video Stabilization • Simulation and Results • Conclusion

44. Conclusion • We demonstrated experimentally that the proposed particle filtering scheme can be used to obtain an efficient and accurate motion estimation in video sequences.

45. Contributed of this paper • Constraining rotation matrix projected onto the plane ?(depth change) • Show using particle filtering can reduce the error variance compared to estimation without particle filtering • Using both Intensity-based motion estimation method (PFME) and feature-based motion estimation (SIFT) method