Optimization & Learning for Registration of Moving Dynamic Textures

1. Optimization & Learning for Registration of Moving Dynamic Textures Junzhou Huang1, Xiaolei Huang2, Dimitris Metaxas1 Rutgers University1, Lehigh University2

2. Outline • Background • Goals & Problems • Related Work • Proposed Method • Experimental Results • Discussion & Conclusion

3. Background • Dynamic Textures (DT) • static camera, exhibiting certain stationary properties • Moving Dynamic Textures (MDT) • dynamic textures captured by a moving camera DT [Kwatra et al. SIGGRAPH’03] MDT [Fitzgibbbon ICCV’01]

4. Background • Video registration • Required by many video analysis applications • Traditional assumption • Static, rigid, brightness constancy • Bergen et al. ECCV’92, Black et al. ICCV’93 • Relaxing rigidity assumption • Dynamic textures • Fitzgibbon, ICCV’01; Doretto et al. IJCV’03; Yuan et al. ECCV’04; Chan et al. NIPS’05; Vidal et al. CVPR’05; Lin et al. PAMI’07; Rav-Acha et al. Dynamic Vision Workshop at ICCV’05; Vidal et al. ICCV’07

5. Our Goal • Registration of Moving Dynamic Textures • Recover the camera motion and register image frames in the MDT image sequence Translation to the left Translation to the right

6. Complex Optimization Problem • Complex optimization • W.r.t. camera motion, dynamic texture model • Chicken-and-Egg Problem • Challenges • About the mean images • About Linear Dynamic System (LDS) model • About the camera motion

7. Related Works • Fitzgibbon, ICCV’01 • Pioneering attempt • Stochastic rigidity • Non-linear optimization • Vidal et al. CVPR’05 • Time varying LDS model • Static assumption in small time windows • Simple and general framework but often under-estimate motion

8. Formulation • Registration of MDT • I(t), the video frame • , camera motion parameters • y0 , the desired average image of the video • y(t), appearance of DT • x(t), dynamics of DT

9. Generative Model Generative image model for a MDT

10. First Observation • Good registration • A good registration according to the accurate camera motion should simplify the dynamic texture model while preserving all useful information • Used by Fitzgibbon, ICCV’01, Minimizing the entropy function of an auto regressive process • Used by Vidal, CVPR’05, optimizing time varying LDS model by optimizing piecewise LDS model

11. Second Observation • Good registration • A good registration according to the accurate camera motion should lead to a sharp average image whose statistics of derivative filters are similar to those of the input image frames. • Statistics of derivative filters in images • Student-t distribution/heavy-tailed image priors • Huang et al. CVPR’99, Roth et al. CVPR’05

12. Prior Models • The Average Image Prior • The Motion Prior • The Dynamics Prior

13. Average Image Priors • Student-t distribution • Model parameters / contrastive divergence method (a) Before registration, (b) In the middle of registration (c) After registration

14. Motion / Dynamics Priors • Gaussian Perturbation (Motion) • Uncertainty in the motion is modeled by a Gaussian perturbation about the mean estimation M0 with the covariance matrix S ( a diagonal matrix) • Motivated by the work [Pickup et al. NIPS’06] • GPDM / MAR model (Dynamic) • Marginalizing over all possible mappings between appearance and dynamics • Motivated by the work [Wang et al. NIPS’05], [Moon et al. CVPR’06]

15. Joint Optimization • Generative image model • Optimization • Final marginal likelihood • Scaled conjugate gradients algorithm (SCG)

16. Procedures • Obtaining image derivative prior model • Dividing the long sequence into many short image sequences • Initialization for video registration • Performing model optimization with the proposed prior models until model convergence. • With estimated y0, Y and X, the camera motion is then obtained iteratively by Maximum Likelihood estimation using SCG optimization

17. Obtaining Data • Three DT video sequences • DT data [Kwatra et al. SIGGRAPH’03] • Synthesized MDT video sequence • 60 frames each, no motion from 1st to 20th frame and from 41st to 60th • Camera motion with speed [1, 0] from 21st to 40th

18. Grass MDT Video • The average image (a) One frame, (b) the average image after registration, (c) average image before registration

19. Grass MDT Video • The statistics of derivative filter responses

20. Evaluation / Comparison • False Estimation Fraction • Comparison with two classical methods • Hybrid method, [Bergen et al. ECCV’92] [Black et al. ICCV’93] • Vidal’method, [Vidal et al. CVPR’05]

21. Waterfall MDT Video • Motion estimation • Ground truth, (b) by hybrid method, (c) by Vidal’s, (d) by our method

22. Waterfall MDT Video • The average Image and its statistics The average image and its derivative filter response distribution after registration by: (a) our method, (b) Vidal’s method, (c) hybrid method

23. FEF Comparison • On three synthesized MDT video

24. Experiment on real MDT Video • Moving flower bed video • 554 frames total • Ground truth motion 110 pixels • Estimation 104.52 pixels ( FEF 4.98%)

25. Conclusions • Proposed: • Powerful priors for MDT registration • Solution for: • Camera motion, Average image of video, Dynamic texture model • What have we learned? • Correct registration simplifies DT model while preserving useful information • Better registration leads to sharper average image

26. Thank you !

27. Future work • More complex camera motion • Different metrics for performance evaluation • Multiple dynamic texture segmentation

28. Experiment on real MDT Video • Moving flower bed video • Our method • 554 frames total • Ground truth motion 110 pixels • Estimation 104.52 pixels ( FEF 4.98%) • Vidal’s method • 250 frames [Vidal et al. CVPR’05] • Ground truth motion 85 pixels • Estimation 60 pixels (FEF 29.41%)