Video Face Recognition: A Literature Review

1. Video Face Recognition: A Literature Review Hao Zhang Computer Science Department

2. Problem Statement Verification Identification A A B B C D Same / Different persons? Which has the same identity as A?

3. Solutions • Extensions of still face recognition algorithms • 3D model reconstruction • Employing temporal information • Set-to-set matching methods

4. Extensions of still face recognition algorithms • Jointsparse representation probe gallery Data: k-th partition of a query video Dictionary: a concatenation of all dictionaries of k-th partition of training videos

5. Extensions of still face recognition algorithms • Joint sparse representation : Conclusion • Joint sparse representation • Only suitable for face identification • Cannot handle new faces • Violates the protocol of face verification

6. Extensions of still face recognition algorithms • Multiple metric learning (MML) Video Volumes Patches Feature Extraction MML * A part of this figure is from [5]

7. Extensions of still face recognition algorithms • Multiple metric learning (MML): A conclusion • It can be easily adapted to solve both still and video problems. • It discards additional information in the video.

8. 3D model reconstruction • From a single frontal image: Analysis reconstructed 3D shape Mean training 3D shape PCA projection matrix of training 3D shapes 2D mappings of input 2D shape scale and translation term * The two above images are from [8]

9. 3D model reconstruction • Reconstruction from a single image: Synthesis Pose Illumination Expression * This figure is from [8]

10. 3D model reconstruction • Reconstruction from a single image: Conclusion • Handle pose and illumination variations • 2D images of good quality • Synthesis of lighting and expression is far from perfect

11. Employing temporal information • Dynamic system model, ARMA : state vector encoding pose at time t : face appearance at time t Video similarity is computed using an observability matrix formed by A and C.

12. Employing temporal information • Dynamic system model: Conclusion • Incorporate time information for recognition • Linear assumption • Manifold learning methods can be applied using the observability matrix

13. Employing temporal information • Probabilistic model Can be adapted to handle occlusion : Image I’s distance to the manifold of k-th video : probability of image I’s projection in * The figure is from [9]

14. Employing temporal information • Probabilistic model: Conclusion • Incorporate time information to make decisions more robustly • Error can propagate • Majority voting

15. Set-to-set matching • Manifold-manifold distance Clustering criteria: distance Manifold A Manifold B

16. Set-to-set matching • Manifold-manifold distance: Conclusion • Overcomes the drawbacks of voting methods • Clustering results will be different due to random initialization

17. Set-to-set matching • Affine Hull Representation Affine hull Convex hull Reduced affine hull:

18. Set-to-set matching • Affine Hull Representation: Conclusion • “Size changeable” affine hulls • Unclear which representation is better Which to use: convex hull, affine hull or linear span?

19. Set-to-set matching • Statistical methods on Grassmann manifolds Distribution is defined on the tangent plane of Karcher mean Local mapping using exponential map preserves geodesic distance

20. Set-to-set matching • Statistical methods on Grassmann manifolds: Conclusion • Distribution models on manifold • A video is simply represented as a linear space • Too few samples • Thoughts: • Partition the video to obtain multiple points on Grassmann manifold

21. A summary for each category

22. Important Datasets 2001 2003 2009 2011 2013

23. Comparing Results? Data set Alg

24. Summary • Current trends: • Extensions of still face recognition algorithms • Set-to-set matching methods • Common issues: • Computational burden • Pose variations • Thoughts: good training data and transfer learning • Need common protocols and datasets • Much better recently

25. References • [1]  G. Aggarwal, A. K. R. Chowdhury, and R. Chellappa. A system identification approach for video-based face recognition. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, volume 4, pages 175–178. IEEE, 2004. • [2]  J. R. Beveridge, P. J. Phillips, D. Bolme, B. A. Draper, G. H. Givens, Y. M. Lui, M. N. Teli, H. Zhang, W. T. Scruggs, K. W. Bowyer, et al. The challenge of face recognition from digital point-and-shoot cameras. IEEE Conference on Biometrics: Theory, Applications and Systems, 2013. • [3]  H. Cevikalp and B. Triggs. Face recognition based on image sets. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2567–2573. IEEE, 2010. • [4]  Y.-C. Chen, V. Patel, S. Shekhar, R. Chellappa, and P. Phillips. Video-based face recognition via joint sparse representation. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on, pages 1–8, 2013. • [5]  Z. Cui, W. Li, D. Xu, S. Shan, and X. Chen. Fusing robust face region descriptors via multiple metric learning for face recognition in the wild. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3554–3561, 2013. • [6]  G. Doretto, A. Chiuso, Y. N. Wu, and S. Soatto. Dynamic textures. International Journal of Computer Vision, 51(2):91–109, 2003. • [7]  R. Gross and J. Shi. The cmu motion of body (mobo) database. Technical Report CMU-RI-TR- 01-18, Robotics Institute, Pittsburgh, PA, June 2001. • [8]  D. Jiang, Y. Hu, S. Yan, L. Zhang, H. Zhang, and W. Gao. Efficient 3d reconstruction for face recognition. Pattern Recognition, 38(6):787–798, 2005. • [9]  K.-C. Lee, J. Ho, M.-H. Yang, and D. Kriegman. Video-based face recognition using probabilistic appearance manifolds. In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, volume 1, pages I–313. IEEE, 2003. • [10]  P. J. Phillips, P. J. Flynn, J. R. Beveridge, W. T. Scruggs, A. J. OToole, D. Bolme, K. W. Bowyer, B. A. Draper, G. H. Givens, Y. M. Lui, et al. Overview of the multiple biometrics grand challenge. In Advances in Biometrics, pages 705–714. Springer, 2009. • [11]  J. B. Tenenbaum, V. De Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319–2323, 2000. • [12]  P. Turaga, A. Veeraraghavan, A. Srivastava, and R. Chellappa. Statistical computations on grassmann and stiefel manifolds for image and video-based recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11):2273–2286, 2011. • [13]  R. Wang, S. Shan, X. Chen, and W. Gao. Manifold-manifold distance with application to face recognition based on image set. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1–8. IEEE, 2008. • [14]  L. Wolf, T. Hassner, and I. Maoz. Face recognition in unconstrained videos with matched back- ground similarity. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 529–534. IEEE, 2011.