Enforcing Convexity for Improved Alignment with Constrained Local Models

1. Enforcing Convexity for Improved Alignment with Constrained Local Models Authors: Yang Wang, Simon Lucey, Jeffrey F. Cohn 讲解人: 赵小伟 文章出处:CVPR’08

2. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

3. 第一作者 • Yang Wang • Research scientist at Siemens • Robotics Institute, Carnegie Mellon University • Ph.D. in Computer Science, August 2000 - December 2006 Stony Brook University, New York, USA • B.S. in Computer Science, September 1993 - July 1998 Tsinghua University, Beijing, China • Research Areas • Computer Vision, Graphics, Medical Image Analysis, Biometrics, Machine Learning, Computer Animation, and Augmented Reality • Publication • PAMI(07, 09) , IJCV(08), IVC(08, 09, 10), ECCV(02, 08), CVPR(04, 06, 07, 08, 10), ICCV(05, 07, 09), FG(08) • Homepage • http://www.cs.cmu.edu/~wangy/

4. 第二作者 • Simon Lucey • PhD Student, Universitat Pompeu Fabra • Research Interest • I am passionate about gaining a deeper understanding of perception, learning and intelligence. My practical interests are in analyzing faces, biometrics and human event recognition. From an academic perspective I am extremely interested in computer vision, machine learning and how these evolving topics relate to deeper questions concerning Artificial Intelligence (AI).  • Publication • PAMI’10 , IVC’10, IJCV’08, PRL’07, Multimedia’05, ICCV, CVPR • Homepage • http://www.cs.cmu.edu/~slucey/Main.html

5. 第三作者 • Jeffrey F. Cohn • Jeffrey Cohn is Professor of Psychology at the University of Pittsburgh and Adjunct Faculty at the Robotics Institute at Carnegie Mellon University. • Research Interest • He has led interdisciplinary and inter-institutional efforts to develop advanced methods of automatic analysis of facial expression and prosody; and applied those tools to research in human emotion, social development, non-verbal communication, psychopathology, and biomedicine. • Database • Cohn-Kanade AU-Coded Facial Expression Database. CK • Cohn-Kanade Expanded. CK+ • CMU MultiPie. • Homepage • http://www.pitt.edu/~jeffcohn/

6. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

7. 文章信息 • 文章出处 • CVPR 2008 • 相关文献 • [7] D. Cristinacce and T.F. Cootes. Feature detection and tracking with constrained local models. In BMVC, pages 929-938, 2006 • [3] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unifying framework: Part 1: The quantity approximated, the warp update rule, and the gradient descent approximation. IJCV, 2004. • [19] Y. Wang, S. Lucey, and J. Cohn. Non-rigid object alignment with a mismatch template based on exhaustive local search. In IEEE Workshop on Non-rigid Registration and Tracking through Learning, 2007.

8. Abstract • Constrained local models (CLMs) have recently demonstrated good performance in non-rigid object alignment/tracking in comparison to leading holistic approaches (e.g., AAMs). A major problem hindering the development of CLMs further, for non-rigid object alignment/tracking, is how to jointly optimize the global warp update across all local search responses. Previous methods have either used general purpose optimizers (e.g., simplex methods) or graph based optimization techniques. Unfortunately, problems exist with both these approaches when applied to CLMs. • In this paper, we propose a new approach for optimizing the global warp update in an efficient manner by enforcing convexity at each local patch response surface.

9. Abstract • Furthermore, we show that the classic Lucas-Kanade approach to gradient descent image alignment can be viewed as a special case of our proposed framework. • Finally, we demonstrate that our approach receives improved performance for the task of non-rigid face alignment/tracking on the MultiPIE database and the UNBC-McMaster archive.

10. 摘要 • 与基于全局的方法相比(例如AAM)，带有局部约束的模型(CLMs: Constrained Local Models)在非刚性物体的对齐和跟踪方面展示了更好的性能。对于非刚性物体的对齐和跟踪，一个主要的阻碍CLMs进一步发展的问题是:“如何根据局部搜索的响应，对全局形变的更新参数(Global warp update)进行联合优化？”之前的方法要么采用general的优化方式(例如单纯形法),要么采用基于图的优化技术。不幸的是，当应用于CLMs时，这些方法都存在问题。 • 本文提出了一种新的方法，强制每个局部patch的响应曲面为凸，这样就可以以一种高效的方式对全局形状更新进行优化。进一步，我们证明经典的基于Lucas-Kanade方法进行梯度下降的图像对齐可以看做本文提出的框架的一个特例。 • 最后，在非刚性的人脸对齐和跟踪方面，我们的方法在Multi-PIE和UNBC-McMaster数据库上取得了更好的性能。

11. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

12. Overview of Constrained Local Models • an exhaustive local search for feature locations to get the response maps • (ii) an optimization strategy to maximize the responses of the PDM constrained landmarks. Saragih, J.M.; Lucey, S.; Cohn, J.F.; , “Face alignment through subspace constrained mean-shifts,” ICCV, 2009, pp.1034-1041

13. Key Steps of CLMs • Estimating patch/region experts • Obtaining local responses • Estimating PDM(point distribution model) • Constrained local model fitting

14. Key Steps of CLMs • Estimating patch/region experts • Obtaining local responses • Estimating PDM(point distribution model) • Constrained local model fitting

15. Estimating patch experts • Arbitrary classifier can be employed to learn patch experts within a CLM framework • boosting schemes (e.g., AdaBoost, GentleBoost, etc.) • relevance vector machine (RVMs) • A linear SVM classifier was chosen, due to • computational advantages

16. Key Steps of CLMs • Estimating patch/region experts • Obtaining local responses • Estimating PDM(point distribution model) • Constrained local model fitting

17. Obtain local responses (a) is the source image to be aligned, while the black box stand for the search window (25*25), the red cross illustrate the ground truth alignment. (d) and (e) show the estimated logistic regression weight values of (b) and (c), respectively. (b) shows the local search responses using patch experts trained by 125 positive examples and 15k negative examples. (b) shows the local search responses using patch experts trained by 125 positive examples and 8k negative examples.

18. Key Steps of CLMs • Estimating patch/region experts • Obtaining local responses • Estimating PDM(point distribution model) • Constrained local model fitting

19. Estimating PDM • A point distribution model (PDM) is used for a parametric representation of the non-rigid shape variation in the CLM. • The non-rigid warp function can be described as , where , p is a parametric vector describing the non-rigid warp, andV is the matrix of concatenated eigenvectors. N is the number of patch-experts. • Principal component analysis (PCA) is then employed to obtain shape eigenvectors V that preserved 95% of the similarity normalized shape variation in the train set.

20. Key Steps of CLMs • Estimating patch/region experts • Obtaining local responses • Estimating PDM(point distribution model) • Constrained local model fitting

21. CLM fitting • Based on the patch experts, non-rigid alignment as be posed as the following optimization problem: • where is the inverted classifier score function obtained from applying the th patch expert to the source image patch intensity • The displacement is constrained to be consistent with the PDM • The matrix can be decomposed into submatrices for each patch expert, i.e.

22. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

23. 拟解决的问题 • How to jointly optimize global warp update across all local search responses? • In general, it is difficult to solve for p, as is a discrete function due to only taking on integral values and there is no guarantee for being convex.

24. A Sub-optimal Approach • Exhaustive Local Search (ELS) • Instead of optimizing for the holistic warp update p directly, ELS optimizes for N local translation updates by exhaustively searching local regions of the object • Where is the local warp update displacement of the kth region/patch (k=1,…N) within a local search region. Then • Where V is the matrix of concatenated eigenvectors. W is weighting matrix, Y. Wang, S. Lucey, and J. Cohn., Non-rigid object alignment with a mismatch template based on exhaustive local search, In IEEE Workshop on Non-rigid Registration and Tracking through learning, 2007.

25. 本文的解决思路Learning from Lucas-Kanade (1/3) • Let us assume that we are attempting to solve for N local translation updates as in the following equation • When a sum of squared differences (SSD) error function is employed: where T is an arbitrary defined template. We no longer have to exhaustively search a local region around .

26. 本文的解决思路Learning from Lucas-Kanade(2/3) • Equation can be rewritten by employing a first Taylor series approximation at . which can be expressed generically in the form of a quadratic, given,

27. 本文的解决思路Learning from Lucas-Kanade(3/3) • Since is virtually always guaranteed of being positive definite, this implies the quadratic is convex, and has a unique minima. Since the summation of N convex functions is still a convex function, it is possible to solve not only for the local translation undates but the entire warp update explicitly, where V is the matrix of concatenated eigenvectors describing the PDM, and the matrix A has the form

28. 本文的解决思路Generic Convex Quadratic Curve Fitting • When is not a SSD classifier, but any function that gives a low value for correct alignment, • For 2D image alignment, the problem can be further simplified as where

29. 本文的解决思路Generic Convex Quadratic Curve Fitting • The above optimization is a quadratically constrained quadratic program (QCQP) and in general costly to be solved directly. • So, is enforced to be a diagonal matrix with non-negative diagonal elements. More specially, • So, Convex quadratic fitting (CQF), which can be solved efficiently.

30. 本文的解决思路Generic Convex Quadratic Curve Fitting • Algorithm outline

31. 进一步的改进 • Robust error function • In particular, the robust error function can be defined as

32. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

33. Example Fits Examples of fitting local search responses: (a) is the local search responses in Figure 1 (d) using patch experts trained by a linear support vector machine (SVM). (b-d) show the surface fitting results. More specifically, (b) picks the local displacement with the minimum response value in the search window, while (c) and (d) fit the local search response surface by a quadratic kernel in Equation 15 and a quadratic kernel with a robust error function in Equation 16, respectively. The brighter intensity means the smaller matching error between the template and the source image patch. In each search window, the red cross illustrates the ground truth location. As we can see, in most cases, the above three methods can all achieve good performance, while the proposed convex quadratic fitting (CQF) (c) and the robust convex quadratic fitting (RCQF) (d) methods are less sensitive to local minima than the exhaustive local search (ELS) method (b).

34. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

35. 本文方法与已有方法的对比

36. 本文方法与已有方法的对比

37. 提纲 • 作者信息 • 文章信息 • 背景知识/(Constrained Local Models) • 拟解决的问题与采用的思路 • 实验 • 结论 • Demos

38. 本文可以借鉴的地方 • Formulation

39. Demos • CLMs Demo: http://web.mac.com/jsaragih/iWeb/FaceTracker/FaceTracker.html

40. 谢谢！

41. 附录

42. ASM(1/7) • 一个物体的几何描述分为两部分： • 相似变换（旋转、缩放、平移） • 形状

43. ASM(2/7) • ASM的任务： • 得到姿态参数 • 得到形状的低维表示，即参数b • ASM匹配的基本过程： • 搜索 • 在马氏距离下搜索与相应灰度梯度分布模型最匹配的特征点 • 调整 • 对搜索得到的形状进行调整，以确保获得的形状是可用的

44. ASM(3/7) • ASM模型： • 要完成ASM搜索与匹配的过程，必须要有相应的统计模型做支撑 • ASM模型分为： • 每个标注点的灰度梯度分布模型 • 点分布模型

45. ASM(4/7) • 每个标注点的灰度梯度模型的构建 • 取标注点 的profile，并计算得到该profile的归一化的灰度梯度向量 • Profile灰度采样： • 梯度 • 归一化 • 若训练集中有M个形状，每个形状有N个标注点，那么对于每个标注点 ，有协方差矩阵 ，这N个协方差矩阵就构成 了灰度梯度模型

46. ASM(5/7) • 点分布模型的构建 • 对于单个形状，每个标注点的坐标为 • 将所有标注点的坐标串接起来，就得到一个形状向量 • 若训练集中有M个形状，那么我们就有M个形状向量 • 于是就可以训练形做PCA，得到能描述 训练集形状变化的特征值与特征向量， 即得到了点分布模型

47. ASM(6/7) • ASM搜索 • 对沿标注点 的profile线的每个像素点 取梯度向量 • 搜索点为：

48. ASM(7/7) • ASM调整 • 假定经过一步搜索之后得到形状 • 将形状进行PCA投影得到参数b • 利用b重新计算形状