Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004

1. Real-time Combined 2D+3D Active Appearance ModelsJing Xiao, Simon Baker,Iain Matthew, and Takeo KanadeCVPR 2004 Presented by Pat Chan 23/11/2004

2. Outline • Introduction • Active Appearance Models AAMs • 3D Morphable Models 3DMMs • Representational Power of AAM • Combined 2D+3D AAMs • Conclusion

3. Introduction • Active Appearance Models are generative models commonly used to model faces • Another closely related type of face models are 3D Morphable Models • In this paper, it tries to model 3D phenomena by using the 2D AAM • Constrain the AAM with the 3D models to achieve a real-time algorithm for fitting the AAM

4. Active Appearance Models (AAMs) • 2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh. • Shape scan be expressed as a base shape s0. • pi are the shape parameter. • s0 is the mean shape and the matrices si are the eigenvectors corresponding to the m largest eigenvalues

5. A0(u) A1(u) A2(u) A3(u) Active Appearance Models (AAMs) • The appearanceof an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u∈ s0 • A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance • Coefficients λi are the appearance parameters.

6. Active Appearance Models (AAMs) • The AAM model instance with shape parameters pand appearance parameters λ is then created by warping the appearance Afrom the base mesh s0 to the model shape s. Piecewise affine warp W(u; p): (1) for any pixel u in s0 find out which triangle it lies in, (2) warp u with the affine warp for that triangle. M(W(u;p))

7. u u u u Fitting AAMs • Minimize the error between I (u) and M(W(u; p)) = A(u). • If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p). • At pixel u the AAM has the appearance • At pixel W(u; p), the input image has the intensity I (W(u; p)). • Minimize the sum of squares of the difference between these two quantities:

8. DEMO Video – 2D AAMs

9. DEMO Video – 2D AAMs

10. 3D Morphable Models (3DMMS) • 3D shape of 3DMM is defined by 3D triangulated mesh and in particular the vertex location of the mesh. • The ŝ can be expressed as a based shape ŝ0 plus a linear combination of m shape matrices ŝi:

11. 3D Morphable Models (3DMMS) • The appearance of a 3DMM is defined within a 2D triangulated mesh that has the same topology as the base mesh ŝ0. • The appearance Â(u) can be expressed as a based appearance Â0(u) plus a linear combination of l appearance images Âi(u).

12. 3D Morphable Models (3DMMS) • To generate a 3DMM model instance, an image formation model is used to convert the 3D shape ŝ in to 2D mesh. • The result of the imaging 3D point x = (x, y, z)T is: • i, j are the projection axes, o is the offset of the origin • Given shape parameters picompute 3D shape  map to 2D  compute appearance  warp onto 2D mesh (defined by mapping from 2D vertices in ŝ0 to 2D vertices for 3D ŝ.)

13. Representational Power of AAM Can 2D shape models represent 3D? • The 2D shape variation of the 3D model: • The projection matrix can be expressed as: • Therefore 3D model can be represented as combination of: • The variation of the 3D model can therefore be represented by an appropriate set of 2D shape vectors, such as: 6*(m^+1) 2D shape vectors needed to represent a 3D model

14. Representational Power of AAM • Experiments • Use 3D-cube:ŝ = ŝ0 + p1ŝ1 • Generate 60 sets p1 and P randomly • Synthesize 2D shapes of 60 3D model instances • Compute 2D shape model by performing PCA on 60 2D shapes • Result: 12 shapes vectors for each 2D shape mode • Confirm: 6*(m^+1) 2D vector is required • However, 2D models generate invalid cases. • Constrains is need to add on the model

15. Combined 2D + 3D AAMs • At time t, we have • 2D AAM shape vector in all N images into a matrix: • Represent as a 3D linear shape modesW = MB =

16. Compute the 3D Model • Perform singular value decomposition (SVD) on W and factorize it into: • The scaled projection matrix M and the shape vector matrix B are given by: • G is the corrective matrix. • Additional rotational and basis constrain to compute G  M and B can be determined • Thus, the 3D shape modes can be computed from the 2D AMM shape modes and the 2D AAM tracking results.

17. Calculate the Corrective Matrix • Rotational constraints and basis constraints are used. • Rotational constraints (denote GGTby Q): • where ˜M2*i−1:2*irepresents the ithtwo-row of ˜M • c is the coefficient and R is rotation matrices • Due to orthogonormality of rotation matrices and Q is symmetric,

18. Calculate the Corrective Matrix • Basis constraints: • We find K frames including independent shapes and treat those shapes as a set of bases, the bases are determined uniquely, we have

19. Compute the 3D Model AAM shapes AAM appearance First three 3D shapes modes

20. Constraining an AAM with 3D Shape • Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes: • and the 2D shape variation of the 3D shape modes over all imaging condition is: • Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:

21. Fitting with 3D Shape Constraints • AAM fitting is to minimize: • I.e the error between the appearance and the original image • Impose the constrains of 2D projected 3D shape equals the 2D shape of AAM as soft constrains on the above equation with a large K:

22. Fitting with 3D Shape Constraints • Optimize for the AAM shape p, q, and the appearance λ parameters: • Calculate the square difference between the appearance and the original image and project the difference into orthogonal complement of the linear subspace spanned by the vectors A1, …, Al. • It is optimized by using inverse compositional algorithm, I.e. iteratively minimizing: • Then, solve the appearance parameters using the linear closed form solution:

23. Experimental Results Estimated 3D shape Estimates of the 3D Pose extracted from the current estimate of the camera matrix P Initialization 2D AAM After 30 Iterations Converged

24. Experimental Results Results of using the algorithm to track a face in 180 frame video sequence by fitting the model in each frame

25. DEMO Video -- 2D+3D AAMs

26. 2D+3D AAM Model Reconstruction Input Image Tracked result (2D+3D fit result) Shows two new view reconstruction 2D+3D model reconstruction

27. Compare the fitting speed with 2D AAMs • Frames per second of 2D+3D > 2D AAM • Iteration per second of 2D > 2D+3D, but 2D need more iteration for convergence

28. Conclusion • 2D AAMs can represent any phenomena that 3DMMs can. • Showed how to compute the equivalent 3D shape models from a 2D AAM with basis constrains, rotational constrains. • Improve the fitting speed of the 2D AAMs with 3D shapes constrains • 2D + 3D AAM is the ability to render the 3D model from novel viewpoint.

29. Q & A