Last tuesday , you talked about active shape models

1. Last tuesday, you talked about active shape models • Data set of 1,500 hand-labeled faces • 20 facial features (eyes, eye brows, nose, mouth, chin) • Train 40 individual regressors (x and y positions for each facial feature)

2. Problems/Solutions • Our regressors aren’t perfect, but they’re often wrong in conflicting ways • Facial feature positions aren’t independent of each other • eg. knowing something about the position of the eyes ought to provide a clue about the position of the nose • Face alignment needs to be more than the sum of its parts • Active shape model

3. Shape model • Given many examples of face shapes (and ignoring the image data) • Find a lower dimensional model that can still represent the high dimensional examples

4. The Active Shape Model framework • Build a model of face shapes to go along with our facial feature models, then combine the two into one output Alignment Shape Features

5. Shape model • Problem: We tried to describe our face using a 40-dimensional vector. That gives us a little bit too much flexibility: • Solution: If 40 dimensions are too many, let’s build a low-dimensional model Face Dog (?)

6. Learning a shape model • Use Principal Component Analysis (PCA) on the feature positions • Represent each example as a 40-dimensional vector, (x1, y1, x2, y2, …, x20, y20) • Subtract the mean from each vector • Find the most important (principal) axes • PCA gives us a total of 40 axes, but if we only select a few of them (say, the top six), we’ll get a low dimensional parameterization

7. Learning a shape model • Compute mean face shape

8. Learning a shape model • Compute principal components

9. ACTIVE APPEARANCE MODELS • “Interpretation through synthesis” • Form a model of the object/image (Learnt from the training dataset) I. Matthews and S. Baker, "Active Appearance Models Revisited," International Journal of Computer Vision, Vol. 60, No. 2, November, 2004, pp. 135 - 164.

10. Face appearance has two components: Model “texture” “Shape”

11. Active Shape Models training set

12. (Hard to automate) • Shape Vector provides alignment = 42 Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

13. Random Aside (can’t escape structure!) Antonio Torralba & Aude Oliva (2002) Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images. Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

14. Random Aside (can’t escape structure!) Tomasz Malisiewicz, http://www.cs.cmu.edu/~tmalisie/pascal/trainval_mean_large.png

15. “100 Special Moments” by Jason Salavon Jason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml

16. Given Shape Vectors (positions of set of keypoints) Collect keypoints from training data in Ms whose columns are training shapes. Subtract mean shape (s), then run SVD to get (Ms = Us S Vs’). si= s + Usai • Shape Vector provides alignment = 42 Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

17. “Shape free” Texture Models warp to mean shape

18. Once you have the shape model for each image, warp the model to the mean configuration: Now, all faces are the same size and shape and place. So you can create a column vector out of the relevant pixels. Put all those columns together (one for each training image) into Mt Subtract mean texture (t), then run SVD to get Mt = Ut S Vt’. ti = t + Utbi

19. Face model • All faces parameterized by mean texture t and mean shape s. • Each face has shape parameter ai, and texture paramter bi. • Given a shape and a texture, we construct a face as: • Texture = t + Utb • Shape = s + Usa • Make the texture, then warp it onto the new shape. • Awkward Mathematical notion. • Warp(Image, shape parameters) • W(t + Utb, s + Usa)

20. PCA Craziness The complete description of the face is [ai,bi]. Let’s put those into one column, make a matrix M for all those columns and run PCA. Why the heck would we do this?

21. Average faces. • Alignment is the key! 1. Warp to mean shape 2. Average pixels Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt

22. Random Aside • Enhancing Gender more same original androgynous more opposite D. Rowland, D. Perrett. “Manipulating Facial Appearance through Shape and Color”, IEEE Computer Graphics and Applications, Vol. 15, No. 5: September 1995, pp. 70-76

23. Playing with the Parameters First two modes of shape variation First two modes of gray-level variation First four modes of appearance variation

24. Essence of the Idea (cont.) • Explain a new example in terms of the model parameters

25. Active Appearance Model Search (Results)

26. Final Projects • 1 person: Implement any recent paper from CVPR/ ECCV/ ICCV, and apply to some new data. • 2-3 people, small research project; perhaps starting with recent paper and substantial extension, or one of the following ideas

27. Example 1 person projects… • Single Image Haze Removal Using Dark Channel Prior • http://research.microsoft.com/en-us/um/people/jiansun/papers/dehaze_cvpr2009.pdf

28. Example 2-3 person projects Matching historical and modern photos • http://www.thirdview.org/3v/rephotos/index.html

29. What year was this photo taken?