Exchanging Faces in Images

1. Exchanging Faces in Images SIGGRAPH ’04 Blanz V., Scherbaum K., Vetter T., Seidel HP. Speaker: Alvin Date: 21 July 2004

2. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

3. Introduction • Pasting somebody’s face into an existing image. • A novel type of image manipulation: • Always needs pairs of images with the same viewpoint and the same illumination. • The system only need one image, and can across large differences in viewpoint and illumination. Exchanging Faces in Images

4. Introduction (cont.) • Manual interaction: • Click on a set of about 7 feature points. • Mark the hairline in the target image. • Example • Two applications: • Virtual try-on for hairstyles • Face recognition Exchanging Faces in Images

5. Previous Works Exchanging Faces in Images

6. Previous Works (cont.) Exchanging Faces in Images

7. Previous Works (cont.) Exchanging Faces in Images

8. Previous Works (cont.) Exchanging Faces in Images

9. Previous Works (cont.) Exchanging Faces in Images

10. Previous Works (cont.) Exchanging Faces in Images

11. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

12. Morphable Models • A vector space of 3D shapes and textures. • Derived from 200 texture Cyberware (TM) laser scans. • 100 male and 100 female. • In a cylindrical representation with radii r(h, Φ) of surface points • 512 equally-spaced angles Φ. • 512 equally-spaced vertical steps h. Exchanging Faces in Images

13. Morphable Models (cont.) • Dense correspondence is computed automatically with an algorithm derived from optical flow. Exchanging Faces in Images

14. Morphable Models (cont.) After performing a PCA m = 149 Exchanging Faces in Images

15. Fitting Exchanging Faces in Images

16. Light Direction And Intensity Estimation Exchanging Faces in Images

17. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

18. Estimation • All parameters are estimated simultaneously in an analysis-by-synthesis loop. • All scene parameters are recovered automatically, starting from a frontal pose in the center of the image, and at frontal illumination. Exchanging Faces in Images

19. Estimation (cont.) • Cost Function Exchanging Faces in Images

20. Estimation (cont.) • The optimization is performed with a Stochastic Newton Algorithm. • The linear combination of texture Ti cannot reproduce all local characteristics of the novel faces. • Extract the texture by an illumination-corrected texture extraction method. Exchanging Faces in Images

21. References • A morphable model for the synthesis of 3D faces. SIGGRAPH’99, pp. 187–194. • Face recognition based on fitting a 3D morphable model. IEEE Trans. on Pattern Analysis and Machine Intell. 25, 9 (2003), 1063– 1074. Exchanging Faces in Images

22. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

23. Exchanging Faces • Both 3D shapes are aligned to each other in 3D with 3D Absolute Orientation Algorithm. • Both textures have similar illumination. • Illumination-corrected Texture Extraction Algorithm. • Render the face that was reconstructed from the source image with the rendering parameters that were estimated from the target image. Exchanging Faces in Images

24. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

25. Compositing Exchanging Faces in Images

26. Background Layer • The scene of target image, and the original person’s face, hair and body. • The novel face may be smaller than the original. • Solved by a background continuation method. • Based on a reflection of pixels beyond the original contour into the face area. Exchanging Faces in Images

27. Face Layer • The silhouette of this region: • Occluding contours. • Boundaries of hair regions that occlude the skin. • Mesh boundaries at the neck and the forehead. • Skin may be partly covered by hair. This hair would be mapped on the face as a texture. Exchanging Faces in Images

28. Hair Layer • Drawn in front of the face. • Can be used for all faces. • Automated classification of pixels into skin and hair is a difficult task. • Manually define alpha values for opacity. Exchanging Faces in Images

29. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

30. Applications • Current systems are restricted to frontal view of faces. Exchanging Faces in Images

31. Applications (cont.) Exchanging Faces in Images

32. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

33. Results Exchanging Faces in Images

34. Results (cont.) Exchanging Faces in Images

35. Results (cont.) Exchanging Faces in Images

36. Outline • Introduction • Morphable Models • Estimation • Exchanging Faces • Compositing • Application • Results • Conclusions Exchanging Faces in Images

37. Conclusions • A novel way of processing images on a high level. • Only needs simple manual processing steps. • For a wide range of applications. • Transferring technology from CG to CV. • Combines the benefit of image-based method with the versatility of 3D graphics. Exchanging Faces in Images

38. Future Works • Fully automated: • Detecting facial features. • Hair Segmentation. • Exchange faces in video sequences. • Tracking head motion. Exchanging Faces in Images

39. Thank you for your patience Exchanging Faces in Images

40. Example Exchanging Faces in Images

41. Feature Points Exchanging Faces in Images