Digital Face Replacement in Photographs

1. Digital Face Replacement in Photographs CSC2530F Project Presentation By: Shahzad Malik January 28, 2003

2. Face Replacement Motivation Currently done manually by graphic artists using photo editing software An automatic system has many potential uses: Hollywood special effects �Personalized� movies Framing someone�

3. Required Components Need the following subsystems: Face detection (and tracking for videos) Head pose estimator Illumination extractor (*) Facial expression synthesis Merging/replacement algorithm (*)

4. Light Estimation Assuming a Lambertian reflectance model:

5. Approximate Skin Tone Cannot assume 3 basis images for arbitrary photographs Use an approximate image to generate basis

6. Fitting a Generic 3D Model Need geometry to create basis images Fit a generic 3D face mesh to images �Lift� a texture using planar mapping

7. Generate Basis Images Set 3 linearly independent light positions Relight skin tone model with each light

8. Determining the Coefficients Compute a least squares solution to:

9. Re-illuminating the Target Face Set intensities of the 3 light sources to the coefficient values Render the target face with these lights

10. Flesh Pixel Detection Match non-mesh skin pixels to new skin tone Use a histogram-based skin classifier

11. Histogram Matching Generate histograms for newly lit face Match the Gaussian distribution from original face to newly lit face For each flesh pixel in original image, choose a new color with a similar location on the Gaussian bell curve

12. Weighted Color Blending

13. Results

14. Results (continued)

15. Results (continued)

16. Results (continued)

17. Summary Presented a face replacement system Takes lighting and merging into account Future research areas: Face detection and tracking (for videos) Expression synthesis More sophisticated reflectance model Automatic and precise model-fitting