460 likes | 1.04k Views
3D Face Modeling. Michaël De Smet. Topics to Discuss. 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization. Topics to Discuss. 3D Morphable Models 3D face reconstruction Face recognition Lip synchronization. 3D Morphable Models.
E N D
3D Face Modeling Michaël De Smet
Topics to Discuss • 3D Morphable Models • 3D face reconstruction • Face recognition • Lip synchronization
Topics to Discuss • 3D Morphable Models • 3D face reconstruction • Face recognition • Lip synchronization
3D Morphable Models • Statistical model of shape and texture • Derived from laser scans • USF DARPA HumanID 3D Face Database • Processing • Hole filling • Surface smoothing • Albedo estimation • Dense correspondence
-2s +2s -2s +2s … … 3D Morphable Models
Topics to Discuss • 3D Morphable Models • 3D face reconstruction • Face recognition • Lip synchronization
3D Face Reconstruction Fitting the 3DMM to one or more images of the same face • Scale • Rotation • Translation • Illumination ? ? • Shape • Texture Optimization problem with > 100 parameters
3D Face Reconstruction Feature points Feature alignment Fitting result
Dealing with Occlusions Without occlusion handling With occlusion handling
Topics to Discuss • 3D Morphable Models • 3D face reconstruction • Face recognition • Lip synchronization
Face Recognition Fit the 3DMM to an image of an unknown face • Scale • Rotation • Translation • Illumination ? ? • Shape • Texture Compare to database Recognition
Pose 1 100.0% Pose 2 100.0% Pose 3 N/A Pose 4 100.0% Pose 5 95.7% Face Recognition • In controlled settings, almost perfect recognition is possible Training view
Face Recognition • Uncontrolled environments are challenging • Face orientation unknown • Difficult illumination • Facial expressions • Occlusions • Low resolution • Motion blur • …
Face Recognition European parliament video: 21 persons, 86% correct
Face Recognition • VRT news broadcasts: 12 persons
Face Recognition • VRT news broadcasts: 12 persons 82.3% correct recognition
Topics to Discuss • 3D Morphable Models • 3D face reconstruction • Face recognition • Lip synchronization
Lip Synchronization • Speech driven animation • Texture based, i.e. shape is fixed • Strategy: • Extract 3D model of speaker’s face • Track rigid motion of the face in video • Extract texture for each frame • Compute PCA model of texture • Train ANN to link phonemes and PCA coefficients (visemes)
System Overview Automatic Phone Recognition Neural Network Face Synthesis Speech feature vectors Facial feature vectors
Training Setup Automatic Phone Recognition Neural Network Training Face Analysis Speech feature vectors Facial feature vectors
Video Processing • 3D face model acquisition • Rigid motion tracking • Normalized texture extraction • Texture feature extraction (PCA)
Conclusion • 3DMMs are a very powerful tool for face modeling • Many applications in computer vision and computer graphics