3D face verification using shape and texture 3D face registration and landmark localization

1. 3D face verification using shape and texture3D face registration and landmark localization Project participants: UniS: Josef Kittler, Miroslav Hamouz, Jose Rafael Tena BU: Lale Akarun, Berk Gökberk, Albert Ali Salah, Hatice Çınar Akakın, Bülent Sankur UNISS: Massimo Tistarelli, Manuele Bicego, Enrico Grosso

2. Overview • Objectives: • Landmarking and dense registration for 3D faces • Coupled registration and classification • Evaluation methodology • Work carried out so far: • Automatic facial landmarking (BU) • Data collection (UNIS) • Definition of evaluation methodologies (UNIS) • Fusion of 2D and 3D (UNISS) • Joint landmark estimation/registration (BU & UNIS)

3. Registration Methods • Rigid dense registration (ICP) • to each face in gallery • to an average face model (AFM) • benefits from facial landmarking • Nonrigid dense registration (TPS) • sample face to AFM • AFM to the sample face • requires especially good landmarks

4. Lighting variations In-plane and out-of-plane rotations Problems of facial feature localization Variability across subjects Expression differences Scale and pose differences Self-occlusion and accessories Holes or protrusions Sample density

5. Proposed Scheme

6. Structural Correction (GOLLUM)

7. GOLLUM Example

8. GOLLUM Example

9. GOLLUM Example

10. GOLLUM Example

11. 2D scheme Gabor wavelets in 8 orientations 3D scheme Depth map 3D-assisted 2D scheme Lambertian illumination model Albedo recovery with spherical harmonics basis projection Feature Extraction

13. Localization Results (t=3 pixels)

14. Localization Results (t=3 pixels)

15. Figueiredo & Jain vs. IMoFA-L

16. Data collection • 3dMD sensor: texture and shape in good correspondence • Focus on expression variability

17. Evaluation methodologies • FRGC database- currently largest 3D face database - has its own protocol using one-to-one comparisons => unsuitable for verification (no client model) • Verification protocol on FRGC defined allowing building client models (Surrey protocol)

18. Surrey protocol • Training & validation (threshold tuning): 163 clients-3 training images: 2D/3D per client -2 validation images: 2D/3D per client 45 validation impostors-2 validation images: 2D/3D per impostor • Test: 925 client accesses 14018 impostor accesses -test impostors different from validation impostors • World model: 168 people, 7 images: 2D/3D scans per person

19. Combined 3D Landmarking & Registration • Initial experiments with confidence based ranking of landmarks • Iterative landmark correction by global matching optimization • Preliminary results promising Deformed average model Sample face With iterative scheme

20. 2D-3D Fusion • Rationale: • Typical fusion schemes [Bowyer et al. 06] • In feature level fusion there is a single matching process • In other fusion schemes, shape matching and texture matching are separated processes, combined later Matching decision Shape score Shape score fusion Decision fusion Matching Texture score decision Texture Final decision

21. 2D-3D fusion • One matching process could pass information to the other • We tested a simple scheme injecting info from shape matching to texture matching Matching decision Shape score Shape Info score fusion Decision fusion Matching Texture score decision Texture Final decision

22. 2D-3D fusion • Shape matching process: • Template and test shapes are registered • Matching score: average of pairwise distances between corresponding points • By-product: confidence of registration in each point • How distant is the corresponding point • How similar are the two face shapes in that point

23. 2D-3D fusion • Texture matching process: • Basic rule: average of pairwise distance between “corresponding patches” • “Corresponding patches”: points extracted in the neighbourhood of two corresponding points (as depicted by shape) • 3D driven rule (info injection): only K most confident points are used in the matching (confidence is derived from the shape matching process) • Texture matching is performed only on the parts of the face with similar geometry (best aligned parts) • Dissimilarity in geometry is already captured by shape • Texture matching is used to “confirm” or to “refute” high shape similarities

24. 2D-3D fusion • Preliminary results • Subset of FRGC v.2: 20 subjects (5 scans each) • Cross Validation Leave One Out Recognition • Score level fusion (sum + min-max normalization)

25. 2D-3D fusion • Positive comments: • Injecting information into texture matching before fusion seems beneficial (a thorough experimentation is needed) • Texture matching is performed using only 40% of the points (computational saving) • Negative comments: • “We trust more in shape” • Accurate pairwise registration is needed • Next Issue: • Is it possible to reverse the process (texture matching driving the shape matching?) • Better: can we design a mutual interactions system?

26. References • Çınar Akakın, H., A.A. Salah, L. Akarun, B. Sankur, “2D/3D Facial Feature Extraction,” SPIE Conference on Electronic Imaging, 2006 • Hamouz, M., J. R. Tena, J. Kittler, A. Hilton, and J. Illingworth, “3D Assisted Face Recognition: A Survey”, a chapter in “3D Imaging for Safety and Security”, Springer 2006 (to appear). • Hamouz, M., J. R. Tena, J. Kittler, A. Hilton, J. Illingworth, “Algorithms for 3D-Assisted Face Recognition”, in IEEE Signal Processing and Communications Applications Conference2006. • Salah, A.A., H. Çınar, L. Akarun, B. Sankur, “Robust Facial Landmarking for Registration”, Annals of Telecommunications special issue on Multimodal Biometrics, 2006 (to appear). • Salah, A.A., L. Akarun, “3D Facial Feature Localization for Registration,” International Workshop on Multimedia Content Representation, Classification and Security, 2006. • Salah, A.A., L. Akarun, “Gabor Factor Analysis for 2D+3D Facial Landmark Localization,” IEEE Signal Processing and Communications Applications Conference, 2006. • Tena, J.R., M. Hamouz, A. Hilton, J. Illingworth, “A Validated Method for Dense Non-rigid 3DFace Recognition”, Int. Conf. on Advanced Video and Signal Based Surveillance, 2006.