Berk G ö kberk Bo ğaziçi University – Perceptual Intelligence Lab Turkey

1. Principles of 3D Face Recognition Berk Gökberk Boğaziçi University – Perceptual Intelligence Lab Turkey

2. Promises and motivations • 2D face recognition still requires help • Pose, expression, illumination variations • Possible solutions: other modalities? • Video, infra-red, stereo, 3D • Promises of 3D facial recognition • High-security applications • 3D shape information invariance • Pose and illumination problems can be solved • Better facial feature localization

3. Some scenarios • 3D-to-3D • 2D-to-2D via 3D • 2D-to-3D or 3D-to-2D

4. How do we get 3D facial data? • Stereo cameras • Quality: low to medium • Speed: fast • Problems: reconstruction • Structured-lights • Quality: medium • Speed: fast • Problems: intrusive • Laser scanners • Quality: high • Speed: slow • Problems: intrusive, • Shape from {shading, motion, video} • N/A

5. 3D Facial Recognition Pipeline Face Normalization/Alignment 3D Face Detection Pre-proc. Features Pattern Classifier Point Clouds Depth Images Cropping Coarse Alignment Noise Removal Hole Filling Smoothing Landmark Finding Fine Alignment

6. 3D Face Detection • This problem has not been touched so far! • Simple heuristics such as nose tip • In complex scenes, curvature analysis is generally used Ref: 3D face detection using curvature analysis, Alessandro Colombo, Claudio Cusano, Raimondo Schettini, Pattern Recognition 39 (2006) 444 – 455

7. Pre-processing • Artifact removal • Noise removal: spikes (filters), clutter (manually), noise (median filter) • Holes filling (Gaussian smoothing, linear interpolation, symmetrical interpolation)

8. Face Normalization/Alignment • Coarse alignment by • Centre of mass, • Plane fitted to the data • Facial landmarks (eyes, nosetip) • Fine alignment • ICP • Warping • Elastic deformations

9. 3D Facial Features

10. -ICP gives the distance -Hausdorff -Too many points PC

11. -ICP gives the distance -Hausdorff -Too many points PC SN -Enhanced Gaussian Image -Too many normals

12. -ICP gives the distance -Hausdorff -Too many points PC SN -Enhanced Gaussian Image -Too many normals PRO -Sparse -Easy to compare -Not fully descriptive

13. -ICP gives the distance -Hausdorff -Too many points PC SN -Enhanced Gaussian Image -Too many normals PRO CURV -Landmark detection -Segmentation -Sensitive to noise and the quality of data Principal directions -Sparse -Easy to compare -Not fully descriptive Mean Shape Index Gaussian

14. -ICP gives the distance -Hausdorff -Too many points PC PCA DI SN -Benefit from 2D literature -Easy to fuse with texture -Applicable to 2.5D only -Enhanced Gaussian Image -Too many normals PRO CURV -Landmark detection -Segmentation -Sensitive to noise and the quality of data Principal directions -Sparse -Easy to compare -Not fully descriptive Mean Shape Index Gaussian

15. PCA -ICP gives the distance -Hausdorff -Too many points TEX PC Gabor PCA DI SN -Benefit from 2D literature -Easy to fuse with texture -Applicable to 2.5D only -Enhanced Gaussian Image -Too many normals PRO CURV -Landmark detection -Segmentation -Sensitive to noise and the quality of data Principal directions -Sparse -Easy to compare -Not fully descriptive Mean Shape Index Gaussian

16. Baseline algorithms • When you design a new system, which algorithms should be selected to show your algorithm’s superiority? Baseline Algorithm 1 PCA Match score for texture Texture & Shape Use weighted sum PCA Match score for shape Baseline Algorithm 2 Face A Perform ICP Output the surface matching error Face B

17. What are the scenarios tested? • The discriminative power of texture and shape? Taken from: “Three-dimensional face recognition”, Bronstein, A.M., and Bronstein, M.M., and Kimmel, R. International Journal of Computer Vision 2005, Vol.64 No.1 p.5-30

18. What are the scenarios tested? • The discriminative power of texture, shape, or texture+shape? • Pose variations • No disciplined analysis to compare 2D and 3D under pose variations • Expression variations • Most of the databases do not contain expression variations • No comparison to 2D • Illumination variations • Image relighting • Albedo estimation

19. Open Issues & Challenges • Uncontrolled acquisition • Non-cooperative • Different lighting conditions • Texture map + shape map inconsistenies • Real-time 3D video data • Computational complexity • Issues related to performance assesment • Publicly available standard face databases • Quality (resolution) of the data? • Artifacts such as eyeglasses Images taken from: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition, Kevin W. Bowyer, Kyong Chang, Patrick Flynn, Computer Vision and Image Understanding 101 (2006) 1–15

20. Quick notes on 3D acquisition systems & 3D face databases

21. 3D Acquisition Systems • Face specific • Biometrics • A4Vision • Geometrix • Modeling • Cyberware • Genex • Inspeck • Medeim • Breuckmann • General sensors • Minolta V-910

22. 3D Face Databases • UND • 275 subjects, 943 scans • Shape + texture • FRGC • 400 subjects, 4007 scans • Shape + texture • 3D_RMA • 120 subject, 6 scans • Shape only • GavabDB • 61 subjects (9 scans) • Shape only • Pose, expression variations • USF database • 357 scans • 3DFS generator • Custom face databases • 12 persons to ~6000 persons (A4Vision) UND USF GavabDB 3DFS

23. Conclusions • 3D face recognition systems were proposed to overcome expression, illumination, and pose challenges • Illumination correction is simpler • Facial landmark localization is better • Baseline recognizers • Combining depth maps with texture channel at the decision level • The core algorithm, ICP, has limited capabilities • Not suitable for non-rigid deformations • Lots of research on shape channel representation • Few in combining shape + texture

24. References • Boğaziçi University • Perceptual Intelligence Lab (PILAB) • Signal and Image Processing Lab (BUSIM) • Relevant Papers: • Comparative analysis of decision-fusion methods for 3D face recognitionB. Gökberk, L. Akarun, submitted for publication. • Exact 2D-3D Facial Landmarking for Registration and RecognitionSalah, A.A., H. Çınar, L. Akarun, B. Sankur, submitted for publication. • 3D Shape-based Face Representation and Feature Extraction for Face RecognitionB. Gökberk, M. O. İrfanoğlu, L. Akarun, Image and Vision Computing (accepted). • 3D Face Recognition by Projection Based MethodsH. Dutağacı, B. Sankur, Y. Yemez, in SPIE Conf. on Electronic Imaging, 2006. • 2D/3D facial feature extractionÇınar Akakın, H., A.A. Salah, L. Akarun, B. Sankur, in SPIE Conf. on Electronic Imaging, 2006. • Selection and Extraction of Patch Descriptors for 3D Face RecognitionB. Gökberk, L. Akarun, ISCIS 2005, LNCS, Vol. 3733 Springer 2005, pp. 718-727. • 3D Face Recognition for Biometric ApplicationsL. Akarun, B. Gökberk, A. A. Salah, EUSIPCO2005, Antalya, Turkey. • Rank-based Decision Fusion for 3D Shape-based Face RecognitionB. Gökberk, A. A. Salah, L. Akarun, AVBPA 2005, LNCS, Vol.3546 p.1019-1028. • 3D Shape-based Face Recognition using Automatically Registered Facial SurfacesM. O. İrfanoğlu, B. Gökberk, L. Akarun, ICPR2004, pp.183-186 • Contact: • Berk Gökberk, e-mail: gokberk@boun.edu.tr

25. Additional Material

26. Face Recognition Grand Challenge 3D Shape+Texture 3D Texture Only High Resolution 2D vs 2D 3D Shape Only Reference: Jonathon Phillips, FRGC Workshop, CVPR’05