Soft Biometrics at CUBS

1. SoftBiometrics at CUBS Venu Govindaraju CUBS, University at Buffalo govind@buffalo.edu www.cubs.buffalo.edu

2. Background • Traits of biometrics • Universality • Distinctiveness • Permanence • Collectability • Acceptability • Present perfect? • No biometric is truly universal. It is estimated that 2-4% of the population have unusable fingerprints • Each biometric has a lower bound for errors (constraint of algorithm + individuality) • Individual biometrics need to be augmented by other biometrics (multi-modal) or traits (soft biometrics)

3. Soft Biometrics Definition[1] • Soft biometric traits are those characteristics that provide some information about the individual but are not distinctive enough to sufficiently differentiate any two individuals [1] Soft Biometrics • Not very distinctive • Can be used to augment regular biometrics • Not typically used during verification/identification • More intuitive than strong biometrics [1] A. K. Jain, S. Dass, K. Nandakumar, “Soft Biometrics for Personal Identification”, SPIE Defense and Security Symposium 2003

4. Soft Biometrics : Examples Other classification • Continuous: Age, Height, Weight etc. • Discrete: Gender, Eye Color, Ethnicity etc.

5. Motivation • Heckathorn[3] have shown that a combination of personal attributes can be used to identify the individual reliably • Binning and Indexing • Hardening primary biometric • Speech Recognition • Can be used to tune individual biometrics • Socially aware computing (call centers)?

6. Extracting Soft Biometric Traits • Devices • Color video • Stereo images • Challenges • Controlled vs Uncontrolled environment • Pose variations • Illumination variation • Complex backgrounds • Feature selection and extraction • Features used in traditional biometrics do not encode soft biometric traits • Decision systems (soft thresholds)

7. Problems in Representation Fuzzy class boundaries Purely statistical features

8. Soft Biometrics Research at CUBS • Speech • Gender Identification • Accent Identification • Face • Face Catalog: Semantic Face Retrieval • Gender Classification • Skin • Skin spectroscopy

9. Soft Biometric Traits in Speech • Gender • There exists a difference in the pitch period between genders • This difference is fundamental in the discrimination between males and females • Accent[1] • Temporal features: onset time, closure/voicing/word duration • Prosodic/Intonation slope patterns • Formant frequencies • Age • The average power measurement and speech rate are used as indicators for measurement of agedness in a speaker [1]A Study of Temporal Features and Frequency characteristics in American English Foreign Accent L.M. Arslan, J.H.L. Hansen , Journal of the Acoustical society of America, July 1997

10. Uses of Soft Biometrics in Speech Soft Biometrics for binning P(w|x1) P(w|x1y) Primary Biometric Soft Biometric(s) Soft Biometrics for improving accuracy

11. Loose Gender Classification (PITCH) Results • 3 Methods • Fast Fourier Transform • Linear Predictive Analysis • Cepstral Analysis • Data • 75 files • Males -41, Females -34 Male Low Male Medium Male High Female Low Female Medium Female High 132Hz 156Hz 171Hz 205Hz 230Hz 287Hz

12. Definition of Accent (linguistics) • An accent is the perceived peculiarities of pronunciation and intonation of a speaker or group of speakers • A foreign accent is defined in a way that the phonology of the spoken language is modified by the phonology of another language, more familiar to the speaker • 3 major language groups • American • Chinese • Indian

13. Proposed Approach for Accent • First identify the accent markers • Determine the effect of gender and co-articulation • Initially develop a text dependent model • Accumulate evidence over time • Features: • formants • phoneme duration • instantaneous (mel)cepstral slopes • HMMs

14. Accent Markers • A look at various non-native pronunciations of English • CHINESE • ‘r’ read sometimes as ‘l’ or ‘w’ • ‘v’ read as ‘w’ • ‘th’ read as ‘d’ • ‘n’ and ‘l’ often confused • Often drop articles like ‘the’ and ‘a’ • INDIAN SUBCONTINENT • Use of the rhotic ‘r’ • Use of rolling ‘l’ • Fast speech tempo with choppy syllables • Rhythmic variation of pitch Webster’s Revised Unabridged Dictionary Definition of non-native pronunciations of English – wordIQ.com

15. MALES – PHONEME CONTAINING ‘L’ American - Indian - F3 F3 PLEASE STELLA F2 F2 F3 F3 SLABS PLASTIC F2 F2

16. F3 F3 BRING RED F2 F2 F3 F3 FRESH F2 ASK F2 MALES – PHONEMES CONTAINING ‘R’ AND ‘AA’ American - Indian -

17. F3 F3 PLEASE STELLA F2 F2 F3 F3 RED ASK F2 F2 FEMALES – SEGMENTED PHONEMES ‘L’, ‘R’, ‘AA’ American - Indian -

18. Soft Biometrics for Law Enforcement Novel Forensic System

19. Law Enforcement Application: Face Catalog • User can select some facial feature to describe. • System will prompt the user after each query with the best feature for the next query.

20. Related Work • Identikit [1] composes faces by putting together transparencies of facial features. • Evofit [2], automate the process of identikits. • Phanthomas [3] face composition using elastic graph matching. • CAFIIRIS [4] and Photobook [5] use PCA for face composition and matching. • But general description of users are semantic! • V. Bruce, “Recognizing Faces”, Faces as Patterns, pp. 37-58, Lawrence Earlbaum Associates, 1988 • Frowd, C.D., Hancock, P.J.B., & Carson, D. (2004). “EvoFIT: A Holistic, Evolutionary Facial Imaging Technique for Creating Composites”, ACM TAP, Vol. 1 (1) • “Phantomas: Elaborate Face Recognition “.Product description: http://www.global-security-solutions.com/FaceRecognition.htm  • J. K. Wu, Y. H. Ang, P. C. Lam, S. K. Moorthy, A. D. Narasimhalu, ”Facial Image Retrieval, Identification, and Inference System” • A. Pentland, R. Picard, S. Sclaroff, “Photobook: tools for content based manipulation of image databases”, Proc. SPIE: Storage and Retrieval for Image and Video Databases II, vol. 2185

21. Face Catalog System Overview Semantic Face Retrieval System Input Image Face Detection Meta Database Lip Location and parameterization Face Image Database Semantic Description Eye Location Parameterization of other Features Sorted Images Query Sub-System Prompting Sub-System user

22. Enrollment Sub-System • Face Detection. • Lips and eye detection. • Locate and parameterize other features.

23. Query Sub-System • Pruning images based on descriptions given? • What if user makes a mistake in one of the description. • Ranking images based on their probability of being the required person is a better idea. • Bayesian learning can be used to update probability of each face being the required one. • Prompting users the feature with highest entropy at each step.

24. Example Query Query = [] Query = [Spectacles = Yes] Query = [Spectacles = Yes + Mustache = Yes] Query = [Spectacles = Yes + Mustache = Yes + Nose = Big] Probabilities of Faces

25. Top 5 Top 10 Top 15 Features Number of False Accepts Number of False Rejects Average no. of queries. 7.12  5.08 2.49 Spectacles 1 2 Mustache 2 4 Beard 4 0 Long Hair 2 8 Balding 1 0 Results • Results of Enrollment Sub-system (Database of 150 images) • Results of Query (25 users, 125 test cases)

26. Gender Classification in Images • Gender classification • Identifying male or female from facial image • Existing approaches • Geometric feature based [1]-[2] • Appearance feature based (raw data feature or PCA + classifier) [3] • Approaches using other features, e.g., wrinkle and skin color [4] [1] A. Burton, V. Bruce and N. Dench, “What’s the difference between men and women? Evidence from facial measurements,” Perception, vol. 22, pp.153-176, 1993. [2]R. Brunelli and T. Poggio, “Hyperbf network for gender classification,” DARPA Image Understanding Workshop, pp. 311-314, 1992. [3]B.A. Golomb, D.T. Lawrence, T.J. Sejnowski, “Sexnet: A Neural Network Identifies Sex from Human Faces,” Advances in Neural Information Processing Systems3, R.P Lippmann, J.E. Moody, D.S. Touretzky, eds. Pp. 572-577, 1991. [4] J. Hayashi, M. Yasumoto, H. Ito, H. Koshimizu, “Age and gender estimation based on wrinkle texture and color of facial images,”, Proceedings of 16th International Conference on Pattern Recognition, vol. 1, pp. 405 - 408, 11-15 Aug. 2002

27. Preprocessing (Face detection, normalization, etc.) Feature Extractor Using Gabor Wavelet SVM Classifier Raw Image Decision Gabor Feature based gender classification system

28. Facial image Normalization • Mapping feature points to fixed positions • Feature points • Centers of two pupils • Tip of the nose • Normalized image • 64 by 64 • Convert from color to grayscale by averaging RGB components

29. Gabor feature • Gabor filter and Gabor wavelet [B.S. Manjunath, et al, PAMI, 1996] Gabor Filter: Fourier Transform of g(x, y): Gabor Wavelet:

30. Gabor feature (cont.) • Redundancy reduction [B.S. Manjunath, et al, PAMI, 1996] • Let and denote the lowest and highest frequencies of interest • are determined by

31. Gabor feature (cont.) • Characteristics of Gabor wavelet • A powerful tool to capture changes of signals • Selective on certain frequency and orientation by setting parameters m, n • Gabor feature for gender classification • Gabor WT at 4 scalses, 4 orientations (m = 0, .., 3; n = 0, …, 3) • Each output image of Gabor WT (64 by 64) is divided into non-overlapping blocks of the size 2m+2 by 2m+2 (m: the scale number). • Average of magnitudes in each block as a feature • Total number of features

32. Gabor feature (cont.)

33. Classification • Features • 1360-dimensional training and testing vectors fed into SVM classifier • Classifier • SVM with Gaussian RBF kernel [6] (B. Moghaddam, et al, PAMI 2002) • Adjust γto minimize error rate • 1360 features from Gabor WT (in 4 scales, 4 orientations) of 64×64 input image • Training and testing vectors (of 1360 dimensions) normalized into unit vectors

34. Experimental Results • Dataset: AR face database [A.M. Martinez and R. Benavente, “The AR face database,” CVC Tech. Report #24, 1998] • Overall: 3265 frontal facial images including 136 Caucasian people (768 by 576, color) • Training: 2246 samples including 91 individuals • Testing: 1019 samples including 45 individuals • Test #1 • 393 regular samples. Accuracy: 96.2% • Test #2 • 626 irregular samples (occluded by dark sun-glasses or masks) Accuracy: 92.7%

35. Skin Spectroscopy • Measures the composition of the skin using IR(Deep tissue biometric) • Based on spectroscopy • Fool proof against fake fingers (Can detect liveness) • Can be easily integrated into solid state devices • Immune to surface degradations • Currently implemented by only one Vendor (Lumidigm Inc) Skin composition

36. Chromophores in skin • Melanin • Absorbs light at all wavelengths • Absorbance decreases with increase in wavelength • Hemoglobin • Strongest absorption bands in 405 – 430 nm and 540 – 580 nm. • Lowest absorption beyond 620 nm • Can be used for liveness testing • Collagen, Keratin, Carotene

37. Spectra of Melanin and Hemoglobin

38. Sample Skin Spectrum

39. Sample skin spectrum (contd.)

40. Sample skin spectrum (contd.)

41. Results so far • Soft classification based on skin color • Melanin index used as indicator of skin color • Spectral difference noticed between different skin locations on the same individual

42. Thank You ssc5@cedar.buffalo.edu