Preliminary Assessment of Discrimination of Twins in Photographs based on Facial Blemishes

1. Preliminary Assessment of Discrimination of Twins in Photographs based on Facial Blemishes Nisha Srinivas1, Matthew Pruitt1, Gaurav Aggarwal1, Patrick Flynn1, Richard Vorder Bruegge2 1University of Notre Dame 2Frederal Bureau of Investigation, Digital Evidence Lab

2. Problem Statement • Investigate the usefulness of facial blemishes to distinguish between identical twins • Moles, Freckles, Scars, etc     • Determine • Whether facial blemishes and locations can be used to distinguish between identical twins • Whether the distributions of facial blemishes are “more similar” for identical twins than unrelated persons?

3. Freckle and Freckle Group Lightened Patch Darkened Patch Mole Raised Skin Scar (Round) Pockmark Pimple Facial Blemishes • Types of facial blemishes • Mole • Freckle • Freckle Group • Pimple • Darkened Patch • Lightened Patch • Splotchiness • Birthmark • Raised Skin • Pockmark • Scars • Linear • Round

4. Proposed System Overview Manual Annotation Feature Extraction Geometric Normalization Performance Evaluation Biometric Verification Point Cloud Matching

5. Manual Annotation Display Module Annotation Module Tool Module

6. Facial Blemishes Identified by Observer 1

7. Facial Blemishes Identified by Observer 2

8. Facial Blemishes Identified by Observer 3

9. Facial Blemishes Identified by Observers Total Number of Facial Blemishes Annotated by each Observer Observer 1: 3785 Observer 2: 2311 Observer 3: 5100

10. Facial Blemishes Matching N Nodes M Nodes Moles

11. Matching Contd. • The Edges in the bipartite graph correspond to potential matches • Each potential match has a cost associated with it which is a function of the euclidean distance between the centroids of the blemishes being compared.

12. Matching Contd. Match Match Similarity metric=Number of matches/Max(N,M)

13. Data • Twin face images were collected at the Twins Days Festival in Twinsburg, Ohio in August 2009. • High Resolution Images: 4310 rows x 2868 columns • Dataset Attributes • Frontal (yaw=0), Indoor, No Glasses, Neutral Expression • Number of Images: 295 • Number of Subjects: 152 • Number of Twins Pairs: 76 • Terminology • Target set: “gallery” of persons to be recognized • Query set: a set of images of unidentified persons to be matched against the target set

14. Experimental Setup • Perform two different experiments • Individual Observer Analysis • Query set and Target set are annotated by same observer • Inter-Observer Analysis • Query set is annotated by one observer and the Target set is annotated by another observer • Observer 1 vs Observer 2 • Observer 2 vs Observer 3 • Observer 3 vs Observer 1

15. Subset of Facial blemishes • FM={moles, freckles, freckle group, pimple, birthmark, darkened patch, lightened patch, splotchiness, raised skin, pockmark, scar round, scar linear} • FM1=FM-{pimple} • FM2={moles, freckles} • FM3={moles, freckles, pimple}

16. Target Set Twins vs Twins Setup: Query Set Match Comparison Subject 1, Twin A Non-Match Comparison Subject 2, Twin B Subject 3, Twin A Subject 4, Twin B

17. Individual Performance Evaluation- Observer 3

18. Target Set All vs All Setup: Query Set Match Comparison Subject 1, Twin A Non-Match Comparison Subject 2, Twin B Subject 3, Twin A Subject 4, Twin B Subject 5, Twin A

19. Individual Performance Evaluation- Observer 3

20. Comparison: All vs All and Twins vs Twins

21. Inter-Observer Performance Degradation in Performance when comparing facial marks annotated by different observers

22. Conclusion • There appears a correlation between the distribution of facial blemishes across twins. • The number of facial blemishes across twins appears to be similar. • Facial blemishes can be used as a potential biometric signature. • Consistent annotation is a challenging process • It is difficult to achieve consistency

23. Thank You This research was supported by • NIJ/OJP award 2009-DN-BX-K231 • FBI through TSWG/ARMY RDECOM contractW91CRB-08-C-0093