Making faces with ID: an eigenface based construction system

1. Making faces with ID: an eigenface based construction system Colin Tredoux David Nunez, Oliver Oxtoby, Bhavesh Prag & Taryn Sullivan Department of Psychology University of Cape Town February 2007

2. Introduction • Witnesses are often required to attempt the construction of a facial likeness • Various technologies are used, ranging from portraiture to Photo-composites to software compositors (e.g. Identikit, E-Fit) • Empirical research shows that these produce poor quality composites, which are difficult to match to target faces, especially from memory • We have developed ID, a configural, eigenface compositor as an alternative

3. Existing technologies • Sketch artists • Manual montage(featural) systemse.g. Identikit, • Electronic montage (featural) systems e.g. E-Fit; Mac-a-mug • Eigenface technologiese.g. Spot-it, Evo-fit, ID

4. Problems with the technologies • Poor quality composites - ‘unfacelike’ - even when the target face is in full view [this has improved recently] • Produce poor matching to target faces in laboratory studies • most empirical tests of composite systems show matching results little better than chance levels (see Davies & Valentine, 2007, for review) • Computerised systems no better than manual systems, at least until 2000 • e.g. Kovera, Penrod, Pappas & Thill (1997), Davies et al. (2000) • Most recent data from the Stirling laboratory (Frowd, Bruce, Hancock) suggest computerised featural systems have improved; achieve 20% naming rates

6. Reasons for composite problems • Dependency on featural composition • Most psychological research points to the importance of configuration in face recognition. • Inherently limited nature of facial feature databases • Absence of a representational or computational theory • e.g. a theory underlying the kinds of features that are likely to appear together, how many instances are needed etc. • Fallibility of human memory • Need for memory analysis of the task – is it recognition or recall, for instance?

7. ID - an eigenface compositor

9. Eigen • PCA is conducted on the normalized faces, to derive ‘eigenfaces’ • Each face in the dataset can then be expressed as a weighted sum of the eigenfaces • E.g. Face 1 = 0.12 E1 + 0.03 E2 + … + Ek • Faces not already in the dataset can also be expressed as a weighted sum, but the representation will have a small amount of error

10. ID • Search can proceed with either of two algorithms (PBIL vs Mchoice), in shape-free or shape-variable modes • User interface designed to be simple – goal is for use to be totally transparent to witness; no expert operator needed • Has not worked in practice, people get better with practice; addition of fine-grained controls

11. ID - key ideas 1 • A set of face images can be represented in a common principal component (PC) space • A new face - i.e. not in the original set - can be reconstructed by projecting the face into PC space • To put this another way, for every face, there is a representation by a unique set of PC coefficients • Therefore, the task of reconstructing a face is the same as finding such a set of coefficients • This can be conceptualised as an optimization problem, and standard optimization techniques can be applied

12. ID - key ideas 2 • PBIL (population based incremental learning) was identified as one useful technique by Rosenthal (1998), and applied to the problem • A random set of faces is generated, using random estimates for each of the PC coefficients • The witness chooses the best match (i.e. closest to the perpetrator) in this set of faces • The coefficients of the ‘best match’ are used to generate the next sample, or the witness decides that no better match is possible.

13. ID - key ideas 3 An alternate ‘variance reduction’ algorithm was devised, as simulations showed PBIL to be very slow

14. ID - advantages • Face construction is holistic; faces are always constructed as complete entities, and not as montages of features • ID enables the witness to use recognition memory, and does not depend on recall memory ?? • A good approximation to the target face is guaranteed, if the database of faces is a reasonable match to the group the target is drawn from • Statistical validity for composite faces, translates to more face-like images

15. Sample reconstruction – texture only

16. Sample reconstruction – texture only

17. Sample reconstruction – shape + texture

18. System development cycle Software & technical construction SimulationsEmpirical experiments (recognition tasks, similarity rating tasks)

19. Extensions to the basic model : Re-introducing shape • ID revised to introduce shape as a searchable and manipulable aspect • Two models • Shape and texture treated as separate spaces • Independent searches of texture and shape spaces • Shape and texture combined into an ‘appearance model’ (Cootes, 2001) • Search of joint space

20. Expert reconstructions; shape added

21. Prag & Tredoux, 2003 • Corporeal staged crime; four perpetrators • Interview conditions: In-view, Police interview, Cognitive interview • Composite systems: FACES vs ID • Similar outcome measures to previous studies • Lineup tasks • Similarity rating tasks • Mugshot tasks • Sorting tasks

23. In-view ID Composites and Task Means Sorting (Hit Rate%) 33.33 64.17 59.17 58.33

24. In-view FACES Composites and Task Means Sorting (Hit Rate%) 50.83 62.50 88.33 72.50

25. Back to the Feature… • Original starting point was to try and avoid featural reconstruction • A lot of witness feedback and direct observation persuaded us that we needed to put featural search into the system • The system underlying ID (i.e. image landmarking, warping, eigenface derivation, appearance model construction etc.) replicated per feature • i.e. separate searchable space for each feature, with same algorithms, procedures

26. Additional sullying of the system… • Search acceleration to be controlled by the witness (through a ‘similarity scaling’ control) • Starting point to be obtained by allowing a witness to select one of the faces in the set used to build the model(s)

27. Comparing composite systems 1:experienced operators • Four systems identified for testing • sketch artist [some data suggesting superiority] • Pro-Fit [gets good results in some UK studies] • Identikit [preferred system in South Africa] • ID [holistic system] • Each operated by experienced operators • 32 Participant witnesses exposed to one of four faces [2 white male, 2 black male], staged encounter in tutorial room • After 48 hour delay asked to reconstruct face image with assistance of (‘blind’) system operator

29. Results • Structured similarity rating task 326 evaluators recruited for online task, judged similarity of a composite to a set of face images, presented in sequence. The set included the target, and the foils were selected on a similarity gradient • Mugshot selection task 287 evaluators were shown 28 mugshot images + a composite, on a computer screen, and were asked to select all images that they could not exclude as possible matches for the composite

31. Correcting for ratings of foils

32. Mugshot selection task • Need to balance ‘hits’ against ‘false positives’ • Standardized weighted utility calculation (10*hits – 1*false positives)

33. Comparing composite systems 2:long delays, guided memory interview • Two systems identified for testing • FACES [widely used in US, SA] • ID [holistic system] • Each operated by student witnesses, with training and assistance from practiced operator • 48 Participant witnesses exposed to one of three white female faces in 20 minute numerology ‘encounter’ • After 5 week delay asked to reconstruct face image with assistance of (‘blind’) system operator, and with presence/absence of guided memory interview

35. Evaluations conducted with lineup task • One lineup constructed for each target [with low bias and high effective size estimates]

36. Results from lineup task

37. Next steps • Simplify the models • We currently base our models on face collections of approx. 350 • We use all the eigenvectors from the appearance models, makes for enormous complexity • Even if we took only the first 100 components, and allowed the coefficients to take only three values (-, , +) we would still have enormous complexity • A lot of the complexity is perceptually unimportant • We are busy planning a set of studies that will reduce the complexity • e.g. by constraining coefficient range, finding what coefficients are important, and so on.

38. Is the problem intractable? • Most new generation featural and configural systems are reasonably accurate at in-view reconstructions • Good reconstruction from memory is rare, and unpredictable, especially when encoding is relatively poor • More often than not the case with eyewitnesses? • Preliminary research with context-reinstatement interviews is not promising • Are we chasing vapours?