“Now! – That should clear up a few things around here!”

1. “Now! – That should clear up a few things around here!”

2. The Challenge of Recognition

3. The Challenge of Recognition

4. The Importance of Recognition Navigation Foraging Social interactions Object recognition Offspring care Sexual selection Danger avoidance Pavlovian conditioning

5. The Importance of Recognition “Not only did Dr. P fail to see faces, but he saw faces when there were no faces to see. In the street he might pat the heads of water hydrants and parking meters, taking these to be the heads of children; he would amiably address carved knobs on the furniture and be astounded when they did not reply. Such incidents multiplied, causing embarrassment, perplexity and fear.” From ‘The man who mistook his wife for a hat’ by Dr. Oliver Sacks

6. Brain Mechanisms of Recognition Pawan Sinha Department of Brain and Cognitive Sciences MIT • Current understanding • Ongoing research • Real-world applications • What the future holds

8. Current Understanding – Lesion Studies Which parts of the brain are involved in visual recognition? Initial clues: Kluver-Bucy syndrome (1939) • Temporal lobe lesions in humans • and monkeys lead to: • Visual agnosia, and • Hypersexuality

9. Responses of a Patient with Temporal Lobe Damage (Farah, 1995)

10. Current Understanding – Electrophysiology Are there specific neurons involved in visual recognition? Face-cell 1 Face-cell 2 Desimone, 1984

11. Current Understanding – Brain Imaging Are there specific brain regions involved in visual recognition? “Face area” Kanwisher et al, 1997

12. Current Understanding – Summary The temporal lobe is involved in visual recognition.

13. Current Understanding – Summary The temporal lobe is involved in visual recognition. So what?

14. Current Understanding – Summary The temporal lobe is involved in visual recognition. So what? This doesn’t tell us how the brain recognizes objects.

15. How Does the Brain Recognize Objects? Hubel and Wiesel (1977) Primary Visual Cortex Oriented bar and edge detector neurons

16. A Proposal for How the Brain Recognizes Objects Marr (1979) Edges! Further processing

17. A Proposal for How the Brain Recognizes Objects Edge map Image 3D estimate Recognition! Binocular processing

18. A Proposal for How the Brain Recognizes Objects What underlies the researchers’ fascination for edges? Fine edges, but not the coarse structure, are expected to be invariant to imaging variations. In principle, this can make the recognition task very easy.

19. The Problem with Edges… In practice, fine edges turn out to be highly unstable. Even after more than two decades of research, we have been unable to create a robust recognition system based on the proposed model.

20. The Problem with the Rest of the Model… Recent experimental results suggest that recognition may precede the intermediate steps. Sinha & Poggio, Nature, 1996 Jones, Sinha, Poggio & Vetter, Current Biology, 1997 Bulthoff, Bulthoff & Sinha, Nature Neuroscience, 1998 Edge map Image 3D estimate Recognition! Binocular processing

21. The Million Dollar Question… ? Edge map Image 3D estimate Recognition! Binocular processing If recognition has to happen before the image can be finely analyzed, what then is the minimum image information that suffices for recognition?

22. Reducing Image Information An ecologically sound method: Progressive blurring (Equivalent to recognition at increasing distances) Criterion Recognition Performance Amount of Blur

23. Face Detection – Experimental Protocol Subject’s task: Given an image, to determine whether it is a face. Targets Distractors Random patterns Symmetric patterns False-alarms from an artificial detection system All stimuli are presented at several blur levels.

24. Face Detection – Results Hit Rate Criterion FA Criterion Amount of Blur (radius of Gaussian)

25. Face Detection – Analysis Are there any useful invariants at such high levels of blur? Yes! Sinha, 1994, 1995; Lipson, Grimson, Sinha, 1997 Concept learning algorithms Ratio-template: A stable face-signature that comprises pairwise ordinal brightness relationships http://www.ai.mit.edu/projects/cbcl/web-pawan/cartoon/cartoon.html

26. Face Detection – Analysis (contd.) Does the brain use a ‘ratio-template’ like invariant for detecting faces? There is no direct evidence yet. However, there is some indirect evidence. • Neurons in the visual cortex have the required properties • needed to implement this model. • Computer implementations of this model yield good • performance.

27. Performance of Ratio-templates

28. Benefits of Low-Resolution Approach • Permits face detection at a distance • Is robust to image degradations • Can generalize across facial variations • Is computationally simple

29. An Application of Our Face-detection System The Nielsen People Meter

30. Beyond Mere Detection – Face Recognition Low-resolution images may suffice for detection, but, surely, we must have fine detailed information for recognition… A popular approach to face recognition – feature matching

31. Beyond Mere Detection – Face Recognition Prediction of such an approach: Recognition Performance Amount of Blur

32. Face Recognition – Experimental Protocol Subject’s task: To recognize celebrity images subjected to different levels of blur. Stimuli: Blur series for 36 celebrity faces.

33. Face Recognition – Experimental Protocol

34. Face Recognition - Results 10 x 12 pixels/face

35. Face Recognition - Results 10 x 12 pixels/face 70 x 70 pixels/face

36. Face Recognition - inference Overall face configuration supports much more robust recognition as compared to individual features.

37. Face Recognition - inference Overall face configuration supports much more robust recognition as compared to individual features. “By and large, Dr. P recognized nobody: neither his family, nor His colleagues, nor his pupils. He recognized Einstein because He picked up the characteristic moustache, and the same thing Happened with one or two other people. ‘Ach, Paul!’ he said, When shown a portrait of his brother. ‘That square jaw, those Big teeth –I would know Paul anywhere!’ From ‘The man who mistook his wife for a hat’ By Dr. Oliver Sacks

38. Face Recognition - inference Overall face configuration supports much more robust recognition as compared to individual features. Sinha & Poggio, Nature, 1996

39. The Two Million Dollar Question Which aspects of facial configuration are important and which are not?

40. The Two Million Dollar Question Which aspects of facial configuration are important and which are not? Caricaturists probably know the answer to this question…

41. The Two Million Dollar Question Which aspects of facial configuration are important and which are not? Caricaturists probably know the answer to this question… …but it is difficult for them to articulate this intuitive knowledge, sometimes even to themselves.

42. Caricaturists in their own words… “When I’m having difficulty caricaturing someone, I just keep drawing by doing ten, twenty, thirty, forty sketches of the subject…” - Bill Plympton “I once spent about eighteen hours trying to caricature Barry Manilow. It was frustrating not being able to draw someone who is so funny looking in the first place.” - Taylor Jones

43. The Hirschfeld Project An attempt to make explicit the intuitive knowledge that caricaturists possess and, in the process, to gain insights into the brain’s face recognition strategies.

44. The Hirschfeld Project - Goal Given multiple caricatures corresponding to every face image in a large set… n caricatures m face images …the goal is to determine which facial measurements Caricaturists consistently emphasize (or de-emphasize) and how the extent of distortion relates to the deviation of a given face from the population average.

45. The Hirschfeld Project - Caveats “To effectively caricature a subject, I must feel that person’s personality. A caricature to me is not just a big nose, big ears And a big head on a little body – it is much more.” • Gerald Scarfe • Caricaturist

46. The Hirschfeld Project - Caveats “To effectively caricature a subject, I must feel that person’s personality. A caricature to me is not just a big nose, big ears And a big head on a little body – it is much more.” • Gerald Scarfe • Caricaturist “I wouldn’t be surprised if some day they would create a computer program to do caricature, but it would be terrible. what a caricaturist does involves his whole life experience, education and some indefinable thing that makes it all work.” • Robert Grossman • Caricaturist

47. The Hirschfeld Project – in 10 Steps Step 1: Caricature database creation (~50 caricaturists; ~100 faces of celebrities and others) Step 2: Assessment of recognizability of caricatures Step 3: Database digitization

48. The Hirschfeld Project – in 10 Steps Step 4: Measurements for each database entry

49. The Hirschfeld Project – in 10 Steps Step 4: Measurements for each database entry

50. The Hirschfeld Project – in 10 Steps Step 5: Average face construction and measurement