MITRE Corporation

MITRE Corporation Pose Correction for Automatic Facial Recognition Team: Elliot Godzich, Dylan Marriner, Emily Myers-Stanhope, Emma Taborsky (PM), Heather Williams Liaisons: Josh Klontz ’10 and Mark Burge Advisor: Zachary Dodds

Automated Facial Recognition • Algorithmic identification of faces from images • Commercial systems exist; MITRE is building a U.S. system for flexibility and security • Unobtrusive relative to other biometric techniques, but with similar applications: • Fraud detection • Aid distribution • Law enforcement • National security

PrivacyConcerns

PoseCorrection • Off-pose images are a significant challenge for automated facial recognition • Many current algorithms, including MITRE's, do not include pose correction

PoseCorrection • Our approach to pose-correction involves finding and matching facial features in different images • Feature-finding and shape transformation, are also useful for other image-processing tasks

Problem Statement Our goal is to research, implement, and test a pose correction library that improves MITRE's existing facial recognition system. research: use and extend existing approaches implement: within MITRE's existing codebase test: using MITRE's test scaffolding and databases

Pose-correction pipeline Pixels Features Shape ASEF ASM Average of Synthetic Exact Filters Active Shape Model

Features • Facial features, or landmarks, can support both recognition and pose-correction • Features are based on spatial geometry and/or appearance

ASEF filtercreation For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye. training image (with known right-eye location) human-designed synthetic output

ASEF filtercreation We want to create a filter that exactly transforms a training image into the desired synthetic output = * training image (with known right-eye location) filter transforming the image at left into the image at right human-designed synthetic output

ASEF filtercreation In the Fourier domain, we want where Synthetic, Image, and Filter are the 2D Fourier transforms of the synthetic output, image, and filter. Complex division thus provides the filter:

ASEF filtercreation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this…

ASEF filtercreation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this… …to obtain the final filter?

ASEF filtercreation We take the average of all of the synthetic exact filters to define, here, a final right-eye filter We average 517 filters like this… …to obtain the final filter.

ASEF filterapplication We apply the filter in the Fourier domain; the peak in the spatial domain is a first estimate of the feature location The filter’s strongest response is most right-eye-eylocation in the image Unfiltered image Filtered image

Finaloutput

Gallery

ASEF results Many images' eyes are found quite accurately, but there are also some dramatic outliers: Units are fraction of interocular distance Percentage of pictures

Influence of ASEF’sGaussians synthetic outputs Radius, s = 2px Radius, s= 15px Radius, s= 25px ASEF filters

ASEF tradeoffs • Testing changes in Gaussian radii (s) Radius, s= 5px Radius, s= 20px more accurate localization – and more outliers the opposite tradeoff left eye error (units of interocular distance) left eye error (units of interocular distance)

ASEF improvements Using spatial heuristics as weights original 1.0 * original 0.5 * original Unweighted filtered image Spatially weighted filtered image

ASEF improvements Using spatial heuristics as weights With weighting Without weighting these clusters show mis-identifying the left or right eye left eye error right eye error (units of interocular distance) right eye error (units of interocular distance)

Pose-correction pipeline Pixels Features Shape ASEF ASM Average of Synthetic Exact Filters Active Shape Model

Active ShapeModels (ASM) • Describe classes of objects with varying shapes • Each shape is a set of points • ASM trains on a training set of shapes, creating a statistical model of the variation within that shape-family. geometric arrangement of facial features: eyes, nose, …

ASM, step 1: Procrustesfitting Procrustes analysis determines a scaling, rotation, and translation that best align a family of shapes. training data (hundreds of faces) mean face (not necessarily angry) We use this approach to align all of the training faces and extract the mean face.

ASM, step 2: Estimating face space • ASM uses principal components analysis to build a model of representative transformations of a face We use the most descriptive eigenvectors to describe the allowable shape domain.

ASM, step 3: Transformingfaces We can apply realistic transformations to the mean face along face space’s eigenvectors. Independent face-shape axes s = 0 (mean face) +3s -3s

Second semester plans • 1) Multi-resolution and weighted ASEF feature finding • 2) Adding pixel appearance to the ASM shape models • 3) Implementing pose-correction techniques (for pixels) shape space: yaw • First approach: apply ASM's transformations to generate poses at desired values of pitch and yaw.

Spring Schedule MITRE clinic, spring 2012 schedule

Questions? Pixels Features Shape ASEF ASM Average of Synthetic Exact Filters Active Shape Model

Gallery

Second semester plans • The spring term will focus on researching and implementing landmark-based pose correction techniques. • First approach: apply transformations given by ASM to generate poses at varying degrees of pitch and yaw. yaw pitch

ASEF results Comparing image pre-processing techniques Fraction of interocular distance Percentage of pictures

Active AppearanceModels (AAM) AAM adds color or grayscale information to ASM’s model. AAM can generate photorealistic faces, not just geometrically realistic ones. Shown here are faces generated by varying the central face’s appearance parameters by ±3 salong two appearance axes. • from T.F. Cootes, G.J. Edwards, and C.J. Taylor, Active Appearance Models

Face-recognitionpipeline Fall term’sfocus Input image Face detection Landmarking oldpipeline new pipeline Output ID Recognition Posecorrection Spring term’sfocus

NextSteps Pixels Improving ASEF: We will experiment with image processing techniques and weighting based on expected pose and image complexity Extending ASM: We will implement Active Appearance Models to extend face pose-generation to face image-generation. Implementing Pose Correction: ASEF and ASM provide a baseline approach: namely, transforming a query image to a standard face pose Features Shape

Automated Facial Recognition • Use of computers to identify faces from images • Commercial systems exist, but MITRE is developing a system specifically for the US for flexibility and security • Unobtrusive relative to other biometric techniques, but with similar applications:

Motivation: Uses for Biometrics • Law enforcement and national security • Fraud detection • Aid distribution • Social networking

ASEF improvements Mapping Without cosine window With cosine window

Face-recognitionpipeline Last semester This semester

Face-recognitionpipeline Input image Face detection Landmarking oldpipeline new pipeline Output ID Recognition Posecorrection

ASM, step 2: Mean-face finding We use this approach to align allof the training faces and thus find the mean face. Training Data Average Face

We gotthis… ?

Centered! • ASEF’sright-eyefilter in the spatialdomain

Face-recognitionpipeline

Landmarkingalgorithms Pixels Average of Synthetic Exact Filters (ASEF) Landmarks Active Appearance Model (AAM) Shape Model

ASEF filtercreation For each training image we create a synthetic output with the correct position of the feature, e.g., the right eye. training image (with known right-eye location) human-designed synthetic output

MITRE Corporation