Facial Tracking and Animation

1. Facial Tracking and Animation Todd Belote Bryan Harris David Brown Brad Busse Facial Tracking and Animation

2. Problem Background • Speech driven facial animation • Correlate captured facial movements to audio patterns • Capture facial movements • Analyze corresponding audio Facial Tracking and Animation

3. Goals and Objectives • Develop an inexpensive, robust real-time system to track facial motion and process corresponding audio. The system must: • Cost around $1000 • Run on a personal computer • Allow for long periods of data acquisition • Handle head movements • Recover from point occlusion • Output only necessary information Facial Tracking and Animation

4. System Functionality • Capture Live Data • Record Video • Capture From Video • Generate FAP from output file • Process audio of .wav file Facial Tracking and Animation

5. POINT INITIALIZATION POINT TRACKING FAP GENERATION DATA ACQUISITION AUDIO PROCESSING System Description • Top level system organization • Illustrates data flow • Functional block division Facial Tracking and Animation

6. CAPTURE CARD EH WAV FILE AUDIO PROCESSING MICROPHONE CAMERA EH VIDEO PROCESSING FRAMEGRABBER AVI MOVIE FILE SW TIMER Data Acquisition Video Data – 320x240 at 30 fps (bmp) Audio Data – 16 bit at 16kHz (wav) Facial Tracking and Animation

7. Point Initialization and Tracking • Process frame data • Locate all areas in the frame which constitute a point • Identify points • Initialization – algorithm to determine identity • Tracking – algorithm to match current frame points with previous location of identified points Facial Tracking and Animation

8. Point Transform Approach: Criminisi et al. • Maps any arbitrary quadrilateral onto any other • This can account for all six degrees of freedom as well as perspective distortion, greatly simplifying the computation required to reorient the face • When using an orientation square that encompasses most of the face, this algorithm can be made as accurate as necessary Facial Tracking and Animation

9. Point Transform: Demo Actual Camera Data Re-oriented Processed Data Facial Tracking and Animation

10. FAP Generation and Audio Processing • FAP Generation • Process text file of point movement • Output FAE Engine compliant file of point displacements • Audio Processing • LPC • MFCC • Pitch • Power Facial Tracking and Animation

11. Environmental and Health Considerations • All hardware is off the shelf • No harm from infrared light • No harm from other products • Eg. Reflective markers Facial Tracking and Animation

12. Social, Political and Ethical Considerations • Provide low cost audiovisual capture • Increase research in field by removing cost barrier • Further advances • Eg. Phone for the deaf • No Ethical Issues • No Political affects Facial Tracking and Animation

13. Economics and Sustainability • No economies of scale due to narrow scope • IBM PupilCAM is hard to locate and therefore sustainability with current hardware is issue • Other cameras could provide the same function Facial Tracking and Animation

14. Thanks For the Contributions Facial Tracking and Animation