Facial Tracking and Animation

Facial Tracking and Animation Project Proposal Computer System Design Spring 2004 Todd Belote David Brown Brad Busse Bryan Harris

Problem Background • Speech Driven Facial Animation (PRISM) • Facial animation from processed speech • Previous Research (Jablonski & Zavala) • Low cost facial motion and speech processing using facial markers • An infrared camera is used with infrared reflectors to pick up facial markers • Microphone array for audio • Winnov capture card (640x480 at 30 fps) • Areas of Desired Improvement • There is no current recovery method for point loss • Feature points skewed in perspective create playback artifacts • There are only 22 feature points, which cannot fully describe a face • Initialization requires mouse-clicking the markers on the first frame • The current algorithm is costly

Problem Statement • Design a low cost computer system which can decode and analyze audio/video data in real-time and save the resulting analysis to disk.

Design Objectives • Analyze video at 30 fps; generate an FAP(Facial Animation Parameter) file. • Continue with current audio analysis. • All processing done in real-time.

Existing Solutions • Lin et al, from National Taiwan University use a system with mirrors and camera to determine (x,y,z) coordinates of feature points • Essa et al, from MIT Media Lab use computer recognition to analyze video in non real-time setting

Microphone Array, Camera, and Capture Card • Audio sample rates range from 8 to 48 kHz • Video sample rate will be 30 fps, and can be captured at 640x480 resolution • Capture card brings audio and video into computer in synchrony

IBM Pupil Cam • Camera designed to identify human pupils by emmitting infrared light using LEDs • We will use infrared reflective markers to track facial movements

Design Constraints and Feasibility • Cost and Speed • The system must run in real-time • Portability • Hardware specific system • Quicktime would make a more portable system • Usability • Point initialization • Freedom of movement for each user • Recovery from point occlusion

Alternative Solutions • Leave system as-is • Use mirrors to find (x,y,z) coordinates • Quicktime libraries • Individual facial templates

Design Validation • Can generate FAP file in real-time • Lost points can be recovered • User can rotate her head without data loss • Audio is analyzed in real-time • Automatic point initialization

Societal, Safety and Environmental Analysis • Primary use as a research tool • Low bandwidth face to face phones using texture maps • Low bandwidth phone support for the deaf • Audio-visual data recording

Management • Todd Belote – Data Aquisition • David Brown – Marker Initialization • Brad Busse – Marker Tracking Algorithms • Bryan Harris – Facial Relationships

Scheduling • Mondays – 4:20 to 6:30 PM • With Steve Ortiz • Wednesdays – 5:30 to 7:00 PM • With Steve Ortiz and Marco Zavala • Steve Ortiz – Project Advisor • Marco Zavala – Previous Project Owner

Scheduling – Gant Chart

Pert Chart

Facial Tracking and Animation