Digital Face Cloning

1. Digital Face Cloning David Bennett Christophe Hery Steve Sullivan George Borshukov J.P. Lewis Lance Williams Paul Debevec Fred Pighin Li Zhang

2. Seattle Boston San Francisco Los Angeles David Bennett Christophe Hery Steve Sullivan George Borshukov J.P. Lewis Lance Williams Paul Debevec Fred Pighin Li Zhang

3. Definition • Process of capturing an actor’s performance and/or likeness in a digital face model

4. E.g. photogrammetry (input)

5. E.g. photogrammetry (output)

6. Motivation • Replicate a live actor (e.g. celebrity) • Stunts • Extreme makeup • Younger/older/slimmer clone • Creating realistic faces from first principles is difficult (digitizing is easier than modeling)

7. Purpose of the course • Review the field • Suggest directions

8. Digital Clones: History

9. CG Humans in the movies • First 3D actor representation in the movies: Peter Fonda in FutureWorld (1976!) • Not intended to be photorealistic

10. 1990s

11. 1990s • Terminator 2 (brief, texture projection) • Jurrasic Park (legs only, does not count) • Titanic (distant) • Startup attempt: Virtual Celebrity/Global Icons, purchased virtual rights to Sammy Davis Jr., others

12. 1998: Hollywood is ready • "I don't see them as competing with real actors,” Richard Masur, head of Screen Actors Guild • Projected uses: de-aging, celebrity vacuum ads. • A television series with a virtual cast was considered

13. Recent and current

14. Recent and current • Space Cowboys (2000), Enemy at the Gates (2001): brief and distant, but successful • Disney Facial Cloning test (afternoon session) • Matrix sequels (2003): “superpunch” full-screen for 13 seconds (afternoon session) • SpiderMan 2 (reflectance capture) • Lemony Snicket: CG baby in several shots (afternoon session) • Discovery Channel Churchill/Hitler clones (MPC)

15. Observation • There has not been a “breakthrough” introduction of digital clones • Rather, clones were initially used in the distance, and briefly; they are getting closer, and withstanding longer study. • Current state of the art: full screen for several seconds (see afternoon Perception session)

16. Syllabus – morning (theory) 8:30 Introduction and Overview Fred Pighin 9:00 Face Scanning and Dense Motion Capture TechnologiesLi Zhang 10:00 Break or Mathematical Background (schedule change) 10:15 Reflectance Modeling and Capture Paul Debevec and J.P. Lewis 11:15 Facial Parameterization and Cross-Mapping J.P. Lewis and Fred Pighin 12:15 Lunch

17. Syllabus – afternoon (practice) 1:30 Case study: Face Cloning at ILM Steve Sullivan and Christophe Hery 2:15 Perception Experiment (schedule change) 2:30 Case Study: Disney Facial Cloning Lance Williams 3:15 Case study: Leaping the Uncanny Valley with Data   (Face Cloning in the Matrix sequels) George Borshukov 4:00 Break 4:15 Case study: Polar Express David Bennet 5:00 Perception of Facial Realism J.P. Lewis 5:15 Panel on the Future of Digital Face Cloning All

18. Face Cloning at Siggraph • Courses • 10. Realistic Materials in Computer Graphics • 28. From Mocap to Movie: The Making of "The Polar Express“ • 29. High-Dynamic-Range Imaging and Image-Based Lighting • Papers(Skin & Faces session on Monday) • Automatic Determination of Facial Muscle Activations From Sparse Motion Capture Marker Data • Face Transfer With Multilinear Models • Postproduction Relighting and Reflectance Transformation with Time-Multiplexed Illumination • Sketches • Image-Based Rendering From a Sparse Set of Images • Implementation of Modeling Hair From Multiple Views • Model Flowing: Capturing and Tracking of Deformable Geometry • Performance Geometry Capture for Spatially Varying Relighting • Joint Motion and Reflectance Capture for Relightable 3D Video