Advanced Programming for 3D Applications CE00383-3

1. Advanced Programming for 3DApplicationsCE00383-3 Motion Capture Lecture 2 Week 3 Bob Hobbs Staffordshire university

2. Definition of Motion Capture Motion capture is the recording of human body movement (or other movement) for immediate or delayed analysis and playback. “MoCap”

3. History • Late 1970’s – Rebecca Allen created the first primitive form of motion capture. “Rotoscoping” • 1980-1983- Tom Calvert invented “Geniometers” • 1982-1983- Ginsberg & Maxwell invented “Graphical Marionette” • 1988- deGraf/Wahrman invented “Mike the Talking Head”

4. 1988- Pacific Data Images invented “Waldo C. Graphic” • 1989- Kleiser- Walczak invented “Dozo” • 1991- Videosystem invented “Mat the Ghost” • 1992- SimGraphics invented “Mario” • 1992- Brad deGraph invented “Alive” • 1993- Acclaim invented two character animation motion capture

5. Applications • Entertainment • Medicine • Arts / Education • Science / Engineering

6. Entertainment: Live Action Films • Computer generated characters in live action films (e.g. Battle Droids and many others in Star Wars Prequels, Gullum in The Lord of the Rings, King Kong inKing Kong)

7. Entertainment: 3D computer animations • Characters in computer animated files (e.g. Polar Express, Monster House)

8. Entertainment: Video Games • Video games by Electronic Arts, Gremlin, id, RARE, Square, Konami, Namco, and others, (e.g. Enemy Territory)

9. Medicine • Medicine (e.g., gait analysis, rehabilitation) • Sports medicine (e.g. injury prevention,performance analyses, performance enhancement) Gait Analysis Service

10. Arts / Education • Dance and theatrical performances • Archiving (e.g., Marcel Marceau) OSU/ACCAD

11. Science / Engineering • Computer Science (e.g., human motion database, recognitions) • Engineering (e.g., Biped robot developments) • Ergonomic product design • Military (e.g., field exercises, virtual instructors, and role-playing games)

12. Types of mocap equipment • Magnetic systems • Mechanical systems • Optical systems

13. Magnetic systems • Utilize sensors placed on the body to measure the magnetic field generated by a transmitter source.

14. Magnetic systems • Advantages • Require no special lighting condition. • Sensors are never occluded. • Disadvantages • Require a metal-fee environment. • Electro-magnetic interference

15. Mechanical systems • Exoskeleton with angle sensors.

16. Mechanical systems • Advantages • Measure joint angles (no marker ID problems). • Sensors are never occluded. • Disadvantages • Breakable! • Configuration of sensors is fixed. • Constrains on joints.

17. Optical systems • The cameras are equipped with infrared LED's and filters. (Filters enhance the contrast of the image.) • The cameras see reflector markers.

18. Optical systems • Advantages • Higher sampling rate. • Larger capture space. • Disadvantages • Markers are sometimes occluded -> marker ID problems. • Provide only positional data -> joint angles need to be computed.

19. Typical Mocap system • Vicon optical system - Best system in Academia! • 8 high-speed MX 13 (up to 1000 fps) and 8 high-resolution MX 40 (4 million pixels) cameras. • Capture up to 5 performers at once.

20. Mocap animation • Motion capture animation is different from keyframe animation in terms of how motion is created. • Same principles apply to mocap animation & keyframe animation! • A combination of motion capture animation and keyframe animation is often used.

21. Keyframe animation • A keyframe is a drawing of a key moment in an animated sequence, where the motion is at its extreme. • Inbetweens fill the gaps between keyframes. • Every motion is created by animators.

22. Advantages of mocap animation • Faster to create (only if an established production pipeline exists.) • Secondary motions and all the subtle motions are captured -> more realism. • Physical interactions between performers and props can be captured.

23. Disadvantages of mocap animation • Cost. • Manipulating mocap data is often difficult -> Re-capturing or key framing a shot with bad data is often easier. • Mapping mocap data of a performer to a character with a different proportion often causes problems.

24. Process of MoCap • Data needs to be manipulated • Transform data into file format • Build specific models • Attaching the mesh

25. movement flowchart for games • Planning and Directing Motion Capture For GamesBy Melianthe Kines Gamasutra January 19, 2000URL: http://www.gamasutra.com/view/feature/3420/planning_and_directing_motion_.php

26. processing passive markers • each camera records capture session • extraction: markers need to be identified in the image • determines 2d position • problem: occlusion, marker is not seen • use more cameras • markers need to be labeled • which marker is which? • problem: crossover, markers exchange labels • may require user intervention • compute 3d position: if a marker is seen by at least 2 cameras then its position in 3d space can be determined

27. Collinearity Condition A B Direct Linear Transformation B=cA

28. Distortion Effects The DLT model of Abdel-Aziz/Karara only accounts for errors in perpendicularity between axes • Achieves accuracy of 1:2000 Other types of distortion • Radial • Pin Cushion • Barrel • Tangential • De-centering

29. 16 parameter DLT • Equations representing non-linear distortions • L12-L14 represent symmetrical lens distortion • L15 & 16 represent asymmetrical/ de-centering distortion

30. The Problem • The original DLT method contained 11 unknowns Uo,Vo 2 scaling factors Xo,Yo,Zo d 3 rotation angles • Only 10 are actually independent • d and the 2 scaling factors are mutually dependent variables

31. Modified Direct Linear Transformation • The Non-Linear constraint is added to the system to ensure orthogonality • To solve: • Compute the 11 parameters normally • Use the value of one parameter to remove it from the equation and estimate the 10 independent parameters • Solve for the value of the removed parameter using the values of the 10 independent parameters

32. Non-Linear MDLT • Adds correction for lens distortion to the MDLT method

33. retargeting: the character • the character is controlled by skeleton • to control the skeleton, need to specify joint rotations • muscles?

34. retargeting • capture motion on performer • positions of markers are recorded • retarget motion on a virtual character • motion is usually applied to a skeleton • a skeleton is hierarchical • linked joints • need rotation data! • need to convert positions to rotations

35. performer→ actor → character • the actor is used to convert marker positions to rotational data • markers are handles on the actor • actor should have similar proportions as the performer • joint rotations of the actor are applied to the character • there are still issues with proportions • Video example Alias Motionbuilder: actor and markers

36. retargeting problems

37. retargeting problems

38. Production pipeline overview • Calibrate the system. • Fit a generic skeleton to the subject’s proportion (subject calibration). • Capture shots & reconstruct 3D trajectories using the calibrated subject. • Link the subject specific skeleton to a CG character’s skeleton and edit motion (in MotionBuilder). • Add skin to the CG character, edit motion, and render (in Maya).http://atec.utdallas.edu/midori/Handouts/mocap_pipeline.htm

39. 2D Image 2D Camera Data 3D Markers Positions Trajectories System Calibration and Capturing and Processing Data C3D Circle fitting Triangulation Reconstruction Positional Data IQ Subject Calibration VSK VST(subject template) Actor Setup MOTION BUILDER • Capture Range of Motion (ROM) • Reconstruct trajectories of ROM • Label markers • Place markers on Actor ( intermediate skeleton ) Calibrated Subject The processes that you go through for each character Character Setup Mocap Pipeline Flow Chart HIK • Correlate : • Actor • (C3D, HIK) • & • Character • (FBX) • Edit motion • Bake motion data to the skeleton Markers Set Character Setup • Create a skeleton • Bind skin to the skeleton • Rig the character FBX MAYA Skeleton Only The processes that you repeat for each shot Rendering FBX • Marge the rigged character (MB) and the skeleton with motion data (FBX) • Edit motion (IK/FK blend, Trax) • Render MB Skeleton with motion data ( joint rotation angles ) Maya Scene File

40. References • Sturman, David. “A Brief History of Motion Capture for Computer Character Animation.” 13 March 1999. MEDIALAB. 25 Nov. 2005. http://www.siggraph.org/education/materials/HyperGraph/animation/character_animation • “When Motion Capture Beats Keyframing.” Sept. 1997. Game Developer. http://www.gdmag.com • http://www.kwon3d.com/theory/dlt/dlt.html • http://www.kwon3d.com/theory/dlt/mdlt.html • http://express.howstuffworks.com/gollum3.htm

41. Related Work • Motion capture search • Arikan and Forsyth 2002; Kovar et al. 2002; Lee et al. 2002; Li et al. 2002; Metoyer 2002 • Physics and motion capture • Rose et al. 1996 • Popovic and Witkin 1999; Pollard 1999; Pollard and Behmaram-Mosavat 2000 • Shapiro et al. 2003 • Physically based reactions • Oshita and Makinouchi 2001; Zordan and Hodgins 2002 • Komura et al. 2004; Mandel 2004 • Industry Software • Natural Motion’s Endorphin