730 likes | 1.12k Views
Chap 6 Motion Capture (Mocap). What is motion capture?. Key-frames, forward kinematics, inverse kinematics A daunting task for creating physically realistic motion Requires a large amount of animation talent Record the motion and map it into a synthetic object Much easier Realistic motion.
E N D
What is motion capture? • Key-frames, forward kinematics, inverse kinematics • A daunting task for creating physically realistic motion • Requires a large amount of animation talent • Record the motion and map it into a synthetic object • Much easier • Realistic motion CS, NCTU, J. H. Chuang
What is motion capture? • track motion of reference points • convert to joint angles to drive • an articulated 3D model • a deformable surface CS, NCTU, J. H. Chuang
What is motion capture? • Objective • Reconstruct the 3D motion of a physical object and apply it to a synthetic object • Recording 3D live action, including • whole body • face • hands • animals CS, NCTU, J. H. Chuang
Applications • Animation • Special effects • Robot control • Interactive characters • Games CS, NCTU, J. H. Chuang
Computer Puppetry Shin et al., “Computer puppetry” CS, NCTU, J. H. Chuang
VideoOptical Motion Capture in Games CS, NCTU, J. H. Chuang
Video: Lord of the rings: Gollum CS, NCTU, J. H. Chuang
Video • Facial Mocap in King Kong CS, NCTU, J. H. Chuang
Video: Irobot CS, NCTU, J. H. Chuang
Pros and Cons of Motion Capture • Pros • All fine details of human motion will be recorded-- if they can be captured • Cons • Not so easy to • Edit • Generalize • Control • Not cheap CS, NCTU, J. H. Chuang
What is captured? • What do we need to know? • X,Y,Z • roll, pitch, yaw • Errors cause • Joints to come apart • Links to grow/shrink • Bad contact points CS, NCTU, J. H. Chuang
What is captured? • Large and small scale CS, NCTU, J. H. Chuang
How to use the data? • Off-line • Processed by filtering, inverse kinemtics • Produce libraries of motion trajectories • Choose among them • Blend between them • Modify on the fly • On-line (performance animation) • Driving character directly based on what actor does in real time CS, NCTU, J. H. Chuang
History of the technology • Recording motion for biomechanics • High accuracy • Fewer recorded pints • Hand digitizing film • Supplement with force plate, muscle activity • Computer animation • Rotoscoping • VR tracking technology • Less accuracy required • Fewer sensors CS, NCTU, J. H. Chuang
Optical Motion Capture (Passive) • Passive reflection • Camera • Infrared, visible, or near infrared strobes • High resolution (1 to 4 million pixels) • 120-240 frames/sec (max 2000 frames/sec) • Not outdoors • No glossy or reflective materials • Tight clothing • Occlusion of markers by limbs or props CS, NCTU, J. H. Chuang
Optical Motion Capture (Active) • Active output of the LED • No marker confusion problem • Outdoor capture • 120 frames/sec (128 markers or four persons) • 480 frames/sec (32 markers or single person) • 1/3 the cost of passive systems CS, NCTU, J. H. Chuang
Magnetic Motion Capture • Electromechanical transducer • Heavier sensors • Wires on body (wireless back to base station) • Limited accuracy (~10x less accuracy than optical) CS, NCTU, J. H. Chuang
Magnetic Motion Capture (cont.) • Smaller workspace • Sensors are the cost • Sensitive to EMI/metal • Relatively cheaper than optical device Ascension MotionStar Wireless CS, NCTU, J. H. Chuang
Mechanical Motion Capture • Subject wears an exoskeleton • No interference from light or magnetic field • No marker confusions • No range limit • Some restriction of movement • Absolute position is unknown CS, NCTU, J. H. Chuang
Mechanical Motion Capture • Data glove • Bend sensor + optical tracking • 6 DOF • video http://www.vrealities.com/glove.html CS, NCTU, J. H. Chuang
Technology Issues • Resolution/range of motion • Calibration • Accuracy • Marker movement • Sensor noise • Restrictions on the environment • Capture rate • Occlusion/correspondence CS, NCTU, J. H. Chuang
Markers - Examples CS, NCTU, J. H. Chuang
Skin Motion CapturePark & Hodgins, SIGGRAPH’06 • Uses a conventional optical motion capture system 40-60 markers CS, NCTU, J. H. Chuang
A dense set of markers Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang
A dense set of markers Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang
A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang
A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system Data collection and cleaning CS, NCTU, J. H. Chuang
A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system Data collection and cleaning Skin Animation CS, NCTU, J. H. Chuang
What is motion capture?Steps after motion is captured • Process 2D images to locate, identify, and correlated the markers in multiple video streams • Requires image processing techniques • Reconstruct the 3D locations of the markers • Requires camera calibration • Overcome numerical inaccuracies • Constrain the 3D marker locations to a model of physical system whose motion is being captured • Require satisfying constraints between relative marker positions CS, NCTU, J. H. Chuang
Camera calibration • Find the locations and orientations of cameras in world space and the intrinsic properties of the camera such as focal length, image center, and aspect ratio • Use a simple pinhole camera model • An idealized model • Usually sufficient for graphics and animation CS, NCTU, J. H. Chuang
Camera calibration Pinhole camera model CS, NCTU, J. H. Chuang
3D position reconstruction • Locate a marker’s position in at least two views relative to known camera positions CS, NCTU, J. H. Chuang
3D position reconstruction • Locate a marker’s position in at least two views relative to known camera positions CS, NCTU, J. H. Chuang
Fitting to the skeleton • After the motion of the individual markers looks smooth and reasonable, • Attach markers to the underlying skeletal structure that is to be controlled by the digitized motion • The position of each marker is used to absolutely position a specific joint • Is not straight due to noise, smoothing, and inaccuracy • Change in bone length can be significant • Markers are located outside the joints at the surface CS, NCTU, J. H. Chuang
Fitting to the skeleton • Given marker location and the relative distance from marker to the joint is not sufficient since the direction of displacement is not known • Put markers on both side of the joint • Work fine for simple joints, but not the complex joints (such as shoulder and spine) • Use normal of a plane formed by 3 markers • Wrist-elbow-shoulder markers plane normal • Offset the elbow marker in the direction of plane normal by the amount measured from the performer CS, NCTU, J. H. Chuang
Manipulating motion capture data • Processing the signal • Motion signal processing and Motion warping • Considers how frequency components capture various qualities of the motion. And warps the signal in order to satisfy user-supplied key-frame-like constraints • Retargeting the motion • Map the motion onto the mismatched synthetic character and modify it to satisfy important constraints • Combining the motion • Assemble motion segments into longer sequence CS, NCTU, J. H. Chuang
Processing the signal • Motion parameter values as a time-varying signal • Consider values of an individual parameter of the captured motion as a time-varying signal • Signal can be decomposed into frequencies, time-warped, interpolated with other signals • Multidimensional signal (vector-valued signal) • A function of time m(t): R -> Rn • Not really in Rn • 3 DOF in translation, 3 DOF in absolute orientation, many DOF in relative orientations CS, NCTU, J. H. Chuang
Processing the signal • Motion signal processing • Considers how frequency components capture various qualities of the motion. • Lower frequencies – represent base activities (walking) • Higher frequencies – idiosyncratic movements (walking with limp) • Motion manipulation/editing/warping • Warps the signal in order to satisfy user-supplied key-frame-like constraints. CS, NCTU, J. H. Chuang
Processing the signalMotion signal processing • Treat motion as a multi-dimensional signal • Low pass filtering • noise removal • High pass filtering • style change • High frequency component can be motion details, not just noise • Modify a motion through filtering is not easy • Physical constraints (joint limit, ground contact)? • Naturalness? CS, NCTU, J. H. Chuang
Processing the signalMotion signal processing • Frequency bands can be extracted, manipulated, and reassembled to allow the user to modify certain qualities of the signal while leaving others undisturbed. • Signal is successively convolved with expanded versions of a filter kernel. • Gains of each band are adjusted by the user and can be summed to reconstruct a motion signal. CS, NCTU, J. H. Chuang
Motion signal processing • PROBLEM: High frequencies can be important! • Getting rid of them makes motion look soggy • ANSWER: Do not over-apply LPF • Small amounts of Low-Pass Filtering • Noise modeling • Non-linear filters ? CS, NCTU, J. H. Chuang
Motion signal processing High frequencies are important! • Don’t occur often • Always significant • Impact • Rapid, sudden movement • Emphasis • Sensitivity of perception • Eye is sensitive to high frequencies CS, NCTU, J. H. Chuang
Motion manipulation/editing/warpingWhy? • What you get is not what you want • You get observations of the performance • Specific performer (a real human) • Specific motion • With the noise and “realism” of real sensors • You want animation • A character • Doing something • Or something similar but not the same CS, NCTU, J. H. Chuang
Motion manipulation/editing/warping • Manipulate time • Time warp / Speed control • m(t) = m0( f(t) ) • f : R R • Manipulate value • m(t) = f(m0(t) ) • f : Rn Rn CS, NCTU, J. H. Chuang
Motion manipulation/editing/warpingManipulating time m(t) = m0( f (t) ) • Time scaling f(t) = k t • Time shifting f(t) = t + k • Time warping • Interpolate a table • Align events • Speed control • Ease in/Ease out CS, NCTU, J. H. Chuang
Motion manipulation/editing/warpingManipulating values • Scale • Shift • Blending • Filtering • Transition between motions • Cyclification • Change style • Constraints on the motion • Concatenation CS, NCTU, J. H. Chuang
Motion manipulation/editing/warpingMotion warping • Witkin and Popovic, “Motion Warping,” SIGGRAPH’95 • Keyframes as constraints in smooth deformation • Keyframe placing the ball on the racket at impact CS, NCTU, J. H. Chuang
Motion manipulation/editing/warpingMotion warping CS, NCTU, J. H. Chuang
Manipulating motion capture dataRetargeting the motion • Map the captured motion onto the mismatched synthetic character and then modify it to satisfy important constraints • Constraints: • avoid foot penetration of the floor, avoid self-penetration, no feet sliding when walking • A new motion is constructed, as close to the original motion as possible, while enforcing the constraints. • New motion is formulated as a space-time, non-linear constrained optimization problem. CS, NCTU, J. H. Chuang