380 likes | 504 Views
Enforcing Constraints for Human Body Tracking. David Demirdjian Artificial Intelligence Laboratory, MIT. Goal. Real-time articulated body tracking from stereo accounting for constraints on pose. Approach.
E N D
Enforcing Constraints for Human Body Tracking David Demirdjian Artificial Intelligence Laboratory, MIT
Goal • Real-time articulated body tracking from stereo accounting for constraints on pose WOMOT 2003
Approach • Differential tracking: assuming the articulated body pose Pt-1 is known, estimate the pose Pt (or equivalently the set of limb rigid motions xi=(ti,wi)between poses Pt-1 and Pt) that minimizes the distance between the articulated model and the observed 3D data tracking as a constrained optimization problem WOMOT 2003
Approach • Differential tracking: assuming the articulated body pose Pt-1 is known, estimate the pose Pt (or equivalently the set of limb rigid motions xi=(ti,wi)between poses Pt-1 and Pt) that minimizes the distance between the articulated model and the observed 3D data tracking as a constrained optimization problem • Solve unconstrained optimization problem • Project solution on constraint surface WOMOT 2003
Projection-based approach D (unconstrained optimum) D* human motion manifold WOMOT 2003
Approach • Estimate limb motionsxi=(ti,wi) independently using standard multi-object tracking algorithm • Projection: findthe closest body motion D*=(xi’) to D=(xi) that satisfies human body constraints: • articulated constraints • other constraints: joint limit, … WOMOT 2003
Previous work • Particle sampling: Sidenbladh & al. ECCV’00 Sminchisescu & Triggs CVPR’01 • Differential tracking: Plankers & Fua ICCV’99 Jojic & al. ICCV’99 Delamarre & Faugeras ICCV’99 WOMOT 2003
Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003
Multi-object tracking • Assuming the articulated body pose Pt-1 is known, estimate the set of limb rigid motions xi=(ti,wi) minimizes the distance between the (moved) limb and the observed 3D data • Consists in estimatinglimb motionsxi=(ti,wi)independently: • Estimate visible 3D mesh of each limb • Current implementation uses the ICP algorithm to register each limb w.r.t 3D data WOMOT 2003
Iterative Closest Point 3D registration • find the rigid transformation x that maps shape St (limb model) to shape Sr (3D data) x St Sr WOMOT 2003
Iterative Closest Point Matching points • For all points in St, we search for the closest point in Sr by computing the distance and keep the closest St Sr WOMOT 2003
Iterative Closest Point Energy function minimization • Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points St Sr WOMOT 2003
Iterative Closest Point Energy function minimization • Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points St Sr WOMOT 2003
Iterative Closest Point • Optimal rigid transformation x (and uncertainty Lx) found by combining all the elementary displacements WOMOT 2003
Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003
with D=(x1’, x2’… xN’) satisfying articulated constraints Projection The unconstrained optimal body motion is given by D=(x1, x2 … xN) With uncertainty L=(L1, L2 … LN) Articulated constraints enforcement: find D that minimizes the Mahalanobis distance: WOMOT 2003
Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003
Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003
Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j [.]xdenotes skew-symmetric matrix WOMOT 2003
Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j [.]xdenotes skew-symmetric matrix WOMOT 2003
D=(x1’, x2’… xN’) Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003
D=(x1’, x2’… xN’) Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j (Stack for all joints) WOMOT 2003
Articulated motion estimation All the joint constraints can be written as a linear constraint: • is a linear combination of vectors in the nullspace of F. Therefore there exists a matrix V such that: V can be estimated by SVD of F WOMOT 2003
Articulated motion estimation unconstrained motion articulated motion Find minimum of E2 in nullspace of … (linear projection) WOMOT 2003
Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003
Other constraints • Constraints: • Static: Joint angle bounds, gravity law, … • Dynamic: Maximum velocity, … • Motivation: • Using more constraints to reduce local minima and therefore increase tracking robustness • Application context can reduce tremendously the dimension of the pose space WOMOT 2003
Other constraints Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f(P)>0 ex: f(P)=min(g1(P), g2(P), … gN(P)) with g1(P) = angle(l-arm, l-forearm) – min_angle g2(P) = max_angle - angle(l-arm, l-forearm) …. Constraints enforcement: find D* that minimizes the Mahalanobis distance: with D* satisfying FPt-1(D*)=f(D *(Pt-1))>0 WOMOT 2003
Other constraints articulated constrained motion articulated motion (local parameterization) with WOMOT 2003
Constrained optimization algorithm Alternate between binary and stochastic searches d WOMOT 2003
d Constrained optimization algorithm Alternate between binary and stochastic searches WOMOT 2003
d Constrained optimization algorithm Alternate between binary and stochastic searches E2 = E0 WOMOT 2003
d Constrained optimization algorithm Alternate between binary and stochastic searches E2 = E1 WOMOT 2003
d Constrained optimization algorithm Alternate between binary and stochastic searches d* WOMOT 2003
TRACKING SEQUENCE WOMOT 2003
Future work • Quantitative measurement (comparing results with tethered motion capture system) • Appearance/Shape information (learning color distribution + shape of limbs) • Motion/gesture (including dynamic constraints) • Learning human motion constraints (instead of giving them explicitly.. [ICCV’03]) WOMOT 2003
Applications • Multimodal Human-Computer Interaction (gesture + speech) WOMOT 2003
Conclusion • Projection-based approach for articulated body tracking • articulated constraints enforced by (linearly) projecting unconstrained limb motion on articulated motion manifold • other constraints enforced using a stochastic constrained optimization algorithm WOMOT 2003