1 / 38

Enforcing Constraints for Human Body Tracking

Enforcing Constraints for Human Body Tracking. David Demirdjian Artificial Intelligence Laboratory, MIT. Goal. Real-time articulated body tracking from stereo accounting for constraints on pose. Approach.

shepherdc
Download Presentation

Enforcing Constraints for Human Body Tracking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Enforcing Constraints for Human Body Tracking David Demirdjian Artificial Intelligence Laboratory, MIT

  2. Goal • Real-time articulated body tracking from stereo accounting for constraints on pose WOMOT 2003

  3. Approach • Differential tracking: assuming the articulated body pose Pt-1 is known, estimate the pose Pt (or equivalently the set of limb rigid motions xi=(ti,wi)between poses Pt-1 and Pt) that minimizes the distance between the articulated model and the observed 3D data  tracking as a constrained optimization problem WOMOT 2003

  4. Approach • Differential tracking: assuming the articulated body pose Pt-1 is known, estimate the pose Pt (or equivalently the set of limb rigid motions xi=(ti,wi)between poses Pt-1 and Pt) that minimizes the distance between the articulated model and the observed 3D data  tracking as a constrained optimization problem • Solve unconstrained optimization problem • Project solution on constraint surface WOMOT 2003

  5. Projection-based approach D (unconstrained optimum) D* human motion manifold WOMOT 2003

  6. Approach • Estimate limb motionsxi=(ti,wi) independently using standard multi-object tracking algorithm • Projection: findthe closest body motion D*=(xi’) to D=(xi) that satisfies human body constraints: • articulated constraints • other constraints: joint limit, … WOMOT 2003

  7. Previous work • Particle sampling: Sidenbladh & al. ECCV’00 Sminchisescu & Triggs CVPR’01 • Differential tracking: Plankers & Fua ICCV’99 Jojic & al. ICCV’99 Delamarre & Faugeras ICCV’99 WOMOT 2003

  8. Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003

  9. Multi-object tracking • Assuming the articulated body pose Pt-1 is known, estimate the set of limb rigid motions xi=(ti,wi) minimizes the distance between the (moved) limb and the observed 3D data • Consists in estimatinglimb motionsxi=(ti,wi)independently: • Estimate visible 3D mesh of each limb • Current implementation uses the ICP algorithm to register each limb w.r.t 3D data WOMOT 2003

  10. Iterative Closest Point 3D registration • find the rigid transformation x that maps shape St (limb model) to shape Sr (3D data) x St Sr WOMOT 2003

  11. Iterative Closest Point Matching points • For all points in St, we search for the closest point in Sr by computing the distance and keep the closest St Sr WOMOT 2003

  12. Iterative Closest Point Energy function minimization • Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points St Sr WOMOT 2003

  13. Iterative Closest Point Energy function minimization • Estimate the rigid transformation that minimizes the sum of squared distances between corresponding matched points St Sr WOMOT 2003

  14. Iterative Closest Point • Optimal rigid transformation x (and uncertainty Lx) found by combining all the elementary displacements WOMOT 2003

  15. Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003

  16. with D=(x1’, x2’… xN’) satisfying articulated constraints Projection The unconstrained optimal body motion is given by D=(x1, x2 … xN) With uncertainty L=(L1, L2 … LN) Articulated constraints enforcement: find D that minimizes the Mahalanobis distance: WOMOT 2003

  17. Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003

  18. Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003

  19. Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j [.]xdenotes skew-symmetric matrix WOMOT 2003

  20. Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j [.]xdenotes skew-symmetric matrix WOMOT 2003

  21. D=(x1’, x2’… xN’) Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j WOMOT 2003

  22. D=(x1’, x2’… xN’) Articulated motion estimation If Mij is a joint between objects i and j: Motion of point Mij on limb j Motion of point Mij on limb i = (Ri,ti) Mij joint obj. i (Rj,tj) obj. j (Stack for all joints) WOMOT 2003

  23. Articulated motion estimation All the joint constraints can be written as a linear constraint: • is a linear combination of vectors in the nullspace of F. Therefore there exists a matrix V such that: V can be estimated by SVD of F WOMOT 2003

  24. Articulated motion estimation unconstrained motion articulated motion Find minimum of E2 in nullspace of … (linear projection) WOMOT 2003

  25. Plan • Unconstrained problem • Articulated constraints enforcing • Other constraints • Tracking results • Application (Multimodal interface) • Conclusion WOMOT 2003

  26. Other constraints • Constraints: • Static: Joint angle bounds, gravity law, … • Dynamic: Maximum velocity, … • Motivation: • Using more constraints to reduce local minima and therefore increase tracking robustness • Application context can reduce tremendously the dimension of the pose space WOMOT 2003

  27. Other constraints Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f(P)>0 ex: f(P)=min(g1(P), g2(P), … gN(P)) with g1(P) = angle(l-arm, l-forearm) – min_angle g2(P) = max_angle - angle(l-arm, l-forearm) …. Constraints enforcement: find D* that minimizes the Mahalanobis distance: with D* satisfying FPt-1(D*)=f(D *(Pt-1))>0 WOMOT 2003

  28. Other constraints articulated constrained motion articulated motion (local parameterization) with WOMOT 2003

  29. Constrained optimization algorithm Alternate between binary and stochastic searches d WOMOT 2003

  30. d Constrained optimization algorithm Alternate between binary and stochastic searches WOMOT 2003

  31. d Constrained optimization algorithm Alternate between binary and stochastic searches E2 = E0 WOMOT 2003

  32. d Constrained optimization algorithm Alternate between binary and stochastic searches E2 = E1 WOMOT 2003

  33. d Constrained optimization algorithm Alternate between binary and stochastic searches d* WOMOT 2003

  34. TRACKING SEQUENCE WOMOT 2003

  35. Future work • Quantitative measurement (comparing results with tethered motion capture system) • Appearance/Shape information (learning color distribution + shape of limbs) • Motion/gesture (including dynamic constraints) • Learning human motion constraints (instead of giving them explicitly.. [ICCV’03]) WOMOT 2003

  36. Applications • Multimodal Human-Computer Interaction (gesture + speech) WOMOT 2003

  37. WOMOT 2003

  38. Conclusion • Projection-based approach for articulated body tracking • articulated constraints enforced by (linearly) projecting unconstrained limb motion on articulated motion manifold • other constraints enforced using a stochastic constrained optimization algorithm WOMOT 2003

More Related