Human body model and pose estimation and abnormality classification

1. Human body model and pose estimation and abnormality classification Aravind Sundaresan, James Sherman Jr. and Rama Chellappa Center for Automation Research Dept of Electrical and Computer Engineering University of Maryland at College Park, USA 1 Eadweard Muybridge, Human Locomotion, 1890. 2 Marker-based system (https://buffy.eecs.berkeley.edu/PHP/resabs/resabs.php?f_year=2005&f_submit=chapgrp&f_chapter=14) 3 3D model acquired from video data in estimated pose.

2. What is motion capture? • Motion capture (MoCap) is the process of analyzing and expressing human motion in mathematical terms. • Initialization • Tracking • Pose estimation • Applications • Motion analysis for clinical studies • Human-Computer Interaction • Computer Animation Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

3. What do we want to do? • Using videos from multiple cameras • Obtain human body model • Obtain initial pose • Track 3D pose in all segments. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

4. Motivation • Motion capture techniques. • Optical • Magnetic • Electro-Mechanical • Shortcomings of marker-based systems. • Cumbersome to wear gadgets • Introduce artifacts in the motion • Markerless motion capture. • Easy to set up • Normal cameras can be used Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

5. HBM Acquisition Voxels Pixel motion Pose Initialization Silhouettes Pose Tracking Motion residue Overview • Estimate human body model and pose using voxel data. Articulated super-quadric based Human body model (HBM) Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

6. Prior Work • Voxel-based pose estimation and skeletonisation. • Pose estimation (Mikic03, Chu03), Model estimation(Cheung03) • Skeletonisation (Brostow04) • Dimensionality reduction techniques • Laplacian Eigenmaps (Belkin01), Charting a manifold (Brand02), LLE (Roweis00), Reducing articulated objects to pose-invariant structure (Elad03), Tenenbaum00 • Tracking • Monocular (Yamamoto98, Bregler98) • Motion cues: multiview (Theobalt04) • Shape cues: Edges (Wachter99), Silhouettes (Sidenbladh00, Sigal04) Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

7. Capture environment • Cameras pointing at a central capture volume. • Avoid kinematic singularities • Overcome self-occlusion • Camera calibration: • Automatic LED-based calibration (Svoboda05) • Preprocessing: • Background subtraction Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

8. Human body model • Shape parameters: • Articulated structure • Super-quadrics • Pose parameters: • Base body position and orientation and joint angles • Joint Locations: • X1,X2, X3 describe joint locations of shoulder elbow and wrist Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

9. Voxels and segmentation • Use silhouettes to obtain voxel representation. • Space carving • Objective: segment the voxels into different articulated chains. • Trunk, head, and four limbs. • Transform voxels into a different domain: • Segment voxels and register to different chains. • Estimate position of voxel along chain. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

10. Laplacian Eigenmaps • Construct graph: • Obtain mapping: • Neighborhood relations are preserved • We can extract 1-D structure • Let , and the weight matrix • Laplacian eigenmaps (Belkin-Niyogi01) • Minimize or subject to where • Eigenvector problem: Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

11. Voxels in Laplacian eigenspace (dim. 1-3) (dim. 4-6) [yi]j is the ith element of jth eigenvector. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

12. Fitting 1-D splines to voxels • We fit splines to the voxels in eigenspace. • Vector-valued 1-D spline • t describes distance along articulated chain • Cubic smoothing spline • Algorithm • Start with voxel farthest from any given spline • Keep growing spline by adding nearby voxels • Stop when fit error increases beyond a threshold • Begin new spline Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

13. Fitting splines (illustration) Compute 6 spline segments Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

14. Segmentation results Eigenspace (dim. 4-6) Eigenspace (dim. 1-3) • Segment voxels as belonging to different articulated chains. • Normal 3D Space Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

15. Registration of splines • Segmentation. • Registration. • Trunk, neck-head, left arm, right arm, left leg and right leg. • Represent spline as segment. • Find connectivity graph • match with human body model. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

16. Estimating skeleton • Given segmentation, and spline parameters • Represent each chain by a continuous curve • Compute spline in 3D space using the site parameters estimated in Laplacian eigenspace. Compute approximate pose Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

17. Initial skeletal model • Select key frames (successful registration) • Minimize fit error (stature) in key frames. Minimize error (pose and model parameters). Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

18. Super-quadric parameters • Radial profiles of different body segments. • Trunk, head, arm, forearm, thigh and leg. The solid line is the median radial profile. The dotted line is the super-quadric radius with scale parameter set to zero. The dashed line is the super-quadric radius with estimated scale parameter. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

19. Refine pose (complete model) • Use voxels and super-quadric based model to refine pose estimate. • Compute error for voxels as a function of their position with respect to the super-quadric model. • Find pose that minimizes error of all voxels. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

20. 184 cm, 72kg 165 cm, 64kg 172 cm, 52kg 178 cm, 115kg Experiments on real data • Four subjects (male). • Varying stature, body mass, body mass index. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

21. Volume and pose error Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

22. Current work Resolve ambiguity in difficult configurations. Register segments 1, 2,…, 6 to segments b1, b2, …, b6 (the different body segments). Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

23. Probabilistic registration • Obtain six different segments. • Compute probability of each permutation of registration, based on • Segment properties (length, thickness) • Mutual interconnection Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

24. Probabilistic registration results • Test Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

25. Automatically detecting abnormal behavior James Sherman Prof. Rama Chellappa

26. Introduction • Many people have studied how to identify people according to a certain behavior (mainly gait) • Few have looked into automatically classifying abnormal behavior • Walking with an injured ankle • Parkinson’s patient in a sway experiment Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

27. Overview • Based on work done by Ashok Veeraraghavan • Activity is modeled as an ARMA process in Shape space • Classification is done by computing distances between models • k nearest neighbor rule Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

28. Shape • Statistical Theory of Shape by Kendall • Objects are represented by landmarks • Shape lies on a hypersphere • Approximate sphere locally as a plane Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

29. Modeling • Use SVD to approximate data in a hyperplane • Model the data as an ARMA system • Key parameters are A, C Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

30. Distance between Models • Define distance between models as a function of angles between subspaces • Subspaces here are the ranges of the model’s observability matrix Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

31. Distance between Classes • Comparing subspaces results in a vector of angles between subspaces • Different functions can be used to come up with a distance Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

32. Data • Professor John Jeka and Robert Creath have sway data from healthy patients and patients with Parkinson’s Disease • 6 normal, 6 Parkinson • 5 different experimental conditions • 4 trials for each condition and person • Track 5 markers using an Optotrack system • ankle, knee, hip, shoulder, and platform • Sampled at 100 Hz Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

33. Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

34. Experiment • Randomly choose 3 people as training samples from each class • Left 3 people as test class • Selected one experimental condition (out of 5) Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

35. The Good Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

36. The Bad Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

37. Conclusions/Directions • Find a way to learn the differences between behaviors • Learning the system is the best way to represent a signal (like PCA) but maybe not the best way to differentiate signals (similar to FDA) • Find a single model for a class instead of using a k nearest neighbor classifier Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

38. References • Bissacco, A., et. al., Recognition of Human Gaits, CVPR 2001 • Veeraraghavan, A., et. al., Matching Shape Sequences in Video with Applications in Human Movement Analysis, PAMI Dec. 2005 Human body model and pose estimation and abnormality classification Human Motion - Understanding, Modeling, Capture and Animation Workshop

39. Vielen Dank!