Volume Based Human Motion Capture using Example Topology Graphs

1. Volume Based Human Motion Captureusing Example Topology Graphs Atsushi Nakazawa, Hidenori Tanaka Haruo Takemura Cybermedia Center, Osaka University {nakazawa,tanaka,takemura}@lab.ime.cmc.osaka-u.ac.jp

2. Motion Capture Systems • Marker-based Motion Capture • Use for Computer Animation, etc. • Less Usability • Vision based Markerless Mo-Cap using Multiple Cameras • Tracking Image Features • Skin colors, Edges,.. • Use of human body templates • Analyze Reconstructed Volumes • Match to articulated models • Volume analysis technique

3. Related Work • Tracking Human Body Features[Taniguchi2004] • Tracking skin color regions • IK & FK, Projection • Boundary matching • Few DOF are estimated. • Use of Articulated Model [Mikic, IJCV2003] • Compare the articulated model and volume data • Cylinders and ellipsoids • Needs Good Initial Guess • Tracking failure • High computational cost

4. Related Work • Bottom-up Analysis of Volume Data [Chu,CVPR2003] • Map volume data onto ISOMAP space • Limbs are expanded  Easy for segmentation ○ Bottom-up approach  No tracking failure, No initial guess × High computational cost for ISOMAP mapping × Cannot handle the variation of topology in human body Result Skelton Input Volume Skelton Segmentate Isomap Space

5. Concept of Our Method • Bottom-up Approach • Direct analysis of volume data • No initial guess, No tracking failure • Fast Processing • Direct processing of volume data (No conversion to the another space) • Topology Issue • Example based approach • Introduce the topology examples of human body

6. Algorithm Overview Volume Reconstruction and Processing Input Image Background subtraction Background Image Distortion Param. Visual Hull Camera Param. Volume Thinness Joint Estimation Recognition Graph Conversion Joint Estimation Graph Matching Model Graph Database (MGDB)

7. Capturing Volume Data • Silhouette Extraction • Background Subtraction • Binalize • Noise Removal • Volume Extraction • Silhouette based Visual-Hull

8. Extraction of Body Skelton • 3D Sequential Thinness Process [Saitoh, SCIA95] • Volume  Line segments • Preserve original topology • Skelton Features • Topology • Line segments features • Length • Volume around the line segments 細線化の処理過程

9. Input Skelton Skelton Analysis • Extract Joint Positions from Skelton • High curvature points, Line fitting • Need to identify the body portions for each line segments • Topologies of Human Body • A lot of variations • Idea: Example Based Search • Prepare the topology database of human body • Search the most similar example for the current skelton Variations of Human Body Skelton

10. Graph Representation of Skelton • Graph is the good expression of topological data. • Line Segments of Skelton  Graph Nodes • Node Attributes: Length and Surrounding volume • Connections of Line Segments  Graph Edges • No Geometrical Attributes • Invariant to the direction/position of the body portions = One example graph represents a lot of postures Vol. 9 Len. 8 9,8 Vol. 10 10,17 Len. 17 10,16 Vol. 10 Len. 16 20,9 Vol. 20 Len. 9 Vol. 24 Len. 22 24,22 25,25 Vol. 25 Len. 25 Skelton Attributes Graph Expression of Skelton Volume around the line segments

11. Recognition of Body Portions • Model Graph DB (MGDB) • Contains topology examples • Body portions are annotated manually for each graph nodes. • Graph Matching : Recognize the body portions of input graph. Graph Matching Input Skelton Input Graph Recognition Head Head Hand Hand Hand Hand Body Body Foot Foot Foot Foot Model Graph Database (MGDB) Recognized Skelton

12. Graph Matching Algorithm • Graph Edit-distance [Messmer, PAMI98] • [G1]  (Edit Operation)  [G2] • Graph similarity = Minimum edit operation cost • Edit operation: • {Delete, Substitution} of {Nodes, Edges} • Need to define edit operation costs • cost( del_node(n) ) = 0 • cost( sub_node(n1,n2) ) = k1 |n1.vol – n2.vol| + k2|n1.len – n2.len|

13. Joint Position Estimation • Line Fitting Approach • Recognition result of line segments and human body portions  The number of joints N are obtained. • Fit the line segment by N-straight lines

14. Experiments • Preparation of MGDB • Estimate the optimal matching cost k1, k2 • Motion Capture Test • 3 Subjects • Subject 1 : Male， 166cm • Subject 2 : Male， 180cm • Subject 3 : Female，155cm • Capturing Studio • 8 camera systems • 1024×728 pixel / 30 fps, 24bit color images

15. Preparation of MGDB • Skelton Graphs of Subject 1 • 13 Topologies • 23 Example Graphs • Body portions are labeled manually for each nodes.

16. Optimal Cost Function • Edit Cost of “Substitution” operation • Needs to set k1/k2 ratio • cost( sub_node(n1,n2) ) = k1 |n1.vol – n2.vol| + k2|n1.len – n2.len| • Search Optimal k1/k2 through Test Sequence • Graph matching test with some k1/k2 pairs. • Test Sequence • Subject 1: 312 frames out of 3120 frames • Manually check whether the recognition is succeeded.

17. Success Rate and k2/k1 Rate • 81.4% (k2 / k1 = 0.8)

18. Motion Capture Result (Sub.1)

19. Motion Capture Result: Sub.2,3

20. Performance • 6.5 sec/frame • The ISOMAP based one takes 60-90 sec/frame. • Volume Reconstruction: 5.75 sec • Thinness: 0.62 sec • Graph Conversion: 0.097 sec • Graph Matching: 0.059 sec • Joint Point Estimation: 0.006 sec

21. Conclusion • Novel Markerless Human Motion Capture using Volume Data • Direct Analysis of Volume Data • Fast processing, No tracking failure • Consider the variations of human-body topology • Example based approach based on a graph matching method. • Future work • Introduce time-sequence analysis for joint estimation • Introduce new attributes for graph matching

22. Thank you very much. Questions or Comments ?