3D v ideo understanding using a topology dictionary

1. 3D video understanding using a topology dictionary Tony Tung & Takashi Matsuyama Kyoto University, Japan Dagstuhlseminar Oct.14th, 2010

2. 3D video - Markerless motion/surface capture - Image-based system [Matsuyama et al., CVIU'04]

3. 3D video framework - Reconstruction space: 3m x 3m x3m - 16 video cameras UXGA 30 fps - Synchronization by external trigger - Geometrically calibrated

4. 3D video framework 3D surface reconstruction by MVS technique One or several subjects per frame Volumetric graph-cuts (5cm resolution) 1 frame ~ 1.5 MB (30,000 triangles) 5 min ~ 11.25 GB No 3D video tapestry! [Tung et al., CVPR’08] [Tung et al., ICCV’09]

5. 3D video modeling using a topology dictionary • Encoding of 3D video sequences • Description of content for human behavior understanding

6. 3D video modeling using a topology dictionary • Structure of the model 1 - Pattern detection using a topology descriptor (Reeb graph) 2 - Encoding of topology clusters 3 - Probabilistic motion graph • Video skimming • Pose/Action recognition • 3D performance segmentation [Tung et al., CVPR’09] [Tung et al., ICIP’10]

7. 3D video sequence Independent frame reconstruction

8. 3D video sequence Inconsistent topology between frames

9. I – Enhanced Reeb graphs

10. Topology description • Morse theory  : Swith : real continuous function S : manifold surface (mesh surface) Reeb graph = quotient space of the graph of  in S defined by the equivalence relation ~ . (X) = (Y) (X ,Y) S2, X ~ Y . X and Y same connected component as -1((X)) [Reeb, 1946]

11. Shape description • Multiresolution Reeb graphs - Automatic extraction of graphs • R, t, scale invariant • Homotopic • Multiresolution coarse-to-fine matching [Hilaga et al., SIGGRAPH’01] [Tung et al.,CVPR’07]

12. Topology description

13. Topology matching - Invariance to rotation, translation and scale - Coarse-to-fine multiresolution strategy - Matching using topological and geometrical attributes (valency, relative area) - The similarity of two models M,N is obtained by evaluation of the “similarity” of topology consistent pairs {(mi, nj)} at every level of resolution SIM(M,N) =  sim(mi, nj) R r=0 {ij} [Hilaga et al., SIGGRAPH’01] [Tung et al.,CVPR’07]

14. Performance evaluation Pose retrieval in 3D video sequences [Huang et al., 3DPVT'10]

15. II - Topology dictionary

16. Topology clusters Encoding of (repetitive) poses

17. Topology clusters • Encoding of (repetitive) poses

18. Topology clusters • Encoding of (repetitive) poses i i

19. Topology clusters • Motion graph structure SIGGRAPH’02: [Arikan&Forsyth] [Kovar et al.] [Lee et al.] i

20. Topology clusters • Motion graph structure SIGGRAPH’02: [Arikan&Forsyth] [Kovar et al.] [Lee et al.] i SUMMARIZATION

21. Topology clusters • Motion graph structure SIGGRAPH’02: [Arikan&Forsyth] [Kovar et al.] [Lee et al.] i 3D VIDEO SKIMMING SKIMIN

22. Topology clusters Dataset clustering using SSM • Similarity function SIM

23. Topology clusters • Dataset clustering using SSM Repetitive poses Long poses Short poses Transitions • Similarity function SIM

24. Topology clusters flashkick head free

25. Topology clusters lock pop kickup

26. Directed motion graph • Motion graphs allow users to design new sequences by building walks on the graph

27. Directed motion graph • 3D video sequences contain noises and redundancies

28. Probabilistic motion graph • Graph G = (C=U{Ci},E) • Node weight depends on topology cluster size if P(Ci) >> 0, then Ci corresponds to a long pose of a repetitive pose

29. Probabilistic motion graph • Graph G = (C=U{Ci},E) • Node weight depends on topology cluster size if P(Ci) >> 0, then Ci corresponds to a long pose of a repetitive pose • Transition determines how relevant is a motion if P(Ci|Cj) <1, then Cj corresponds to cycle junction node

30. Probabilistic motion graph …P(C’i) P(C’k)… P(C’j) Selection of the most probable paths p = {C'i, e'ij} argmax P(p) = ∏ P(C'i | C‘j) P(C'i) {e'ij}

31. Probabilistic motion graph • 3D video skimming by cycle trimming small cycles (noise, short action) Cycles are identified by cluster index in the sequence

32. Probabilistic motion graph • 3D video skimming by cycle trimming small cycles (noise, short action) Cycles are evaluated by Size and Relevance: Small S(L) and P(L) are first candidates for skimming

33. 3D video skimming 1 - Evaluate cycle size and relevancy • . Small cycles with low probability are first candidates 2 - Compute path probability regarding cycles • . The most probable paths indicate which cycles to remove

34. 3D video skimming

35. 3D video skimming

36. Training dataset with annotation

37. 3D video description

38. Synthetic dataset

39. 3D video description

40. Segmented training dataset

41. Performance segmentation

42. Performance segmentation

43. Performance segmentation

44. Summary 3D video is a markerless motion capture technique which allows to a capture subject as is Topology dictionary model to represent 3D videos Sequence encoding Topology matching using Reeb graphs Probabilistic motion graph structure 3D video skimming/summarization 3D video description and segmentation

45. Future work • How accurate is the matching? • What can we not recognize? • Action -> behavior? • Sequence reconstruction artifacts • Smoothness parameters

46. 3D Shape Reconstructionfrom Multi-viewpoint images and silhouettes Thank you for your attention.