Perceptually Consistent Example-based Human Motion Retrieval

1. Perceptually Consistent Example-based Human Motion Retrieval Zhigang Deng*, Qin Gu, Qing Li University of Houston

2. Introduction • Popularization of human motion capture data in animation and gaming applications • Efficient retrieval of similar motions from a large data repository • Fundamental basis for many motion data based applications e.g. CMU motion capture library. http://mocap.cs.cmu.edu. 2605 trials in 6 categories and 23 subcategories.

3. Related Work – Motion retrieval • Transform original high-dimensional human motion data to a reduced representation [Agrawal et al. 1993; Faloutsos et al. 1994; Chan and Fu 1999; Liu et al. 2003; Chiu et al. 2004; Baciu 2006; Lin 2006] . • Match webs[Kovar and Gleicher 2004] • Describe potential subsequence matches between any pair of motion sequences. • Semantics-based motion retrieval[Muller et al. 2005; Muller and Roder 2006] • Users provide a query motion as a set of time-varying geometric feature relationships.

4. Our Approach Pipeline Motion Data Preprocessing • Human hierarchy construction • Motion segmentation and normalization • Motion pattern detection and indexing • Hierarchical pattern matching • Search result ranking Runtime Motion Query

5. Data Preprocessing - Motion Hierarchy Construction • Decompose human motion into a hierarchical structure [Gu et al. 08] • Local control granularity • Correlations among different human parts are embedded in different layers • 4 layers, 18 parts are used in this work.

6. Data Preprocessing - Motion Segmentation and Normalization • Existing human motion segmentation techniques • Angular acceleration[Zhao 01, Fod et al. 02, Kim et al.03], SVM classifier[Li et al. 07], weighted sum of marker velocities[Gu et al. 08], PCA/PPCA[Barbic et al.04]. • Probabilistic PCA[Barbic et al. 04]is used to segment motion into short motion segments for each body part in the hierarchy. Average Frame Information of segments

7. Data Preprocessing - Clustering • Motion Pattern for each body part • A representative motion segment for a node (i.e.,a body part) in the constructed human hierarchy • Normalization of motion segments • Adaptive K-Means clustering • Increase K when the clustering error metric is larger than a threshold • Resulting data structures • (1) Motion Pattern Library, (2) Pattern Index Lists, (3) Pattern Dissimilarity Maps.

8. Review of Motion Preprocessing

9. Runtime Motion Query • Query motion transformation • Map the query motion into a motion pattern index list for each hierarchy node • Fast (no clustering, just database matching) • Motion similarity score computing • Local motion similarity between two index lists • Extended Knuth-Morris-Pratt (KMP) string matching algorithm [Knuth et al. 77] • Global motion similarity computing and ranking • Hierarchical propagation

10. Local Motion Similarity • Similarity between two pattern index lists • Different length of index lists • Matching of two integer lists • Extended KMP String match algorithm • Introducing “Quasi-Match” based on the pre-constructed pattern dissimilarity maps • Large numbers of different motion segments • Distance is less than a threshold • Update matching score • If the number of consecutive quasi-matches is larger than a threshold, otherwise decrease.. • Score normalization based on the length of index lists

11. Global Motion Similarity • Hierarchical Score Propagation • High local motion similarity does not mean global motion similarity • Nodes in the upper levels encode more global motion information • From bottom to top • Ranking of the final scores at the root node

12. Review of Runtime Motion Retrieval

13. Results and Evaluation • Time and Storage • Search Accuracy • Search Quality • Perceptual Consistency Experiment

14. Results and Evaluations – Time and Storage • We tested our method on four datasets with different sizes • The test computer with a Intel Duo Core 2GHz CPU and 2GB memory. • The average duration of used query motions is 10 seconds. 56MB, 170 motions,68,293 frames 456MB, 396 motions, 556,097 frames 976MB, 542 motions, 1,190,243 frames 1452MB, 941 motions, 1,770,731 frames

15. Results and Evaluations – Search Accuracy • Accuracy evaluation scheme [Kovar and Gleicher 04] • Two different types of datasets: single-type motion datasets (pre-labeled dataset with the same semantic category, walking) – Ground truth, mixed motion dataset (unlabeled, mixed of various types). • True-positive accuracy ratio is defined top N (=20) results from mixed motion datasets are in the correct/expected single-type motion dataset. • 56M test dataset: 170 sequences, 68,293 frames, five categories – walking, running, jumping, kicking, basket-playing.

16. Results and Evaluation – Comparative User Studies • Compare our approach with match-webs approach [Kovar and Gleicher 04], piecewise linear space [Liu et al.05], weighted PCA [Forbes and Fiume 05]. • Semantic-based motion retrieval [Muller et al. 05] was not chosen, because of significant differences in input requirements. • Two usability questions • (a) Perceptual Consistency: Retrieved results (motions) are ranked in a perceptually consistent order? • (b) Search Quality: Motion similarities of retrieved results?

17. Results and Evaluation – Comparative User Studies • Perceptual-consistency • Computer algorithms rank motions in a certain order, C. • Humans rank these (the same) motions in another order, H. • Relationship/consistency between C and H? • Study Experiments • 3 query motions (walking, running, basketball-playing),Top-ranked N (=6) results for query, 4 approaches, total 72 = 3*6*4 results. • Side-by-side comparison and user rating (one is a searched motion, the other is the query motion), in a random order. • Rating is from 1 (“completely different”) to 10 (“identical”). • 24 experiment participants

18. Results and Evaluation – Comparative User Studies • Quality of searched motions • Compute average similar ratings and standard deviation • Higher the average similar rating is, the better quality of search it achieves.

19. Results and Evaluation – Comparative User Studies • Perceptual-Consistency • Plot human-rankings vs computer-rankings in a 2D space. • Ideal consistency is shown as a straight line. • Canonical Correlation Analysis • Scale-invariant optimum linear framework Walking Running CCA Coefficient results Basketball-playing

20. Review of User Studies

21. Conclusions • An efficient, example-based human motion retrieval technique • Major distinctions of our approach • Efficiency • Linear to the size of query motion and database size • Flexible search query • A human motion subsequence, or a hybrid of multiple motion sequences • Perceptually consistent search outcomes • Comparative user studies to find out the correlations between the result-ranking by computer algorithms and the result-ranking by humans

22. Discussion and Limitations • Current approach does not consider the path/motion trajectory of the root of the human in the retrieval algorithm. The search results may enclose different paths/trajectories. • Current approach can only search for single-character motion sequences.

23. Future Work • A number of empirical parameters of current approach may critically affect the search accuracy and outcomes. • Establish quantitative correlations between “parameter setting” and “search accuracy and outcomes”. • Graphics hardware accelerated, motion query processing.

24. Thank You! Questions?