Pre Proposal Time Series Learning completed work

1. Pre ProposalTime Series Learningcompleted work Lei Li Computer Science Department Carnegie Mellon University

2. Outline • Completed Work • Mining w/ Missing Value • Parallel Learning • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

3. Outline • Completed Work • Mining w/ Missing Value • Motivation • Problem Definition • Proposed Method • Results • Parallel Learning • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

4. Occlusion in Motion Capture • Motion Capture: • Markers on human actors • Cameras used to track the 3D positions • Duration: 100-500 • 93 dimensional body-local coordinates after preprocessing (31-bones) • Challenge: • Occlusions • Other general scenario: • Missing value in Sensor data: Out of battery, transmission error, etc • Unable to observe, e.g. historical/future observation From mocap.cs.cmu.edu

5. Problem Definition • Given • To find algorithm for: • mining hidden variables and evolving patterns • recovering missing values • compression/summarization • segmentation Marker/Sensor Time blackout

6. Problem Definition (cont’) • Want the algorithms to be: • Effective • Scalable: to duration of sequences • Blackouts • Automatic: no/few parameters to be set Marker/Sensor Time blackout

7. Proposed Method: Intuition Recover using Correlation among multiple markers Left Hand Right Hand missing

8. Proposed Method: Intuition Recover using Dynamics temporal moving pattern Left Hand Right Hand missing

9. Underlying Model Use Linear Dynamical Systems to model whole sequence. N(z0, Γ) N(F∙z2, Λ) N(F∙z3, Λ) N(F∙z4, Λ) N(F∙z1, Λ) Z1 Z2 Z3 Z4 … N(G∙z1, Σ) N(G∙z2, Σ) N(G∙z3, Σ) N(G∙z4, Σ) X4 X1 X2 X3 z1 = z0+ω0 zn+1 = F∙zn+ωn xn = G∙zn+εn Model parameters: θ={z0, Γ, F, Λ, G, Σ}

10. DynaMMo Intuition • How to recover the missing values?

11. DynaMMo: How to Recover? ×

12. DynaMMo: How to Recover? × × ×

13. DynaMMo: How to Recover? × × × ×

14. DynaMMo: How to Recover? × × × × ×

15. DynaMMo: How to Recover? × × × × × ×

16. How to Compress • Naive idea #1: use SVD • Naive idea #2: store parameters of LDS • Naive idea #3: store parameters of LDS and all hidden variables (expectation) • Proposed Methods: use check points • Fixed hop • Optimal (dynamic programming) • Near optimal (adaptive)

17. DynaMMo Compression: Intuition observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them and Same idea could be used in segmentation and forecasting

18. DynaMMo w/ Optimal Compression: Intuition observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them and Same idea could be used in segmentation and forecasting

19. How to Segment • Segment by threshold on prediction error original data reconstruction error

20. Outline • Completed Work • Mining Missing Value • Motivation • Problem Definition • Proposed Method • Results • Parallel Learning • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

21. Results – Better Missing Recovery Reconstruction error MSVD MSVD Proposed Ideal occlusion length

22. Results – Better Compression error DynaMMo w/ optimal compression Ideal Compression ratio

23. Results – Segmentation • Find the transition during “running” to “stop”. left hip left femur reconstruction error

24. Outline • Completed Work • Mining Missing Value • Contribution: the most accurate mining algorithms for TS with missing value so far. • Parallel Learning • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

25. Outline • Completed Work • Mining Missing Value • Parallel Learning • Motivation • Problem Definition • Proposed Method • Results • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

26. step Challenge for Learning LDS on SMP 1 Position of left elbow Measured * Estimated Time

27. step Challenge for Learning LDS on SMP 2 Position of left elbow Measured * Estimated * Time Intuition: #2 may be close to #1

28. Forward Challenge for Learning LDS on SMP Position of left elbow * * * Measured * * Estimated * Time

29. Backward Challenge for Learning LDS on SMP Position of left elbow Estimated * * * Measured * * * * Time

30. Backward Challenge for Learning LDS on SMP Position of left elbow * * * * * Estimated * Measured * * * * Time *

31. Backward Challenge for Learning LDS on SMP Position of left elbow * * Reconstructed Signal * * * * Measured * * * * Time *

32. Outline • Completed Work • Mining Missing Value • Parallel Learning • Motivation • Problem Definition • Proposed Method • Results • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

33. Problem Definition • Problem: • Given a sequence of numbers, find the best model parameters for Linear Dynamical System • Goal: • Achieve ~ linear speed up on multi-core • Assumption: • shared memory architecture

34. Cut-And-Stitch Intuition z6 z1 z3 z4 z5 z2 y1 y2 y3 y4 y5 y6 υ1,Φ1,η1,Ψ1 υ2,Φ2,η2,Ψ2 Stitch Cut start computation without feedback from previous node reconcile later υ3,Φ3,η3,Ψ3 z1 z'2 z2 z3 z4 z'4 z6 z5 y1 y2 y3 y4 y5 y6

35. Cut-Forward Cut-And-Stitch: illustration 1 Position of left elbow * * Measured * Estimated Time

36. Cut-Forward Cut-And-Stitch: illustration 2 Position of left elbow * * * Measured * * Estimated * Time

37. Cut-Backward Cut-And-Stitch: illustration Position of left elbow * * * * * Measured * * * * Time

38. Stitch Cut-And-Stitch: illustration Position of left elbow * * * * * * Measured * * * * Time * reconciliation reconciliation

39. Outline • Completed Work • Mining Missing Value • Parallel Learning • Motivation • Problem Definition • Proposed Method • Results • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

40. Near Linear Speedup speedup ideal Dataset: CMU Mocap #16 mocap.cs.cmu.edu # of processors

41. No loss of accuracy ~ IDENTICAL

42. Outline • Completed Work • Mining Missing Value • Parallel Learning • Contribution: the 1st parallel algorithm for learning LDS • Natural Motion Stitching • Ongoing & Proposed Work • Other Related Work

43. Outline • Completed Work • Mining Missing Value • Parallel Learning • Natural Motion Stitching • Motivation • Problem Definition • Proposed Method • Results • Ongoing & Proposed Work • Other Related Work

44. How to generate new natural motion? • Computer Game industry • E.g. generate a smooth “goal kick” in soccer game • Movie Industry • E.g. Shrek

45. A Database Approach • Select best stitchable segments from a set of basic motion pieces and generate new natural motions

46. Problem Definition • Given two motion-capture sequences that are to be stitched together, how can we assess the goodness of the stitching? • Euclidean will fail 2 1 Best stitchable motion? 3

47. Outline • Completed Work • Mining Missing Value • Parallel Learning • Natural Motion Stitching • Motivation • Problem Definition • Proposed Method • Results • Ongoing & Proposed Work • Other Related Work

48. Minimizing Stitching Effort • Minimize the energy/effort spent by human during the transition • Compute the effort using dynamics from Kalman Filters

49. Result

50. Outline • Completed Work • Mining Missing Value • Parallel Learning • Natural Motion Stitching • Contribution: A principled distance function for motion stitching • Ongoing & Proposed Work • Other Related Work