The Functional Space of an Activity Ashok Veeraraghavan , Rama Chellappa, Amit Roy-Chowdhury

1. The Functional Space of an ActivityAshok Veeraraghavan , Rama Chellappa, Amit Roy-Chowdhury Avinash Ravichandran

2. Motivation • Variability exists in activity across subjects and across instances. • The Variability can be regarded as 2 types • GlobalThese include changes in the frame rate and the overall duration of the activity • LocalThese include changes in the actual gait , i.e the leg moves a little faster or slower than a previous instance of walking for the same or different person • Ignoring this variability can lead to identical models having large distances in the modeling framework

3. Need for temporal Warping

4. Time warping for activities t ( ( ( ( ( ( ( ) ) ) ) ) ) ( ) [ [ ) ] ( ] [ ) [ ] ] ( ( ) ) ( ) ( ( ) ) f b f f f b l l b b l b l f h b l f f l d d f f b W P T T T T T i i i j i i i i i i 0 0 0 0 1 0 0 1 · · · · t t t t t t t t t t t t t t t t t t t t t a a w w - - : : s c p a a n - c o e p n r o o c a a a e r p e a e a a y r e m c v o m e o s c r s e y o w e r e a v r a p u e e u u n n c c o n o n e a w = ! ! = b b s a a a T ; ; ; ; ; ; . b Goal : Given multiple instances of a gait, to find a model that fits all of instances, and the time warping for each instance Model

5. Properties of the Function Space 1 ¡ ( ( ) ) [ ] ( ( ) ) [ ] ( ( [ ) ( ) ] ( ) ) ( ) f f f 8 f f f l f f ` l 8 f 8 d f 8 ` f f A A W W i i i 0 1 0 1 0 0 0 1 1 0 1 0 0 1 1 ¸ ¸ ¸ ¸ · t t t t t t t t t t t t 2 2 2 2 2 > 2 2 u u s c o n n o u : e s x a s n s c = ! ! = = ; , , , , ; ; ; ; ; ; ; A is convex Physical constraints reduce the space A to a subset of functions W Properties of W W is convex

6. Canonical Form 1 ¡ 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ( ( ( ( ) ) ( ( ( ( ) ) ) ( ) ) ( ) f ) ( ) f ( ( ) ) ( ( ( ( ) ) ( ) ) ) ) ( ) ( ( ( ( ( ) ( ( ( ( ) ) ( ( ) ) ( ( ) ( ) ) ( ) ) ) ( ) ( ) g g ( ) ( ) ) ) ( ( ) ) ) ) ( ) f f b f f f h f b f h l d b f h f l f f l l l l f f l l l l T L W W i 2 2 2 t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ + ¡ u u e e n s u a a u u e u u s u u a c n u a u u u a u r e e q u s v u a e n s = = = = = = = = = = = ( ) l l 1 1 1 1 i i 1 u ¡ u ; , ; ; 1 1 The model representation is not unique Let This is over come by picking a symmetric u,l , as the symmetric representation is unique Given a non symmetric representation, it can be converted into a symmetric form

7. Overall Scheme

8. Learning Model Parameters N N i i ^ ^ ^ 1 1 1 ¡ ¡ ¡ 1 1 = = f g P P ( ( ( ( ) ( ) ) ( ( ) ) ) ) ( ( ( ) ) ) ( ) ( ) ( ) ( ) ( ) ( ( ) ) b l ^ ^ b h l b b b f f f f l l h 8 8 h f f k l b k d f b h d h f b b f B G A W N i i i i i i i i i 0 0 1 1 t t t t t t t t t t t t t t t t t t t t t t t t t t t 2 2 a u u s v s u e m m n m n a n e p x r a r e a c a c z e a a o n s s a e w w e e m e p a n s o a s w u e m s w e e w n e e e n e w s u c a n a = = = = = = N N i i i i i i i i i i i 1 1 1 1 1 i i i i i s 1 1 N N = = ; , , , , ; ; , = = : : : : : : This is done using the dynamic time warping scheme (details on next slide) Symmetric Model

9. Dynamic Time warping • Used in speech recognition systems to warp instances to a template to learn a model • Works with the different spectra components, creating a vector valued signal from scalar speech signals • Based on dynamic programming, searching over a finite grid

10. Dynamic Time Warping ( ( ) ) ( ( ) ) Á Á k k Á Á k k 1 1 1 ¸ · + + ¡ x x x x • Constraints • Endpoint Constraint • Monotonicity Condition • Local Continuity Constraints • Global Path Constraint • Slope Weighting

11. Features for Activity Recognition C X 1 Z C I 1 ¡ = = k k k k k C X • Silhouette shape is used as a feature • Each silhouette has K landmark points, and the trajectory of these K points form a(t) • Invariant to scale and translation • Preshape lies on the unit sphere • Partial procrustes distance is used as a distance metric

12. Activity Recognition ^ ^ ^ ( ( ( ( ) ) ( ( ( ( ) ) ) ) f d d h h f f I i i i i t t t t t t a a r r g g m m n n s s a a = = f W N i i i i i 1 2 = ; ; s : : : i • Assumption : The test sequence is from one of the training classes • 100 sequences for 10 Activities. Training set was 90 sequences of 10 activities, Testing was done on the remaining 10. Best Warping Function Closest Fit to the model

13. Recognition of individuals

14. Discussion • The concept of using DTW for gait is not new. • Previously DTW was used in a template matching framework where all training data is aligned to a common time frame • This works best when the testing data is among the training datasets, for new sequences they still suffer from the issue of time warping • This algorithm is feature independent, silhouettes can be replaced by intensities, joint location etc • Although the algorithm defines the space of all transformation, this is still dependent on the training dataset

15. Other aspects • ClusteringEM like framework to cluster the gaits, 92 % clustering accuracy with the n classes all representing different gaits • Database organization into a tree like structure to reduce the number of distance calculations