Recognition of Human Gaits

1. Recognition of Human Gaits Alessandro Chiuso Dipartimento di Elettronica e Informatica Universita di Padova Stefano Soatto Computer Science UCLA Alessandro Bissacco Computer Science UCLA Yi Ma Electrical and Computer Engineering University of Illinois at Urbana-Champaign Presented by Peter Schwer

2. Summary • Problem Description • Related Work • Overview of Gait Recognition Process • Tracking • Learning the Model • Recognition • Results

3. The Human Gait • Observable from great distances • Discernable independent of skew (relative direction of subject’s gait) • Communicates: • Identity • Mood

4. Good Recognition Traits • Invariant to: • Photometric Factors: • Illumination, clothing • Geometric Factors: • Camera Position • Length of subject’s limbs • Relative direction of gait

5. Good Recognition Traits • Capable of discerning: • Different types of gaits: • Running • Walking • Skipping, etc. • Mood • Not addressed, but dynamical models can be extended to this area.

6. Analyzing gait with spatiotemporal surfaces (1994) Sourabh A. Niyogi, Edward H. Adelson Related Work University of Southampton Carnegie Mellon University (CMU) Massachusetts Insitute of Technology (MIT) Georgia Tech (GATech)University of MarylandUniversity of South Florida

7. The Recognition Problem • The Sub-problems: • Tracking • Learning the model • Recognizing

8. Tracking • “The emphasis of our work is not tracking.” • Images → Skeletons • Results: y(t), t = 1 … τ (Sequence of joint positions)

9. Learning the Model

10. The Right Model • Ideally, • Unique model for each dataset • Method for calculating similarity (distance) between models • Catch: • “The set of dynamical models in canonical form is not a linear space (even if the model itself is linear!)” • Distances between dynamical models are not trivially computed. • Data, as usual, is inherently noisy.

11. Creating the Dynamical Model • Input: y(t) – Sequence of Joint Positions • What are some of the statistical properties of this sequence of joint positions? • Stochastic Stationary Second-order process • How is noise distributed in the data? • We use a Gauss-Markov Model as a representative of each class.

12. The ARMA Model x(t+1) = Ax(t) + v(t), v(t) ~ N(0,Q); x(0) = x0 y(t) = C(x(t) + w(t); w(t) ~ N(0,R)

13. The ARMA Model • Other Applications: • Speech Recognition • Tracking • Geophysics: Forecasting and Simulation • Pattern Recognition • Dynamic Texture Recognition

14. The ARMA model x(t) is a process describing the “state” at time t (vector) y(t) is the measured joint positions (vector) Q,R are input and output noise covariances (matrices) x(t+1) = Ax(t) + v(t), v(t) ~ N(0,Q); x(0) = x0 y(t) = C(x(t) + w(t); w(t) ~ N(0,R) (A:[n × n], B:[m × n]) PROBLEM: Non-uniqueness of A, C, Q, R, S

15. Innovation/Canonical Representation • Identify stochastic subspace models without forming the covariance matrix. • Principal Angles and Directions • Source: P. Van Overschee and B. De Moor Subspace Algorithms for the stochastic identification problem. Automatica, 29:649-660, 1993.

16. A Canonical Representation x(t) is a process describing the “state” (vector) y(t) is the measured joint positions (vector) Q,R are input and output noise covariances (matrices) x(t+1) = Ax(t) + v(t), v(t) ~ N(0,Q); x(0) = x0 y(t) = C(x(t) + w(t); w(t) ~ N(0,R) From Overschee and De Moor we have {A,C,Q,R} A and C describe the gait.

17. Recognizing

18. Our Dynamic Model • A, C, Q, R provided • What are they? • How might they be useful?

19. How to Describe Gait • Tracking produces a “waveform” that describes a series of joint positions/angles: What is the best way to compare these to waveforms? Gait 1 Gait 2

20. Distance Between Models • Subspace/principal angles • Start with: O(M)- infinite observability matrix

21. Form O(m) • O(m) is a subspace • We want to compare the observability subspaces of two modelsM1 and M2. • Form O(m) for each model: • Call the results H1 and H2, respectively.

22. Form Orthonormal Matrices • Gram-Schmidt Orthogonalization on H1 and H2 • Described: http://lagrange.la.asu.edu/VirtualClass/Algebra/GramSchm.html • Applet: http://www.mste.uiuc.edu/exner/ncsa/orthogonal/#simulation • Call the result orthonormal matrices QH1 and QH2

23. Calculate Principal Angles • Compute n-ordered Singular Values of QTH1QH2 • n-ordered singular values can be denoted: cos2(θ1), … , cos2(θn) • Principal Angles between the subspaces H1 and H2 are denoted by the n-tuple: • (θ1 , θ2 ,…, θn ) sorted in descending order.

24. Calculate the Distance • First measure is not necessarily non-negative (less fit) • Second Measure is the largest principal angle (more fit) • The Martin and Finsler Distance, respectively.

25. Or • Just use their matlab code: function theta = subspace_angles(A1,K1,C1,A2,K2,C2) n = size(A1,1); m = size(C1,1); A = [ A1 zeros(n,3*n); zeros(n) A2-K2*C2 zeros(n,2*n); zeros(n,2*n) A2 zeros(n); zeros(n,3*n) A1-K1*C1]; C = [ C1 -C2 C2 -C1 ]; Q = dlyap(A’,C’*C); E = eig([zeros(2*n) pinv(Q(1:2*n,1:2*n))*Q(1:2*n,2*n+1:4*n); pinv(Q(2*n+1:4*n,2*n+1:4*n))*Q(2*n+1:4*n,1:2*n) zeros(2*n)]); E = max(-ones(size(E)),E); E = min(ones(size(E)),E); theta = acos(E(1:2*n));

26. Results

27. Example Gaits

28. Name that Gait Pensive and determined.

29. Obligatory Gait Video 2 Not in a rush.

30. Obligatory Gait Video 3 Sad walking.

31. Obligatory Gait Video 4 There is no stopping me.

32. Results “The pairwise distance between each sequence in the dataset is displayed in this plot. Each row/column of a matrix represents a sequence, and sequences corresponding to similar gaits are grouped in block rows/columns. Dark indicates a small distance, light a large distance. The minimum distance is of course along the diagonal, and for each column the next closest sequence is indicated by a circle, while the second nearest is indicated by a cross.”

33. Results • How accurate are the results? • How does this compare with human ability?

34. Questions?

35. From Discussion Board • Diem: How are the ARMA model and Kalman filters related? • Mei-fang: Is there any method to find the best size training set, and the best k for classifying the new sample? • Matt Clothier: What is the advantage of the Martin distance over the Finsler distance? • Neil and Mike: What do they mean by “Transient Actions” and why does the previous assumption not hold for them?: