Extracting features from spatio-temporal volumes (STVs) for activity recognition

1. Extracting features from spatio-temporal volumes (STVs) for activity recognition Dheeraj Singaraju Reading group: 06/29/06

2. Motivation for dealing with STVs • Optical flow based methods would be able to capture only first order motion. • Methods that use HMMs deal with single point trajectories that carry only motion information and no spatial information We aim at a direct scheme for event detection and classification that does not require feature tracking, segmentation or computation of optical flow We want to detect points in the space-time volume which have significant local variation in both space and time.

3. Approaches that we shall discuss • On Space-Time Interest Points; Ivan Laptev • Local image features provide compact and abstract representations of images, eg: corners • Extend the concept of a spatial corner detector to a spatio-temporal corner detector • Actions as Objects: A Novel Action Represenation; Alper Yilmaz and Mubarak Shah • Concepts of differential geometry: Extract features from the STV based on local variations in curvatures of points on the volume • The curvatures show invariance to rotation and translation

4. Detecting interest points in space • An image can be modeled by its linear scale representation as follows • To look for interest points one analyzes the matrix of 2nd moments : A more familiar form of the matrix

5. Detecting interest points in space (contd.) • We want to choose corners in the image since they have significant spatial variation. • We therefore detect positive maxima of the following function How do we detect interest points in space-time ?

6. Results of detecting interest points in space • Detecting interest points in space gives interest points in the stationary background also • We want to find interest points that have information in the space as well as the temporal domain.

7. Detecting interest points in space-time • A spatio-temporal image sequence can be modeled by its linear scale representation as follows • Note that there are different scales for the spatial and the temporal scale, i.e. and respectively

8. Detecting interest points in space-time (contd.) • To look for interest points one analyzes the matrix of 2nd moments : • We therefore look for the maxima of the following spatio-temporal corner function

9. Results of detecting interest points in the STV • Consider a synthetic sequence of a ball moving towards a wall and colliding with it • An interest point is detected at the collision point

10. Results of detecting interest points in the STV • Consider a synthetic sequence of 2 balls moving towards each other • Different interest points are calculated at different spatial and temporal scales coarser scale

11. Effects of scales on interest point detection Long temporal events are detected for large values of while short events are detected for small values of Long spatial events are detected for large values of while short events are detected for small values of

12. Scale selection in space-time • We consider a prototype event modeled by a spatio-temporal Gaussian blob • The scale space representation of f is hence given by

13. Scale selection in space-time (contd.) • We want to find a differential operator that assumes simultaneous extrema over spatial and temporal scales that are characteristic of this Gaussian prototype event • To recover the spatio-temporal extent of f, we consider second order derivatives of L normalized by the scales as: • By solving for the fact that the above normalized 2nd order derivatives assume maxima at scales and we get a =1, b= ¼,c= ½ and d= ¾.

14. Scale selection in space-time (contd.) • We therefore define a normalized spatio-temporal Laplace operator as follows: • The following plots show that the zero crossings correspond to the maxima that are detected at and

15. Scale adapted space time interest points • So far we have found events that are local extrema in the space time volume at a particular choice of space and time scales • We would like to detect interest points that are extrema over the space time volume as well as over the scale of the scale-normalized Laplace operator • The reason for doing so is that different events would in general have different spatial and temporal extents

16. Algorithm for detecting interest points

17. Results on a previously used synthetic example Note that all the extrema are detected irrespective of their spatial and temporal extents DOUBT Why are these points not detected as interest points ?

18. Results of the algorithm on real seq. Note that events of all spatial and temporal extents are captured. The size of the circle shows the spatial extent of the event

19. Results of interest pt. detection Note that the regularity and extent of the spatio-temporal interest points is actually representative of the true events in time

20. Classification of events • Every interest point is described by its local spatio-temporal neighbor and we compare neighborhoods of events to classify events • The neighborhood of an interest point is defined by evaluating the following event descriptors This normalization guarantees the invariance of the derivative response to image scaling

21. Classification of events (contd.) • To compare two events, we compute the Mahalanobis distance between their descriptors as • To detect similar events in the given data, we apply k-means clustering to the event descriptors and thus detect groups of interest points with similar spatio-temporal neighbourhoods • Once the cluster centers are evaluated from the training data, given a new event, we evaluate its distance from the cluster centers. If the distance from all the centers is above a threshold we declare it as a background event.

22. Results of classification

23. Recognizing gaits • We extract the following features from the spatio-temporal volume • Positions of the interest points: • The corresponding scales: • The class of interest points: • We introduce a state for the model determined by the vector , where the variables are • Position of person in the image: • His/her size: • Frequency of the gait: • Phase of the gait at current moment: • Temporal variations of

24. Recognizing gaits (contd.) • We then have the following model for walking • Such a model helps handle translations as well as uniform rescaling in the image and the temporal domain

25. Recognizing gaits (contd.) • Given a model state X, a current time , a length of time window , and a set of data features detected from the recent time window , the match between the model and the data is defined by a weighted sum of distances h between the model features and the data features . • is a data feature minimizing the distance h for a given and is the variance for the exponential function.

26. Recognizing gaits (contd.) • To find the best match between the model and the data, we search for the model state that minimizes

27. Summary of the approach • An interest point detector is developed that finds local image features that show high variation of the image values in space and in time • The spatio-temporal extents of detected events can be estimated by using a normalized Laplacian operator • The neighborhoods of the events are described using scale invariant spatio-temporal descriptors • Different actions are then compared by checking for the matches between the event descriptors

28. Actions as objects: Action sketches • This methods analyzes the spatio-temporal volume by using the differential geometric surface properties such as peaks, pits, valleys and ridges • The authors claim that these are important action descriptors as they capture both spatial and temporal properties • These descriptors are related to the convex and concave parts of the object contours and/or to the maxima in the spatio-temporal curvature of a trajectory, and are hence view invariant.

29. STV: a collection of contours • In this approach the spatio-temporal volume is really a hollow solid object whose boundaries are defined by the contours of the boundaries of a person in every image frame. • It is assumed that the STV can be considered as a manifold, which helps us to consider small neighborhoods around a point to be nearly flat. • Since the STV is really the time evolution of a contour, we can define a 2D parametric representation by considering arc length s of the contour and time t.

30. STV: a collection of contours (contd.) t varying, s fixed s varying, t fixed The STV is a continuous representation in the normalized time scale and it does not require ay time warping for matching two sequences of different lengths.

31. Action descriptors • We want to compute action descriptors that correspond to changes in direction, speed and shape of parts of contour • Changes in these quantities are reflected on the surface of the STV and can be computed using differential geometry by identifying different landmarks. • These landmarks can be classified by basis of the local curvatures at points on the STV

32. Action descriptors (contd.) • Differential geometry gives us the concept of Gaussian Curvature K and Mean Curvature H that can be evaluated at points on the manifold of the STV. These curvatures exhibit invariance to algebraic transformations such as translation and rotation. • Local extrema of these curvatures can therefore be used to identify interest points for describing actions

33. Action descriptors (contd.) • The following table shows the different surface types and their associated curvatures

34. Analysis of action descriptors • We consider three types of contours: concave contours, convex contours and straight contours • The following contours generate typical landmarks in the spatial-temporal volume • Straight contour: ridge, valley or flat surface • Convex contour: peak, ridge or saddle ridge • Concave contour: pit, valley or saddle valley Shapes generated from straight contours

35. STVs corresponding to hand motion The STV generated by a hand staying stable. Such a motion (or lack of it) creates a ridge

36. STVs corresponding to hand motion The STV created by a hand that first moves downwards and then upwards. Note that a saddle ridge is created at the point of change of motion

37. Properties of the event descriptors • The landmarks discussed so far are essentially produced due to stable motion or change in stable motion. The stability of motion enforces that the STV is smooth enough so that one can consider valid local planar neighborhoods at points • Some of the landmarks are related to the curvature of the point trajectories and body contours as follows

38. View invariance of event descriptors • Since the landmarks are associated with extrema of local curvatures, even when the view changes the transformed landmarks are extrema in the new STV DOUBT: Not very confident about the derivation of the above • Due to this view invariance, comparing two STV volumes is equivalent to checking if there is a valid Fundamental Matrix relating the set of event descriptors in 2 given action volumes. Derived formula relating curvatures of corresponding points in 2 different views

39. Comparing two actions • We check if a linear system of the following kind is satisfied by the event descriptors in both the actions • This boils down to checking if the last singular value of A is 0. From a set of possible matches between the input action sketch and the known action sketches, we select the action with the minimum matching score

40. Summary of the approach • Using concepts of differential geometry, extract interest points; action sketches that have local spatiotemporal information by virtue of being local extrema of curvatures in space-time • These event descriptors are associated with uniform motion or stable changes in uniform motion • Since the action sketches are view invariant, comparing 2 actions is equivalent to checking if there is a valid Fundamental Matrix relating the positions of the action sketches for the individual actions.