Longin Jan Latecki ( latecki@temple ) Computer and Information Science s

1. Context-dependent Detection of Unusual Events in VideosbyGeometric Analysis of Video Trajectories Longin Jan Latecki (latecki@temple.edu) Computer and Information Sciences Temple University, Philadelphia Nilesh Ghubade and Xiangdong Wen (nileshg@temple.edu)

2. Agenda • Introduction • Mapping of video to a trajectory • Relation: motion trajectory  video trajectory • Discrete curve evolution • Polygon simplification • Key frames • Unusual events in surveillance videos • Results

3. Main Tools • Mapping the video sequence to a polyline in a multi-dimensional space. • The automatic extraction of relevant frames from videos is based on polygon simplification by discrete curve evolution.

4. Mapping of video to a trajectory • Mapping of the image stream to a trajectory (polyline) in a feature space. • Representing each frame as: ……… X-coord of the Bin’s centroid Y-coord of the Bin’s centroid Bin’s Frequency Count Frame 0 Bin0 Bin n Frame N Bin n

5. Used in our experiments • Red-Green-Blue (rgb) Bins • Each frame as a 24-bit color image (8 bit per color intensity): • Bin 0 = color intensities from 0-31 • Bin 1 = color intensities from 32-63 • Bin 8 = color intensities from 224-255 • Three attributes per bin: - • Row of the bin’s centroid • Column of the bin’s centroid • Frequency count of the bin. • (8 bins per color level * 3 attributes/bin)*3 color levels = 72 feature

7. Theoretical Results: Motion trajectory  Video trajectory Consider a video in which an object (a set of pixels) is moving on a uniformbackground. The object is visible in all framesand it is moving with a constant speed on a linear trajectory.Then the video trajectory in the feature space is a straight line. If n objects are moving with constant speeds on a linear trajectory,then the trajectory is a straight line in the feature space.

8. Consider a video in which an object (a set ofpixels) is moving on a uniform background. Then the trajectoryvectors are containedin the plane. If n objects are moving, then the dimension of the trajectory is at most 2n. If a new object suddenly appears in the movie, the dimension of the trajectory increases at least by 1 and at most by 3.

9. MovingDotMovieWithAdditionalDot.avi

10. Robust Rank Computation Using singular value decomposition, based on: C. Rao, A. Yilmaz, and M.Shah.View-Invariant Representation and Recognition of actions.Int. J. of Computer Vision 50, 2002. M. Seitz and C. R. Dyer.View-invariant analysis of cyclic motion. Int. J. of Computer Vision 16, 1997. We compute err in a window of 11 consecutive frames in our experiments.

11. MovingDotMovieWithAdditionalDot.avi

12. Interpolation of video trajectory MovingDotMovie_Clockwise.avi

13. MovingDotMovieWithAdditionalDot.avi

14. Polygon simplification Relevance Ranking Frame Number 0 1 1 100 Frames with decreasing relevance 98 12 99 5

15. Discrete Curve Evolution P=P0, ..., PmPi+1 is obtained from Pi by deleting the vertices of Pi that have minimal relevance measure K(v, Pi) = K(u,v,w) = |d(u,v)+d(v,w)-d(u,w)| v v w w u u

16. Discrete Curve Evolution: Preservation of position, no blurring

17. Discrete Curve Evolution: robustness with respect to noise

18. Discrete Curve Evolution: extraction of linear segments

19. Key Frame Extraction

20. Key frames and rank Security1 • Bins Matrix • Distance Matrix

21. err for seciurity1 video

22. M. S. Drew and J. Au: http://www.cs.sfu.ca/~mark/ftp/AcmMM00/

23. Predictability of video parts:Local Curveness computation We divide the video polygonal curve P into parts T_i. For videos with 25 fps:T_i contains 25 frames. We apply discrete curve evolution to each T_iuntil three points remain: a, b, c.Curveness measure of T_i: C(T_i,P) = |d(a, b) + d(b, c) - d(a, c)| b is the most relevant frame in T_i and the first vertex of T_i+1

24. security7

26. err for seciurity7

27. 2D projection by PCA of video trajectory for security7

28. Mov3

31. Mov3: Rustam waving his hand. • Bins Matrix Keyframes = 1 378 52 142 253 235 148 31 155 167 • Distance Matrix Keyframes = 1 378 253 220 161 109 50 155 149 270

32. Hall_monitor

34. err for hall_monitor

35. Hall Monitor: 2 persons entering-exiting in a hall. • Bins Matrix Keyframes = 1 300 35 240 221 215 265 241 278 280 • Distance Matrix Keyframes = 1 300 37 265 241 240 235 278 280 282

36. CameraAtLightSignal.avi

37. Multimodal Histogram Histogram of lena

38. Segmented Image Image after segmentation – we get a outline of her face, hat etc

39. Gray Scale Image - Multimodal Original Image of Lena

40. Thank you