An Interactive Framework for Retrieving Incidents in Transportation Surveillance Video Databases

1. (xcentroid, ycentroid) xt-2 xt-1 xt fdk An Interactive Framework for Retrieving Incidents in Transportation Surveillance Video Databases Xin Chen Advisor: Dr. Chengcui Zhang Department of Computer and Information Sciences, University of Alabama at Birmingham 1. Introduction and Motivation 3. Object Tracking & Trajectory Modeling 2. Related Work & Merits of the Proposed Work Goal:Learn and retrieve semantic events in video databases such as accidents in transportation surveillance videos. Vehicle Segmentation and Tracking: • Segmentation -- Simultaneous Partition and Class Parameter Estimation (SPCPE) [1] algorithm. • Tracking -- Distinguish the static objects from mobile objects in the frame. Trajectory Modeling: • The Least Square Curve Fitting method is used to model the trajectories of vehicles. A trajectory is represented by a kthdegree polynomial: y = a0+ a1x + … +akxk • Advantage: The trajectory can be described by only a few coefficients. Derivatives on the curve are velocities. Related Work: • Three Major Learning Algorithms for Event Detection from Videos: • Hidden Markov Model (HMM)[5] • Belief Networks [4] • Self-Organizing Map (SOM) [7] • Relevance Feedback: • Rui et. al [6] proposed to use RF in Content-based Image Retrieval. • Neural Network is Used: • Mostly in forecasting trends in Time Series Data [2] • Rarely in detecting spatiotemporal patterns [3] Vehicle Segment Main Challenges : • “Semantic Gap”– A gap between high level semantic concepts and low level video features. • In Information Retrieval, there’s no prior knowledge to construct the training set for learning. Solution:Relevance Feedback (RF) [6] Fitted Curve Merits of the Proposed Framework: • Relevance Feedback is well-known in Content Based Image Retrieval. We apply it to the semantic video retrieval. • A mapping between spatiotemporal trajectories and the Neural Network input nodes is designed.The Neural Network model for time series forecasting is adjusted for spatiotemporal semantic events detection. Summary: • An interactive semantic video retrieval framework is proposed. • The user guides the learning and retrieval process through Relevance Feedback. • The Neural Network for Time Series data is the learning algorithm. • The proposed framework is especially useful in mining and retrieving data from large multimedia databases. • This framework can be tailored to apply to many fields. • Experimental results show the effectiveness. Trajectory: Centroids: Tracked Vehicles and Their Centroids 4. Event Modeling 5. Learning and Retrieval (1) Traffic Accidents: • Accident Features: • Sudden Change of the Direction (θ) • Sudden Change of the Velocity (vdiff) • Minimum Distance from the nearest vehicle (mdist) • A Sample Point:αi = [1/mdisti,vdiffi,θi] • Trajectory: α= [α1, α2,…,αn] Data Preparation: • Sampling: Sample centroids along a trajectory at the rate of 5 frames per sampling point. • Window Sliding: Extract trajectory segments by sliding window -- a way to partition time series data. Continuity of the data is kept. Neural Network Design: • Window Size: # frames is the typical sequence length of an event. • Input Nodes: • Hidden Layers: • One hidden layer with sigmoid transfer; The number of nodes equals that in the input layer. • One output layer with linear transfer; There is one output node that scores the likelihood of an event in the sequence. • Initial Weights: • First layer – random weights • Second layer – multiple linear regression weight initialization • Search Algorithm: • Conjugate Gradient The Proposed Learning Framework: • Based on the Neural Network for Time Series Data. • PredictionDetection • Relevance Feedback is incorporated. xt = αi= [1/misti, vdiffi, θi] 6. Learning and Retrieval (2) 8. Conclusions and Future Work 7. Experiments Learning and Retrieval Process: • Initial Iteration: • The user specifies an event of interest (e.g., traffic accidents). • There is no relevance feedback. • Top sequences are returned to the user by heuristic: • Subsequent Iterations: • The user gives feedback to the retrieval results. • Training data = [xt-2, xt-1, xt, fdk, opt] • The learning algorithm refines and returns the retrieval results to the user. • The whole process goes through several iterations until a satisfactory result is obtained. Experimental Results: Experiment Setup: • Tested on two video clips; One is taken in a tunnel featuring single vehicle accidents (2504 frames, 109 trajectory sequences); Another one is taken in an intersection in Taiwan, featuring multiple vehicle accidents (592 frames, 168 sequences). • Sampling rate is 5 frms per sampling point; Window size is 3. • Five iterations of user relevance feedback are performed - Initial (no feedback), First, Second, Third, and Fourth. • The top 20 video sequences are returned to the user. • The percentage of relevant sequences (accuracy) within the top 5, 10, 15 and 20 is calculated. • Compared with Weighted Relevance Feedback Method [6]. Conclusions: • A semantic video retrieval framework is proposed. • The neural network is applied to event detection from video sequences, a special type of time series data. • A mapping between spatiotemporal trajectories and network input nodes is developed. • The proposed work incorporates the Relevance Feedback in interactive video retrieval. Future Work: • The event models for other general events will be constructed and tested. • More video data will be collected with the associated metadata for normalizing all the videos before the storage and retrieval. • The framework will be extended to include query by example, query by sketches, and the customized combination of query types. Interface Fourth Iteration Retrieval Results of the 1st Clip (tunnel) System Overview References: [1] Chen, S.-C., Shyu, M.-L., Peeta, S., and Zhang, C. 2003. Learning-Based Spatio-Temporal Vehicle Tracking and Indexing for Transportation Multimedia Database Systems. IEEE Trans. on Intelligent Transportation Systems, Vol. 4, No. 3, pp. 154-167. [2] Davey, N., Hunt, S.P., Frank, R. J. 2000. Time Series Prediction and Neural Networks. Journal of Intelligent and Robotic System, Vol. 31. [3] Gao, D., Kinouchi, Y., Ito, K., and Zhao, X. 2005 Neural Networks for Event Extraction from Time Series: a Backpropagation Algorithm Approach. Future Generation Computer Systems, Vol. 21, pp.1096-1105. [4] Huang, T., Koller, D., Malik, J., Ogasawara, G., Rao, B., Russell, S., and Weber, J. 1994. Automatic Symbolic Traffic Scene Analysis Using Belief Networks. In Proceedings of National Conference on Artificial Intelligence. [5] Kamijo, S., Matsushita, Y., and Katsushi I. 2000. Traffic Monitoring and Accident Detection at Intersections. IEEE Transactions on Intelligent Transportation Systems (June 2000). Vol. 1, No. 2, pp. 108-118. [6] Rui, Y., Huang, T.S., and Mehrotra, S. 1997. Content-based Image Retrieval with Relevance Feedback in MARS. In Proceedings of the International Conf. on Image Processing, pp. 815-818. [7] Xie, D., Hu, W., Tan, T., and Peng, J. 2004 Semantic-based Traffic Video Retrieval Using Activity Pattern Analysis. In IEEE International Conference on Image Processing (ICIP). Fourth Iteration Retrieval Results of the 2nd Clip (intersection)