Hanning Zhou (hanzhou@amazon) and Don Kimber (kimber@fxpal )

1. Unusual Event Detection via Multi-camera Video Mining Step 1: Temporal segmentation • Introduction • Goal: detecting unusual events from a large amount of multiple stream video. • Challenges: multi-stream video lack of labeled data • Approach: semi-supervised learning Hanning Zhou (hanzhou@amazon.com) and Don Kimber (kimber@fxpal.com ) static scene …… ………………………… ………………… ……… …… • Applications • Online • Detect unusual events alarm in surveillance system • Offline • Detect highlights from sport videos • Analyze business process • Other data streams • audio, text and multimodal data streams Segment k + 1 Segment k • Step 2: Feature extraction • Feature: • size and location of the motion blobs • clustered into GMMs • Advantage: • higher spatial resolution than motion histogram [Zhang ‘05] [Zhong ‘04] • New problem: spatial alignment • Solved with approximate KL-divergence [Goldberger ‘03] • Key Idea • Collaborative mining of multiple streams • Sensor network is prevailing • Events from different sensors are related • Two-Stage Training • Unusual events are rare • Manual labeling is intractable • Step 3: Training a Model for usual event • 1st Stage: Bootstrap • Clustering: keep the large clusters • User feedback: exam the small clusters • 2nd Stage: Train CHMM for usual events • Inference in CHMM is efficient O(T(CN)^2) vs O(TN^(2C)) • statistic model to handle variant durations, noisy observation and asynchrony • The hidden state depends on 3D location of the objects • The observations are 2D projection of the objects • CHMM as a loose stereo • Dependent chains • Previous Work • General Event Detection • specific event with well-defined model • supervised statistical learning (DTW, HMM, factor graph) • Unusual Event Detection • unsupervised [Zhong & Shi ‘04] • semi-supervised [Zhang et al. ‘05] • All above are from single stream • Experimental Results • Trained on one week’s video recorded in FXPAL • Tested on another week’s video • usual events: pass by, pick up print outs • unusual events: distribute mails, open multiple drawers to look for stationery, open the cabinet, multiple people • Quantitative experiments on Terrascope data • 48 segments in 4 scenarios • usual events: group meeting, natural video sequence • unusal events: group exit, intruder, theft, suspicious behavior The 3D location inferred from different cameras are RELATED Exact stereo does not work well, because: gaps between the views wide baseline, few correspondences Examples of usual events • Step 4: Detecting usual event • Evaluating the likelihood with forward-backward algorithm [Brand ‘97] Examples of unusual events ROC curve of HMM vs CHMM on Terrascope data Terrascope dataset camera setup Courtesy of Christopher Jaynes Mailroom camera setup