Multimedia Surveillance Data Mining for Analytics

1. Jai sri ganeshay namah sri saraswatyiy namah om namah shivay jai bajarang bali dada jai sai baba jai gaytri mata jai tirupati dada jai mahalaxmi mata jai kuldevi ma chamunda mata jai surya chandra mangal budh guru sukra sani rahu ketu devta santi bhavantu jai mariyamma mata jai anjnaiya dada jai mahavir swami bhagvan jai swami swarupanand saraswati ji bhagvan jai dakor na thakorji jai servdevodevi namah stute om shanti bhavantu Multimedia Surveillance Data Mining for Analytics

2. Outline • Motivation • Introduction • Problem Definition • Proposed Approach for Evacuation Scenario • Statistical data mining Model • Results Obtained on VAST challenge dataset • Future Work

3. Motivation • Wide use of surveillance system for monitoring the behavior of people, vehicles • Objective: To detect suspicious behavior based on available multimodal data • Strong need for automated or semi automated means for suspicious behavior detection and prediction

4. Introduction • Video Surveillance Systems • Expensive • Rich amount of information • RFID Surveillance Systems • Not very expensive • Limited amount of information Therefore can use appropriate sensory data for the task at hand and can even use multiple modalities for redundancy and cost-savings

5. Introduction • Suspicious movement detection scenarios • Explosion event followed by evacuation • Open firing event followed by chaos • Even a small accident in office or street leads to considerable change in normal movement pattern • Need quick way of analyzing and also the way of predicting suspicious behavior

6. Introduction • Video Surveillance Systems • Observing large volume of data by a few observers • Suspicious patterns may not be explicitly visible to observer • RFID Surveillance Systems • Suspicious patterns are not visible to observer Therefore some automated pattern analysis or data mining is required

7. Problem Definition • To build an intelligent surveillance system’s tool that can, • Help investigate suspicious behavior for different scenarios, • Automatically or semi automatically incorporating the intuitions that are similar to the one that security officer can have. • Where investigation should give answers to when?, where?, who?, what? etc.

8. Evacuation Dataset of IEEE VAST Challenge 2008 • In 2007 an explosive device was set off at a Miami, Florida DOH building, resulted in casualties and damage • Employees & visitors wore badges (RFID) • Data provided • Time: Ticks, representing intervals between tag readings • Person Id: Tag identification of all employees and visitors • Xcor: the location x-coordinate • Ycor: the location y-coordinate • The file includes data before and throughout the incident.

9. Input Trajectory Data • Trajectory of 82 people over total Time Duration of 837seconds on building map of 91x61 grid space. • Making sense of this data seems extremely difficult

10. Questions for the Evacuation Scenario • Where was the device set off? • Identify potential suspects and/or witnesses to the event. • Identify any suspects and/or witnesses who managed to escape the building. • Identify any casualties.

11. Proposed Approach • Gather intuitions (hypotheses) for the scenario • Compute the possibly useful parameters like average speed in certain time interval, average traversed area in certain time interval • Build a statistical model using the computed parameters combined with the hunches • Perform Analysis

12. Intuitions for the Evacuation Scenario • Evacuation Scenario in office environment where explosion event is followed by evacuation. • Intuition 1 [Normal Behavior]: • Usual movement of people will be low before explosion event and it will increase drastically afterwards to evacuate the scene.

13. Intuitions for the Evacuation Scenario • Intuition 2 [Suspicious Behavior]: • Suspicious persons would try to run away from explosive device location before the explosion happens. • Intuition 3 [Victims Behavior]: • Victims would have normal behavior before the explosion event but will be injured or have fainted or be dead on explosion.

14. Formulation of Statistical Model • Parameters for Statistical Model: • Time Window: The analyst needs to input appropriate time window parameter for the statistical model to compute the following • Speed of each Person • Area Traversed by each Person • Average Global Speed of People • Average Global Area Traversed by People

15. When did the Explosion happen? • Obtain the Global Average Speed. • Find the Global Maximum value from • Based on intuition1 we can consider this GM as approximate start time of Explosion

16. Where was the device set off? • Average speed and Average area traversed by the Victims will be almost near to zero after explosion event. • They may not be able to reach to the Evacuation Area. • They will be found within or very near to the explosion area. • Location cluster of such people represents the area of explosive device.

17. Where is Evacuation Location? • Based on intuition1 people are trying to reach to evacuation place. • High density region at end times would be representing evacuation place.

18. Who are the Suspects? • Average speed and Average area traversed by the persons will be higher before explosion event. • Suspicious person should have visited Explosion location just prior to the explosion. • They might either reach Evacuation before others or will escape without entering Evacuation area.

19. INPUT DATA Time & Location of each person Computing required Parameters ( speed, area Covered within time window) Finding the Start Time Of Event (Explosion) Analyzing the speed before Event (Explosion) Analyzing the speed after Event (Explosion) High speed people in this duration is set of Suspicious people Low speed people in this duration is set of Victims Traversed through Event (explosion location) are strong set of suspects Clustered at event (explosion) location Evacuation Model

20. Future work • Need to incorporate other data captured • Video data • Audio data • Fire Alarm, Temperature data etc. • Come up with a Mining/Analytics tool to facilitate such investigations.

21. Thank You

22. Definition • Data mining: • “is the process of automating information discovery” or • “is the exploration and analysis by automatic or semiautomatic means, of large quantities of data in order to discover meaningful patterns and rules” • “multimedia data mining” • “knowledge discovery in a multimedia database” • “extraction of implicit knowledge, mm data relationships or other patterns not explicitly stored in multimedia files”

23. Motivation • Tremendous benefits of traditional data mining is proven for structured data. • Now its time for extending the mining techniques for unstructured, heterogeneous data.

24. MDM Challenges and Problems • Feature Selection Dimensionality Reduction: for reducing the problem size , enables learning algorithms to operate faster and effectively. • Feature construction / transformation: by constructing new features from the basic features set. • How to analyze the heterogeneous data that consist of text, graphs, images, sounds, videos and other kind of sensor data? Multimedia data has complex structures that can not be processed as a whole by available data mining algorithms. • Tokenizing textual document into words and phrases has proven to work reasonably well for retrieval but images, audio, video etc cannot be readily decomposed into such semantic units.