1 / 55

EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance

EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance. Dong Xuan, Ph.D. The Department of Computer Science and Engineering The Ohio-State University http://www.cse.ohio-state.edu/~xuan

maxine-quinn
Download Presentation

EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance Dong Xuan, Ph.D. The Department of Computer Science and Engineering The Ohio-State University http://www.cse.ohio-state.edu/~xuan Key Collaborators: Yuan F. Zheng, Jin Teng, Junda Zhu, Xinfeng Li, Boying Zhang and Qiang Zhai

  2. Outline Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks 2/55

  3. Visual Surveillance • Important for protecting public and personal security 3/55

  4. Visual Surveillance • Massive deployment in urban areas • Over 500 surveillance cameras in a Philadelphia neighborhood (below ) • New York has 4176 video cameras in lower Manhattan area [1]. 4/55

  5. Surveillance in Action Anomaly? Suspicious Action? Online monitoring Offline retrieval Finding all white males in red, medium stature, from Mon through Fri last week 5/55

  6. Failure Examples Chicago police installed 10,000 surveillance cameras in the city, only 1 of 200 crimes is captured by the visual surveillance [2]! In San Francisco, in the first three years after the city installed cameras, they helped police charge suspects in a grand total of six cases [2]! One of the bombers in London bombing (July, 2005) is not identified by the surveillance system and escaped [3]! 6/55

  7. Why fail? Visual technologies are not efficient and accurate enough to do automatic localization and tracking, and a lot of human power is needed! • Large volume of video data • Temporal: 2.07*106 frames per camera per day • Spatial: tons of surveillance cameras in a city • Monitored objects may be visually occluded or have multiple inconsistent appearance 7/55

  8. Outline Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks 8/55

  9. Our Methodology: E-V Integration Combining electronic and visual signals, such as GSM, 3G, WiFi, Bluetooth and NFC signals together for efficient surveillance E-V Integration makes it possible to efficiently and accurately localize and identify objects in large volume of video data 9/55

  10. Visual Signal-based Surveillance Who are they? • Can accurately localize and continuously track a person • But who is he or she? • Difficult to recognize people, e.g., through face recognition 10/55

  11. Electronic Signal-based Surveillance • Besides data exchanged, these communication channels also contain unique and identifiable electronic identities: • IMEI (International Mobile Equipment Identity), IMSI (International Mobile Subscriber Identity), WiFi, Bluetooth MAC addresses, RFID Number. Electronic Signals 11/55

  12. Pervasiveness of Electronic Signals Number Units Sold (millions) Source:Technology Review, Sept/Oct 2011 • Electronic signals are emitted by many mobile device • Mobile device’s popularity is increasing • Smartphone as an example: 450 million shipped in 2011 12/55

  13. Electronic Signal-based Surveillance -The with SIM card number 358985010745743 is here, -But the error can be as large as 100 meters Interferences, e.g., vehicles, building, humans etc. • Cannot accurately localize a person with a mobile device • Large error in localization due to interference • But can identify the device through electronic identifiers 13/55

  14. EV-Surv: A Bird’s Eye View 14/55

  15. EV-Surv: the Workflow • User Description: • -Features, Clothing • -Electronic Identity • Time Range • Area Visual signal Electronic signal Backend Database Visual signal Gateway E-V Integration Electronic signal Other Information Databases E-V Retrieval Visual signal Electronic signal All frames relevant to user inquiry 15/55

  16. Outline • Deficiency of Visual Surveillance Systems • A Brief of Our EV-Surv System • Case Studies • EV-Retrieval • EV-Tracking • A Broader View of Our EV-Surv System • Final Remarks 16/55

  17. Case Study I: EV-Retrieval Jin Teng, Junda Zhu, Boying Zhang, Dong Xuan and Yuan F. Zheng, “E-V: Efficient Visual Surveillance with Electronic Footprints”, IEEE INFOCOM 2012 Introducing electronic signals to help sort out videos for accurate and efficient person identification 17/55

  18. Person Identification A missing child A crime suspect Problem: Given the identity of a person, find his appearance Reference Photo Appearance in video • How a person of interest looks like at the time of surveillance • May be very different from any image or video of him in record • A type of retrieval 18/55

  19. Traditional ways Too Costly • Let people search through a huge pile of videos • If done automatically, computers need to extract all human figures in each frame and compare. 19/55

  20. E Signal-assisted Retrieval • With complementary electronic information, we can find out the electronic identity of the person, and use that information to guide our visual search • In critical surveillance context, it is possible to acquire this information • E.g., the government can request service providers to disclose the user information in anti-terrorism operations • Or, sometimes, it may be the case that we only have an electronic identity • FBI gets a suspicious conversion through phone tapping • Much smaller search space after screening with E signals! • Less processing burden on the V side 20/55

  21. Problem Formulation: Notations • V-sensing: V-ID and V Frame • V-ID: Visual identity, such as human figure • VID*: Our target V-ID • V Frame: a set of V-IDs with some background captured by visual sensors (cameras) in certain area and time • E-sensing: E-ID and E Frame • E-ID: Electronic identity such as MAC address etc. • EID*: Our target E-ID • E Frame: a set of E-IDs captured by electronic sensors in certain area and time 21/55

  22. Problem Formulation • Input: EID*, and a set of E frames and corresponding V frames • Output: VID* in video frames • We consider a baseline case with perfect E-IDs and V-IDs • All E-IDs and V-IDs are clear and distinguishable. No ambiguity. • No false positives or negatives in the detection and extraction of E-IDs and V-IDs • We will discuss more practical cases later 22/55

  23. A Basic Solution • Three steps: • Step 1: Find out all E frames which include EID* • Step 2: Find a subset of E frames, whose intersection is EID* • Step 3: Identify VID* in their corresponding V frames • Comments: Few V frames to process because V frames without VID* are filtered out, but there may be still many V frames 23/55

  24. Example Millions of E Frames E frame N+1 E frame 1 E frame 1 E frame 1 E frame 1 E frame 4 E frame 1 E frame 3 E frame 1 E frame 1 E frame 1 E frame 2 E frame 1 EID4 EID3 EID3 EID* EID* EID* EID* EID* EID* EID* EID* EID13 EID* EID* EID* EID* EID* EID2 EID1 EID5 EID3 EID7 EID3 EID4 EID3 EID2 …… EID12 EID3 EID91 EID2 EID2 EID2 EID2 EID2 EID2 EID2 EID4 Step 1: Extract all E frames with EID* …… 24/55

  25. Example (cont’d) Step 1 E frame 2 E frame 3 E frame 4 EID* EID2 EID* EID2 EID* EID5 EID3 V frame 1 E frame 1 E frame 1 Step 2 EID1 VID1 Step 3 EID* EID1 EID2 VID2 EID* VID* EID2 VID2 EID3 VID3 E frame 3 V frame 3 E frame 2 V frame 2 25/55

  26. A Better Solution • Find the minimum number of E Frames, whose intersection is the given E-ID, i.e. EID* • Further less frames for V side processing E frame 1 E frame 1 EID1 EID* EID1 EID* EID2 EID3 E frame 2 E frame 3 E frame 2 EID* EID3 EID* EID2 Two E Frames are enough identify EID* through intersection. EID2 26/55

  27. Nature of E-Filtering At least one 0 in each non-EID* column • Finding the minimum number of frames, whose intersection is EID* • NP-complete: equivalent to the set cover problem • Whether each E-ID appears in each E frame is summarized in a matrix, with 1 meaning ‘appear’ and 0 ‘not appear’. • At least one 0 in each non-EID* column • Use these 0s to ‘cover’ all non-EID* column (next page) 27/55

  28. Reduction to Set Cover Problem E frame 1 E frame 3 E frame 2 EID1 EID3 EID2 EID2 EID3 Set to be covered EID1 Cover EID2 EID2 Sets to cover Set Cover Problem: Find the fewest sets from a pool of sets (left hand side), whose union includes the set to be covered (right hand side) 28/55

  29. Solution: EDP Algorithm • Element Distinguishing Problem (EDP) • The element to be distinguished is EID* • Greedily select E Frames in which the most number of E-IDs can be told apart from EID* • In the example, the greedy algorithm will select e1 or e3 first, because we can tell two E-IDs are not EID* • Repeat the greedy selection until EID* is distinguishable 29/55

  30. EDP(cont’d) Approximation results can be achieved with the greedy heuristic algorithm for the set cover problem 30/55

  31. VID* Retrieval Find the corresponding VID* from the frames selected by E-Filtering. VID* is the only one that should appear in all the frames after E filtering. Find all V-IDs in the selected frames, then an intersection operation can give VID*. 31/55

  32. More Cases • Vagueness and completeness of V-ID/E-ID • Vagueness: reflect how clearly a V-ID/E-ID can be identified • Completeness: reflect if V-IDs/E-IDs are complete in a V/E frame (false positive/negative) √ √ √ √ √ √ √ √ The baseline case we have studied □ practical case of our focus addressed 32/55

  33. Practical Case I: Handling Vague V-IDs Same? • Vague V-IDs • Do not know for sure which person is which in different frames • Difficulty in the intersection of V frames to find VID* • Solution • nBM algorithm: find the VID with the largest probability of appearing in all V frames. 33/55

  34. The nBM Algorithm • Similarity matrix for all V-IDs which have appeared v1 v2 v3 • n-partite Best Match Problem (nBM) • Put all VIDs in different frames in n different circles • n-partite graph (right) 34/55

  35. nBM (cont’d) VID1 is in v2, and appears as VID2 VID1 is not in v2 • Maximum Likelihood matching • Given the observed VID1 … VIDm • Which VID is the best candidate • Calculate the probability of all VIDi across all V frames • Select the VID with the largest probability 35/55

  36. Solutions to Other Practical Cases Time 1 EIDi 0 smoothing 1 EIDi 0 1 EIDi 0 smoothing 1 EIDi 0 • Careful Deployment • Make sure that the coverage of the camera and the wireless detectors are roughly the same • nBM is probability based, so it is naturally resistant to noises • Select appropriate threshold in nBM for better tradeoff between noise resistance and performance • Generalized EDP • Handle missing/ghost E-ID • Introduction of fuzzy logic to improve the robustness of EDP • Use RSSI for estimation and smoothing 36/55

  37. Implementation • Real world implementation • One camera viewing from above to collect V frames • 1-3 laptops around sniffing the WiFi traffic to collect E frames • Tested on campus • Gymnasium • Library 37/55

  38. Case Study II: EV-Tracking • Combine E and V signals for more accurate localization and tracking • Preliminary work: EV-Loc Boying Zhang, Jin Teng, Junda Zhu, Xinfeng Li, Dong Xuan and Yuan F. Zheng. “EV-Loc: Integrating Electronic and Visual Signals for Accurate Localization”, in ACM MobiHoc12. 38/55

  39. EV-Loc for Localization E-IDi Visual Localization V-IDj Electronic Localization • Basic idea • Simultaneous E and V localization • Same localization result  E-IDi matches V-IDj • Can collect localization results over time and perform statistical matching 39/55

  40. Multiple Objects Localization Localized E object x3 y1 x1 Localized V object y1 x4 xi, yi: Localization results (coordinates) y3 x2 y4 x1 x2 x3 x4 π=(3, 1, 2, 4) y1 y2 y3 y4 Minimize sum of localization differences 40/55

  41. Nature of The Problem v1 e1 e2 v2 e3 v3 v4 e4 Objects’ EIDs Objects’ VIDs • Linear assignment problem (bipartite matching) • Minimum matching cost:argmin πiΣ||xi - yπi|| = ||x1 - y4|| + ||x2 - y3|| + ||x3 - y2|| + ||x4 – y1|| • We can use the Hungarian algorithm to solve this optimization problem 41/55

  42. EV-Loc for EV-Retrieval • In EV-Retrieval, we only use 0/1 to indicate E-IDs’ existence in E frames • However, we can improve it with EV-Loc • E-ID’s existence in a much smaller region • Easier for intersection 42/55

  43. EV-Loc for EV-Retrieval (cont’d) • Suppose all basestations can hear all mobile devices E frame 1 E frame 2 E frame 3 EID* EID* EID1 EID1 EID1 EID* E frame 1 EID* EID1 E frame 2 EID* EID1 EID1 EID1 EID* EID0 E frame 3 EID* EID1 Possible location Original Scheme Now with Localization 43/55

  44. Outline Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks 44/55

  45. EV Surveillance: Problem Space Uncooperative Cooperative Tracking Onsite Offline 45/55

  46. Problem Space (Cont’d) • X: Tracking: offline or onsite • Y: Object of monitoring: • Individual : non-coordinative • Group: coordinative, tied by relation in terms of vicinity, social relationship, appearance, action/behavior • Z: Object friendliness: • Cooperative: E/V signals on purposely for easy tracking • Uncooperative: neutral (E/V signal on/off follows its own cause), and even misleading • Other possible dimensions: objects can be human and vehicles etc. 46/55

  47. Typical Cases • Case 1: <X: offline tracking, Y: individual object, Z: cooperative), i.e. offline tracking of individual and cooperative object • E.g.missing elders searching • Case 2: <X: offline tracking, Y: group objects, Z: cooperative) , i.e. offline tracking of group and cooperative objects • E.g. public health monitoring • Case 3: <X: onsite tracking, Y: individual/group objects, Z: cooperative), i.e. onsite tracking of individual/group and cooperative objects • E.g. sports training/traffic monitoring • Case 4: <X: offline/onsite tracking, Y: individual object/group objects, Z: uncooperative, i.e. tracking of uncooperative objects • E.g. criminal tracking 47/55

  48. Open Issues • E-V sensor deployment • Electronic signal capturing • E-V data analysis • Privacy • Real-world implementation and evaluation 48/55

  49. Electronic and Visual Sensor Deployment Electronic sensor deployment problem Visual sensor deployment problem How to do E/V sensors joint optimal deployment? 49/55

  50. Electronic Signal Capturing Mobile Devices emit electronic signals at different time Electronic interference is always there Non-cooperative targets exist Different electronic signal capturing times vary 50/55

More Related