Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks

1. Thrust Technical Review March 2012 Distributed Sensor Exploitation 6.1Rapid Deployment of Smart Camera Networks Dr. B.S. Manjunath University of California, Santa Barbara manj@ece.ucsb.edu 805-893-7112

2. Distributed Sensor Exploitation 6.1Rapid Deployment of Smart Camera Networks 2012 Thrust Area Technical Review • RESEARCH QUESTIONS: • How can we geo-calibrate a stationary network using mobile sensors and associated GPS and other information? • What are the issues in discovering the geometric/spatial and temporal connections and constraints within a camera network with overlapping and non-overlapping views? • What information needs to be computed at the smart sensor nodes for activity analysis without distributing the raw video? • When and how to fuse multi-modal information, e.g., mobile, airborne, and fixed cameras, for robust object tracking and activity analysis? MILITARY RELEVANCE/OPERATIONAL IMPACT: • Rapid deployment of a network of smart cameras and UAVs can help monitor any potential area, especially in high risk and remote sites. NAVAL S&T FOCUS AREAs ADDRESSED: • Asymmetric & irregular warfare UAV: Provides intelligence from overhead Rapidly Deployed Smart Camera: Provides ground-based views and distributed image analysis SCHEDULE: • TECHNICAL APPROACH: • Geo-calibration using mobile sensors • Auto-calibration and network topology discovery using visual analysis • Detecting and tracking objects in a camera network through fusion of multiple views, despite wireless communication challenges • Analyzing events over non-overlapping camera views • PERFORMER: • University of California at Santa Barbara • B. S. Manjunath, Professor FY11 FY12 FY10 TASKS Camera network setup Auto-calibration and topology Object detection & 2D tracking Object tracking in 3D Event & behavior detection Demos on campus testbed TTA: N/A TECH TRANSITION PATH: DYNAMIC WIKI/RESOURCE MANAGER (D&I)

3. Top Level POA&M (Summary Task) 2012 Thrust Area Technical Review Field deployment Final demonstration

4. Project Work Breakdown Structure 2012 Thrust Area Technical Review

5. Detailed Technical Approach • Objective • Development of a rapidly deployable network of smart camera sensors to gather mission-specific information for surveillance and human activity analysis • Challenges • Minimize human interaction • Wireless communication constraints: limited bandwidth, unpredictable latency, dropped packets, and time synchronization • Few examples to train the system

6. Detailed Technical Approach Project Roadmap

7. Detailed Technical Approach • Project Introduction • Overview of the UCSB Campus Camera Network • Infrastructure funded by a ONR DURIP award • Implementation by students on this project • Research Accomplishments • Distributed Geocalibration • Distributed tracking and fusion • Topology discovery and network data summarization • Browsing and search in a distributed network • Summary and Conclusions

8. Detailed Technical Approach • Camera Placement: Where to place the cameras? • Using prior information, such as GPS tracks, to optimally place cameras • Novel approach to camera placement based upon “optimal” reconstruction of tracks and activities with selected observations • Calibration and Topology: How are the cameras related? • Necessary for most multi-camera applications to understand how the cameras are related in a physical space (calibration) or relatively according to their activities (topology) • Novel methods for calibrating additional cameras in a large calibrated network • Novel methods for calibrating cameras in a network individually, but in a global coordinate system without entering the scene • Examined topology methods on our network

9. Detailed Technical Approach • Object Tracking: How to track or search objects? • Tracking or searching objects in a multi-camera network requires taking advantage of all overlapping and relevant information • Novel method for distributed, online multi-camera tracking using multiple instance learning and particle filtering with appearance and geometric fusion. • Novel trajectory searching with graph modeling • Event/Behavior Analysis: How to perform activity analysis over a distributed camera network? • The ultimate goal of a wide-area network is to make high level decisions and recognitions across many cameras • Novel camera network summarization algorithm • Addressing the relatively unexplored problem of activities analysis in a large, wide-area network with many non-overlapping views

10. Topic relationships Geo-calibration Topology discovery N+1 camera calibration Wide-area activity analysis Network summarization Object searching Distributed object tracking Camera placement

11. UCSB Camera Network Examined challenges through manual network setup Developing optimal camera placement algorithm based on GPS trajectories

12. Project Technical Assesment Recent Progress • New Geo-calibration formulation with mobile sensors • Distributed Object Tracking on a Camera Network • Video Summarization in a Multi-camera setting • Distributed browsing and search 12

13. Project Technical Assesment - 1 • Project Introduction • Overview of the UCSB Campus Camera Network • Infrastructure funded by a ONR DURIP award • Implementation by students on this project • Research Accomplishments • Distributed Geocalibration • Distributed tracking and fusion • Topology discovery and network data summarization • Browsing and search in a distributed network • Summary and Conclusions

14. 1. Geocalibration using mobile devices

15. Geo-Calibration How do you calibrate cameras with non-overlapping views?

16. Geo-Calibration • Geo-Calibration is the process of finding the projective geometry of a camera in a global coordinate system. • Traditional, image-based methods are in a local coordinate system and cannot be shared across different calibrations1 • Existing methods for integrating global coordinates involve GPS tagged objects in the scene or aligning with a map2 1. Zhang, “A flexible new technique for camera calibration,” PAMI 2000 2. Kaminsky et al. “Alignment of 3D Point Clouds to Overhead Images.” W on Internet Vision 2009.

17. Geo-Calibration Method • Take pictures of the camera’s scene using smartphones and collect location and orientation metadata • Location sensor: GPS • Orientation sensor: compass + accelerometer or gyroscope • No objects need to be placed in the scene • Data can be collected rapidly

18. Geo-Calibration Method • Geo-calibration: Given image(s) from a camera • Collect, using smartphone, calibration images with GPS and orientation metadata • Calibrate cameras in a local coordinate system using images • Transform from local to global GPS coordinates • Refine position and orientation estimates • Two approaches: centralized and distributed

19. Distributed Geo-Calibration • Assume GPS positions accurate, then average across all stereo/pairwise estimates • Rotation average using Karcher Mean • Position average using least squares d is the geodesic distance in rotational space SO(3) GPS Points Two-view estimate Least squares intersection 19

20. Simulation Results • 2 quarter circle rows of cameras every 5 degrees • Middle camera used as fixed camera • Random set of points around origin

21. Simulation Results GPS Noise Feature Point Noise Location Error from Orientation Noise Orientation Error from Orientation Noise

22. Real World Results • Smartphone: Data collected using HTC Evo smartphone • Accurate Sensors: Data collected using CyberQuad UAV sensors and camcorder • Error Distance compared against GPS/Sensor from data

23. Smartphone Results 23

24. Accurate Sensor Results

25. Geo-calibration MLE • New formulation as a maximum likelihood • Maximize the orientation and position of the fixed camera given: • Smartphone orientation position and location • Pairwise relationship based on the image measurements 25

26. Smartphone Measurements • Smartphone orientation calculated from compass and gyroscope/accelerometer • Smartphone position calculated from GPS with GPS error/accuracy 26

27. Pairwise Measurements • Find and match features points (Xi, Xq) to find a relative rotation and translation Relative rotation Relative position and unknown depth 27

28. Geo-calibration • Maximum likelihood of probability of fixed cameras orientation and position (Rq, tq) • Optimal Estimates 28

29. Consensus Framework • This estimate can be solved using a consensus algorithm • In a consensus algorithm, each smartphone has a state of the fixed camera’s orientation and location. The states are iteratively updated to a consensus state. 29

30. Geo-calibration Experiments • Mount a camera over GPS Benchmarks • Collect images and sensor metadata using a smartphone 30

31. Geo-Calibration Summary • Technical Report available for initial formulation, Kuo et al. “Map Me: Camera Geo-calibration using mobile devices” • Next: Experimentation on New Formulation

32. Project Technical Assesment- 2 • Project Introduction • Overview of the UCSB Campus Camera Network • Infrastructure funded by a ONR DURIP award • Implementation by students on this project • Research Accomplishments • Distributed Geocalibration • Distributed tracking and fusion • Topology discovery and network data summarization • Browsing and search in a distributed network • Summary and Conclusions

33. Recap: Topic relationships Geo-calibration Topology discovery N+1 camera calibration Wide-area activity analysis Network summarization Object searching Distributed object tracking Camera placement

34. Object Tracking and Search • In a smart camera network setup, • Distributed multi-camera object tracking • Given a target, how do the network of cameras with overlapping views robustly track the target with collaboration despite limited network bandwidth? • Efficient object searching in a large network • How to search for objects of interest efficiently without significant visual processing at search time?

35. Project Technical Assesment Distributed browsing & search

36. Browsing and Searching • Problem:  • A camera network deployed over a large area • No live streaming of any video • Local camera nodes have storage to archive video and limited processing power for simple video analysis • How a human image analyst at a distance central node interacts with the remote cameras? ? 36

37. Browsing and Searching • Envision the following application scenarios: • A user instantiates the interaction with the network by specifying regions on the image plane (cameras, time intervals) of interest. E.g., “FIND object instances related to region A FROM camera 1 OR region B FROM camera 4 between time 9:30am and 9:35am” • With the results from the previous scenario, the user could then identify one specific object of interest to initiate further searching for the same or related objects. E.g., “FIND all objects related to the object instance at region C FROM camera 1 at time 9:32:41.3am”. 37

38. Conventional Approaches • Try to provide human users with an high-level interface, such as dynamic global scene visualization and aim to detect/track all observed objects across the entire camera network • To deal with appearance variations across views, much prior work focused on finding the best matching criterion (Javed ICCV’03,Javed CVPR’05, Farenzena CVPR’10, Zhen CVPR’11, Rios-Cabrera CVPR’11.) • What if we cannot have reliable object detection and tracking? Any possible strategy without pair-wise matching or global trajectory finding? 38

39. Proposed System • Instead of trying to find global trajectories for every object visible in the network, model camera observation directly with a graph model • Act as an intermediate agent between distributed cameras and human image analysts and provide recommendations to the user with a concise and representative set of video snapshots captured by the camera network • Help the image analysts to browse, search and identify objects of interest by giving a canonical overview of the entire set of visual observations in the network 39

40. Proposed System 40

41. Real-time detection & tracking • Real time object detection with background modeling and tracking with mean-shift algorithm • An observation record is generated • for each detected and tracked object • by the camera and sent to the • central node over the network 41

42. Modeling camera observations • Given a user query, we need to find video frames with following properties: • Centrality, representative ones which are closely related to the query and many other observations and hence considered important. • Diversity, covering as many distinct groups as possible • A graph G(V, W) to model relationship among camera observations and perform unified graph ranking for different queries • Individual camera observations (i.e., frames with detected objects) form the vertices set V • Weight matrix W defines the strength of connectivity between camera observations • G(V, W) built at the central server incrementally as the new records are received in real time from the cameras. 42

43. Spatial Temporal Topology • Divide image plane into 8x6 blocks • Model the time delay for an object to travel between any two blocks across cameras with a Gaussian model with known mean and variance 43

44. Demonstration – Test bed • 11 camera nodes (Cisco wireless-G WVC2300) observing bike path • Each camera streams video to a dedicated computer to simulate a smart camera node • Approx. 600 meters in width/length 44

45. Browsing Demonstration A browsing example with regions of interest indicated by the rectangles in camera C8 and C9 45

46. Browsing Demonstration • Top 10 ranked frames (Decreasing order: left to right, top to down) • A total of 10 distinct objects satisfying the criterion. All of them have been identified (labeled in yellow). The 8th ranked frame is a “false positive” (it has not passed the queried regions within the specified time interval). 46

47. Searching Demonstration • Results when searching for object P6 (2nd -11th ranked frame)

48. Recap-Topic relationships Geo-calibration Topology discovery N+1 camera calibration Wide-area activity analysis Network summarization Object searching Distributed object tracking Camera placement

49. Forecasted Key Events UAV: Provides intelligence from overhead Rapidly Deployed Smart Camera: Provides ground-based views and distributed image analysis Soldier image courtesy of US Marines. Map image courtesy of Google Earth.

50. Project Technical Risk Assessment