Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring

1. Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring Ken Goldberg and Dezhen Song (Paul Wright and Carlo Sequin) Alpha Lab, IEOR and EECS University of California, Berkeley

2. Networked Robots

3. internet tele-robot:

4. RoboMotes: Gaurav S. Sukhatme, USC

5. Smart Dust: Kris Pister, UCB (Image: Kenn Brown)

6. Networked Cameras

7. Security Applications Banks, Airports, Freeways, Sports Events, Concerts, Hospitals, Schools, Warehouses, Stores, Playgrounds, Casinos, Prisons, etc.

8. Conventional Security Cameras • Immobile or Repetitive Sweep • Low resolution

9. Fixed lens with mirror 6M Pixel CCD $ 20.0 K 1M Pixel / Steradian Pan, Tilt, Zoom (21x) 0.37M Pixel CCD $ 1.2 K 500M Pixel / Steradian New Video Cameras:Omnidirectional vs. Robotic

10. Where to look?

11. Sensor Net Detects Activity • Activity localization • “Motecams” • Other sensors: audio, pressure switches, light beams, IR, etc • Generate bounding boxes and motion vectors • Transmit to Robot camera

12. Block Motion Estimator Ron Fearing’s Xilinx FPGA board can compute motion vectors in hardware Extract: bounding frames

13. Viewpoint Selection Problem Given n bounding frames, find optimal frame

14. Related Work • Facilities Location • Megiddo and Supowit [84] • Eppstein [97] • Halperin et al. [02] • Rectangle Fitting • Grossi and Italiano [99,00] • Agarwal and Erickson [99] • Mount et al [96] • Similarity Measures • Kavraki [98] • Broder et al [98, 00] • Veltkamp and Hagedoorn [00]

15. Problem Definition Requested frames: i=[xi, yi, zi], i=1,…,n

16. 3z (x, y) Problem Definition • Assumptions • Camera has fixed aspect ratio: 4 x 3 • Candidate frame  = [x, y, z] t • (x, y)  R2(continuous set) • z  Z (discrete set) 4z

17. Problem Definition • “Satisfaction” for frame i: 0  Si  1  =   i  = i Si = 0 Si = 1

18. Similarity Metrics • Symmetric Difference • Intersection-Over-Union Nonlinear functions of (x,y)

19. Satisfaction Metrics • Intersection over Maximum: Requested frame i , Area= ai Candidate frame  Area = a pi

20. Intersection over Maximum: si( ,i) Requested frame i Candidate frame  si = 0.20 0.21 0.53

21. (for fixed z) Satisfaction Function • si(x,y) is a plateau • One top plane • Four side planes • Quadratic surfaces at corners • Critical boundaries: 4 horizontal, 4 vertical

22. Objective Function • Global Satisfaction: for fixed z Find * = arg max S()

23. Properties of Global Satisfaction S(x,y) is non-differentiable, non-convex, but piecewise linear along axis-parallel lines.

24. y x Approximation Algorithm Compute S(x,y) at lattice of sample points: d

25. Approximation Algorithm * : Optimal frame : Smallest frame at lattice that encloses* • Run Time: • O(w h m n / d2) : Optimal at lattice

26. x Exact Algorithm • Virtual corner: Intersection between boundaries • Self intersection: • Frame intersection: y

27. Exact Algorithm • Claim: An optimal point occurs at a virtual corner. Proof: • Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur atboundaries

28. Exact Algorithm Exact Algorithm: Check all virtual corners • (mn2) virtual corners • (n) time to evaluate S for each • (mn3) total runtime

29. Improved Exact Algorithm • Sweep horizontally: solve at each vertical • Sort critical points along y axis: O(n log n) • 1D problem at each vertical boundary O(nm) • O(n) 1D problems • O(n2m) total runtime O(n) 1D problems

30. Examples

32. Summary • Networked robots • High res. security cameras • Omnidirectional vs. robotic • Motion Sensing Network • Viewpoint Selection Problem • Algorithms

33. Future Work • Dynamic Version with motion prediction • Multiple outputs: • p cameras • p views from one camera • “Temporal” version: fairness • Integrate si over time: minimize accumulated dissatisfaction for any frame request • Obstacle Avoidance goldberg@ieor.berkeley.edu

35. Related Work • Facility Location Problems • Megiddo and Supowit [84] • Eppstein [97] • Halperin et al. [02] • Rectangle Fitting, Range Search, Range Sum, and Dominance Sum • Friesen and Chan [93] • Kapelio et al [95] • Mount et al [96] • Grossi and Italiano [99,00] • Agarwal and Erickson [99] • Zhang [02]

36. Related Work • Similarity Measures • Kavraki [98] • Broder et al [98, 00] • Veltkamp and Hagedoorn [00] • Frame selection algorithms • Song, Goldberg et al [02, 03, 04], • Har-peled et al. [03]

37. Problem Definition • Assumptions • Camera has fixed aspect ratio: 4 x 3 • Candidate frame c = [x, y, z] t • (x, y)  R2(continuous set) • Resolution z  Z • Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = largerframe = lower resolution 3z (x, y) 4z

38. Problem Definition Requests: ri=[xli, yti, xri, ybi, zi], i=1,…,n (xli, yti) (xri, ybi)

39. Optimization Problem User i’s satisfaction Total satisfaction

40. Problem Definition • “Satisfaction” for user i: 0  Si  1  = c  ri c = ri Si = 0 Si = 1

41. Coverage-Resolution Ratio Metrics • Measure user i’s satisfaction: Requested frame ri Area= ai Candidate frame c Area = a pi

42. Comparison with Similarity Metrics • Symmetric Difference • Intersection-Over-Union Nonlinear functions of (x,y), Does not measure resolution difference

43. Optimization Problem

44. Objective Function Properties (for fixed z) Requested Frame ri Candidate Frame c

45. Objective Function for Fixed Resolution • si(x,y) is a plateau • One top plane • Four side planes • Quadratic surfaces at corners • Critical boundaries: 4 horizontal, 4 vertical y 3z x 4z 4(zi-z)

46. Objective Function • Total satisfaction: for fixed z Frame selection problem: Find c* = arg max S(c)

47. Objective Function Properties S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines. si y (z/zi)2 x 4z 4(zi-z) 4z si 3z (z/zi)2 x 4z 4(zi-z) y 3(zi-z) 3z 3z

48. y x Plateau Vertex Definition • Intersection between boundaries • Self intersection: • Plateau intersection:

49. Plateau Vertex Optimality Condition • Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof: • Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur at x boundaries S(y) y

50. Fixed Resolution Exact Algorithm Brute force Exact Algorithm: Check all plateau vertices • (n2) plateau vertices • (n) time to evaluate S for each • (n3) total runtime