Frame Selection Algorithms for Collaboratively Tele-Operated Robotic Cameras

1. Frame Selection Algorithms for Collaboratively Tele-Operated Robotic Cameras

2. Conventional robotic webcamera Collaboratively controlled robotic webcamera

5. One Optimal Frame Frame Selection Problem: Given n requests, find optimal frame

6. Requested Viewing Zones Optimal Satellite Frame Oct. 27, 2003

7. Satellite Imaging • 2.44 Billion Market in 2001 • Increasing 14% per year since 1999 • Major clients • Government / Military • Oil Exploration • Weather Prediction • Agriculture Ikonos, 1999

8. Related Work • Facility Location Problems • Megiddo and Supowit [84] • Eppstein [97] • Halperin et al. [02] • Rectangle Fitting, Range Search, Range Sum, and Dominance Sum • Friesen and Chan [93] • Kapelio et al [95] • Mount et al [96] • Grossi and Italiano [99,00] • Agarwal and Erickson [99] • Zhang [02]

9. Related Work • Similarity Measures • Kavraki [98] • Broder et al [98, 00] • Veltkamp and Hagedoorn [00] • CSCW, Multimedia • Baecker [92], Meyers [96] • Kuzuoka et al [00] • Gasser [00], Hayes et al [01] • Shipman [99], Kerne [03], Li [01]

10. Problem Definition • Assumptions • Camera has fixed aspect ratio: 4 x 3 • Candidate frame c = [x, y, z] t • (x, y)  R2(continuous set) • Resolution z  Z • Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = largerframe = lower resolution 3z (x, y) 4z

11. Problem Definition Requests: ri=[xli, yti, xri, ybi, zi], i=1,…,n (xli, yti) (xri, ybi)

12. Optimization Problem User i’s satisfaction Total satisfaction

13. Problem Definition • “Satisfaction” for user i: 0  Si  1  = c  ri c = ri Si = 0 Si = 1

14. Coverage-Resolution Ratio Metrics • Measure user i’s satisfaction: Requested frame ri Area= ai Candidate frame c Area = a pi

15. Comparison with Similarity Metrics • Symmetric Difference • Intersection-Over-Union Nonlinear functions of (x,y), Does not measure resolution difference

16. Optimization Problem

17. Co-Opticon Problem Versions • Fixed Resolution Exact Algorithm • Variable Resolution Exact Algorithm • Approximate Algorithm for Arbitrarily-Shaped Requested Frame • Distributed Algorithms D. Song, A.F. van der Stappen, and K. Goldberg, Exact and Distributed Algorithms for Collaborative Camera Control, the Fifth International Workshop on Algorithmic Foundations of Robotics. Nice, France, Dec 15~17, 2002.

18. Objective Function Properties (for fixed z) Requested Frame ri Candidate Frame c

19. Objective Function for Fixed Resolution • si(x,y) is a plateau • One top plane • Four side planes • Quadratic surfaces at corners • Critical boundaries: 4 horizontal, 4 vertical y 3z x 4z 4(zi-z)

20. Objective Function • Total satisfaction: for fixed z Frame selection problem: Find c* = arg max S(c)

21. Objective Function Properties S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines. si y (z/zi)2 x 4z 4(zi-z) 4z si 3z (z/zi)2 x 4z 4(zi-z) y 3(zi-z) 3z 3z

22. y x Plateau Vertex Definition • Intersection between boundaries • Self intersection: • Plateau intersection:

23. Plateau Vertex Optimality Condition • Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof: • Along vertical boundary, S(y) is a 1D piecewise linear function: extrema must occur at x boundaries S(y) y

24. Fixed Resolution Exact Algorithm Brute force Exact Algorithm: Check all plateau vertices • (n2) plateau vertices • (n) time to evaluate S for each • (n3) total runtime

25. Improved Fixed Resolution Algorithm • Sweep horizontally: solve at each vertical • Sort critical points along y axis: O(n log n) • 1D problem at each vertical boundary O(n) • O(n) 1D problems • O(n2) total runtime y S(y) x y O(n) 1D problems

26. Speed comparison Curve B: Brute force approach Curve V: using line sweeping Random inputs

27. More Improvements for Fixed Resolution • Har-Peled, Koltun, Song, and Goldberg. [03] • Exact algorithm O(n3/2log3n) • Near Linear Approximation Algorithm O(NlogN) • N = O(nE) • E = (log(1/ε)/ε)2, where ε is the approximation bound S. Har-Peled, V. Koltun, D. Song, and K. Goldberg, Efficient Algorithms for Shared Camera Control, In Proceedings of the 19th ACM Symposium on Computational Geometry, 2003.

28. Co-Opticon Problem Versions • Fixed Resolution Exact Algorithm • Variable Resolution Exact Algorithm • Approximate Algorithm for Arbitrarily-Shaped Requested Frame • Distributed Algorithms Dezhen Song, A. Frank van der Stappen, and Ken Goldberg, An Exact Algorithm Optimizing Coverage-Resolution for Automated Satellite Frame Selection, (To appear) IEEE International Conference on Robotics and Automation (ICRA) 2004

29. x Virtual Corner • A two-requested frame case • Requested frame: y

30. Virtual Corner • Virtual corner definition • Real corner: • Extended edge intersections: y x

31. y x Recall: Plateau Vertex Definition • Intersection between boundaries • Self intersection: • Plateau intersection:

32. x Virtual Corner and Plateau Vertex • Intersection between boundaries • Candidate frame: • Frame intersection: y

33. x Virtual Corner and Plateau Vertex • Intersection between boundaries • Candidate frame: • Virtual corner: y

34. Variable Resolution Exact Algorithm • Lemma: At least one optimal frame has its corner overlapped with virtual corner. • O(n2) Virtual corners • One 3D problem→ O(n2) 1D sub problems S(z) y r2 r3 r1 r4 r5 r6 O x z Candidate frame

35. Overall complexity • O(n2) 1D problems • O(n) sub 1D problems • O(n) to compute polynomial coefficient for each sub 1D problem • s(z) = g0z-1+g1+g2z +g3z2 • O(1) to compute the max s(z) for each polynomial • O(n4) in total S(z) z

36. Improved Variable Resolution Exact Algorithm • Incremental computing • Computing polynomial coefficients • O(n) for first smooth segment, • O(1) for additional • Introduce sorting cost • O(n log n) for each virtual corner • O(n3logn) total S(z) z

37. Improved Variable Resolution Exact Algorithm • Diagonal Sweeping • No need to do sorting for each virtual corner • O(n) to get new sorted sequence • Total complexity O(n3) y y r1 r1 r2 r2 x x O O (b) (a) y Order of VCs y r1 r1 r2 r2 x O x O (d) (c)

38. Speed comparison Using Incremental computing Brute force approach Using incremental computing and diagonal sweeping Random inputs

39. Co-Opticon Problem Versions • Fixed Resolution Exact Algorithm • Variable Resolution Exact Algorithm • Approximate Algorithm for Arbitrarily-Shaped Requests • Distributed Algorithms D. Song, K. Goldberg, and A. Pashkevich, ShareCam Part II: Approximate and Distributed Algorithms for a Collaboratively Controlled Robotic Webcam, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2003.

40. Arbitrarily-Shaped Requested Frame Requested frames

41. y x Approximation Algorithm Compute S(x,y) at lattice of sample points: d w, h : width and height, g: size range

42. Approximation Bound Definition c* : Optimal frame : Optimal at lattice (Algorithm output)

43. Derive Approximation Bound c* : Optimal frame : Smallest frame at lattice that enclosesc* : Optimal at lattice (Algorithm output)

44. Derive Approximation Bound c* : Optimal frame : Smallest frame at lattice that enclosesc* • fully enclose c* • What is the ratio between their objective functions if one candidate frame is enclosed by the other?

45. Approximation Bound Requested frames

46. Approximation Bound c Candidate frame Requested frames

47. Approximation Bound ca cb Candidate frames Requested frames

48. Approximation Bound ca cb Candidate frames Requested frames

49. Derive Approximation Bound c* : Optimal frame : Smallest frame at lattice that enclosesc* What is the resolution ratio between a candidate frame and the smallest frame on the lattice that encloses it?

50. Approximation Algorithm dz: Lattice spacing in z axis ca d cb d