Pursuit Evasion Games and Multiple View Geometry

1. Pursuit Evasion Games and Multiple View Geometry René Vidal UC Berkeley

2. “Research Trends” • Control • Hybrid Systems, Embedded Systems • Quantum Control, Multi-Agent Control • Computer Vision • Multi-body Structure from Motion • Omnidirectional Vision • Many more … • Texture, segmentation and grouping, shape, stereo, illumination, recognition, real-time vision

3. Previous Work • Robotics and Control • Pursuit-Evasion Games (ICRA’01-CDC’01-TRA’01) • Decidable Classes of Discrete Time Hybrid Systems (HSCC’00-CDC’01) • Computer Vision • Multiple View Geometry (IJCV’01-ECCV’02) • Optimal Motion Estimation (ICCV’01) • Planar motions with Small Baselines (ECCV’02) • Camera Self-Calibration (ECCV’00-IJCV’01)

4. Pursuit-Evasion Games Cory Sharp, Omid Shakernia, David Shim, Jin Kim Shankar Sastry

5. Pursuit-Evasion Game Scenario Evade!

6. PEG: Map Building (Hespanha CDC’99) • Given measurements build a probabilistic map • Measurement step: model for sensor detection • Prediction step: model for evader motion

7. Greedy Policy Pursuer moves to the reachable cell with the highest probability of having an evader The probability of the capture time being finite is equal to one (Hespanha CDC ’99) The expected value of the capture time is finite (Hespanha CDC ’99) Global-Max Policy Pursuer moves to the reachable cell which is closest to the cell in the whole map with the highest probability of having an evader May not take advantage of multiple pursuers (all the pursuers could move to the same place) PEG: Pursuit Policies (Hespanha CDC’99)

8. strategy planner map builder • position of evader(s) • position of obstacles • position of pursuers Desired pursuers positions obstacles detected pursuers positions evaders detected tactical planner vehicle-level sensor fusion trajectory planner obstacles detected state of helicopter & height over terrain regulation control signals [4n] actuator positions [4n] • obstacles detected • evaders detected lin. accel. & ang. vel. [6n] inertial positions [3n] height over terrain [n] Exogenous disturbances vision actuator encoders INS GPS ultrasonic altimeter agent dynamics terrain evader PEG: Control Architecture (ICRA’01) communications network tactical planner & regulation

9. PEG: Vision System

10. PEG: Experimental Results (Summer’00)

11. PEG: Architecture Implementation (CDC’01) Serial Navigation Computer Vision Computer Strategic Planner Helicopter Control GPS: Position INS: Orientation Camera Control Color Tracking UGV Position Estimation Communication UAV Pursuer Map Building Pursuit Policies Communication Runs in Simulink Same for Simulation and Experiments TCP/IP Serial Robot Micro Controller Robot Computer Robot Control DeadReck: Position Compass: Heading Camera Control Color Tracking GPS: Position Communication UGV Pursuer UGV Evader

12. PEG: Experimental Results (Spring’01)

13. PEG: Experimental Results (Spring’01)

14. PEG: Experimental Results (Spring’01) • Pursuit Policy v.s. Vision System

15. Vision Based Landing

16. Landing: (Sharp & Shakernia ICRA’01)

17. PEG: Summary • Proposed a probabilistic framework and a hierarchical control architecture for pursuit-evasion games • Developed algorithms for • Sensor fusion, helicopter control, vision based detection, vision based landing • Implemented architecture with UGVs and UAVs in real-time

18. Multiple View Geometry The Multiple View Matrix Yi Ma (UIUC), Jana Kosecka (GMU), Shankar Sastry (UCB)

19. Multiple View Geometry (MVG)

20. Multiple View Geometry (MVG) • Obtain camera motion and scene structure from multiple images of a cloud of 3D feature points

21. MVG: Literature Review • Theory • Two views: Longuet-Higgins’81, Huang & Faugeras’89, … • Three views: Spetsakis et.al.’90, Shashua’94, Hartley’94, … • Four views: Triggs’95, Shashua’00, … • Multi views: Faugeras et.al.’95, Heyden et.al.’97,… • Algorithms • Euclidean: Maybank’93, Weng, Ahuja & Huang’93, … • Affine: Quan & Kanade’96, … • Projective: Triggs’96, … • Orthographic:Tomasi & Kanade’92, …

22. surface curve line point practice projective algorithm affine theory Euclidean 2 views algebra 3 views geometry 4 views optimization m views MVG: Anatomy of cases (state of the art)

23. surface curve line point practice projective algorithm affine theory Euclidean rank deficiency of Multiple View Matrix 2 views algebra 3 views geometry 4 views optimization m views MVG: A need for unification

24. MVG: Formulation • Homogeneous coordinates of a 3-D point • Homogeneous coordinates of its 2-D image • Projection of a 3-D point to an image plane • Either Euclidean or Projective

25. MVG: Classical Approach • Given corresponding images of points recover motion, structure and calibration from • This set of equations is equivalent to

26. (generic) (degenerate) MVG: The Multiple View Matrix • WLOG choose frame 1 as reference • Theorem: [Rank deficiency of Multiple View Matrix]

27. MVG: Bilinear and Trilinear Constraints • Multiple View Matrix implies bilinear constraints • Multiple View Matrix implies trilinear constraints • Constraints among more than three views are algebraically dependent (quadrilinear in particular)

28. MVG: Degenerate Cases • Theorem [Uniqueness of the pre-image]: Given vectors with respect to camera frames, they correspond to a unique point in the 3-D space if rank(M) = 1. If rank(M) = 0, the point is determined up to a lineon which all the camera centers must lie.

29. Point Features Line Features MGV: Line Features

30. MVG: Mixed Point and Line Features . . .

32. Point Features Line Features MGV: Planar Features Besides multilinear constraints, it simultaneously giveshomography:

33. Multiple View Landing (ICRA’02)

34. MGV: Multiple View Planar Algorithm Results

35. Optimal Motion Estimation (ICCV’01) Noisy measurements: Minimize error: Subject to: Lagrange optimization + algebraic elimination:

36. Optimal Motion Estimation (ICCV’01) • M is not a regular Euclidean space • Can do Euclidean optimization, but need to project onto M • There are optimization techniques for Stiefel manifolds • Quadratic convergence is guaranteed if Hessian is non-degenerate [Smith and Brockett ’93]

37. Multiple View Matrix: Summary • Points • M implies bilinear and trilinear constraints • Quadrilinear constraints do not exist • Lines • All indep. constraints are among 3 views • Some are trilinear, some are nonlinear • Mixed points and lines • There are multilinear constraints among 2-3 views • There are nonlinear constraints among 3-4 views

38. Current Work: SFM for curves • Differentiating of a point on a curve intensity level sets region boundaries . . .

39. Current Work: SFM for curves • Multiple view matrix for curves • Gives rise to rank condition • Gives rise to set of ODE’s

40. Current Work: Multi-Body SFM • Geometry of multiple moving objects • Given a set of corresponding image points, obtain: • Number of evaders • Evader to which each point belongs to (segmentation) • Depth of each point • Motion of each pursuer and evader • Orthographic case (Costeira-Kanade’95) • Multiple moving points (Shashua-Levin’01) • Can the multiple view matrix approach be used for motion estimation and segmentation?

41. Current Work: Multi-Body SFM • Generalized Multiple View Matrix [Ma et.al. ’01] • Multiple moving points • Projection

42. Current Work: Multi-Body SFM • Linked Multiple Body Motion Estimation • Infinitesimal motion case • Ursella, Soatto and Perona ‘95 • Bregler and Malik ‘98 • Discrete motion case • Generalized multiple view matrix • Paden-Kahan inverse kinematic subproblems

43. Current Work: Multi-Body SFM [Vidal’01] • Costeira-Kanade in perspective • Form optical flow matrices • Obtain number of objects • Segment the points

44. Conclusions • Computer Vision for Real-Time Control • Pursuit Evasion Games • Vision Based Landing • A new approach to Multiple View Geometry • Simple: just linear algebra • Unifying: Euclidean, projective, 2 views, 3 views, multiple views, points, lines, curves, surface? • New Constraints: There are nonlinear constraints