1 / 68

Real-Time Tracking

Real-Time Tracking. Axel Pinz Image Based Measurement Group EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University of Technology http://www.emt.tugraz.at/~tracking http://www.emt.tugraz.at/~pinz axel.pinz@tugraz.at. Defining the Terms.

micol
Download Presentation

Real-Time Tracking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Real-Time Tracking Axel Pinz Image Based Measurement Group EMT – Institute of Electrical Measurement and Measurement Signal Processing TU Graz – Graz University of Technology http://www.emt.tugraz.at/~tracking http://www.emt.tugraz.at/~pinz axel.pinz@tugraz.at

  2. Defining the Terms • Real-Time • Task dependent, “in-the-loop” • Navigation: “on-time” • Video rate: 30Hz • High-speed tracking: several kHz • Tracking • DoF: Degrees of Freedom • 2D: images, videos  2 / 3 DoF • 3D: scenes, object pose  6 DoF

  3. Example: High-speed, 2D

  4. Applications • Surveillance • Augmented reality • Surgical navigation • Motion capture (MoCap) • Autonomous navigation • Telecommunication • Many industrial applications

  5. Example: Augmented Reality [ARToolkit, Billinghurst, Kato, Demo at ISAR2000, Munich] http://www.hitl.washington.edu/research/shared_space/download/

  6. AgendaStructure of the SSIP Lecture • Intro, terminology, applications • 2D motion analysis • Geometry • 3D motion analysis • Practical considerations • Existing systems • Summary, conclusions

  7. 2D Motion Analysis • Change detection • Can be anything (not necessarily motion) • Optical flow computation • What is moving in which direction ? • Hard in real time • Data reduction required ! • Interest operators • Points, lines, regions, contours • Modeling required • Motion models, object models • Probabilistic modeling, prediction

  8. Change Detection [Pinz, Bildverstehen, 1994]

  9. Optical Flow (1) [Brox, Bruhn, Papenberg, Weickert] ECCV04 best paper award • Estimating the displacement field • Assumptions: • Gray value constancy • Gradient constancy • Smoothness • ... • Error Minimization

  10. Optical Flow (2) [Brox, Bruhn, Papenberg, Weickert] ECCV04 best paper award !! Not in real-time !!

  11. Interest Operators • Reduce the amount of data • Track only salient features • Support region – ROI (region of interest) Feature in ROI: Edge / Line Corner Blob Contour

  12. 2D Point Tracking[Univ. Erlangen, VAMPIRE, EU-IST-2001-34401] • Corner detection  Initialization • Calculate “cornerness” c • Threshold  sensitivity, # of corners • E.g.: “Harris” / “Plessey” corners in ROI • Cross-correlation in ROI

  13. 2D Point Tracking[Univ. Erlangen, VAMPIRE, EU-IST-2001-34401]

  14. Edge Tracking[Rapid 95, Harris, RoRapid 95, Armstrong, Zisserman]

  15. Blob Tracking[Mean Shift 03, Comaniciu, Meer]

  16. Contour Tracking[CONDENSATION 98-02, Isard, Toyama, Blake]

  17. CONDENSATION (2) • CONditional DENSity propagATION • Requires a good initialization • Works with active contours • Maintains / adapts a contour model • Can keep more than one hypothesis

  18. AgendaStructure of the SSIP Lecture • Intro, terminology, applications • 2D motion analysis • Geometry • 3D motion analysis • Practical considerations • Existing systems • Summary, conclusions

  19. Geometry • Having motion in images: • What does it mean? • What can be measured? • Projective camera • Algebraic projective geometry • Camera calibration • Computer Vision • Reconstruction from uncalibrated views • There are excellent textbooks [Faugeras 1994, Hartley+Zisserman 2001, Ma et al. 2003]

  20. Projective Camera (1) • Pinhole camera model: • p = (x,y)Tis the image ofP = (X,Y,Z )T • (x,y) ... image-, (X,Y,Z) ... scene-coordinates • o ... center of projection • (x,y,z) ... camera coordinate system • (x,y,-f) ... image plane x x P(X,Y,Z) f o z y y Z X p(x,y) Y

  21. Projective Camera (2) • Pinhole camera model: • If scene- = camera-coordinate system X f o Z x

  22. Projective Camera (3) • Frontal pinhole camera model: • (x,y,+f) ... image plane • Normalized camera: f=+1 P(X,Y,Z) f x x o p z y y Z X Y

  23. Projective Camera (4) • “real” camera: • 5 Intrinsic parameters (K) • Lens distortion • 6 Extrinsic parameters (M: R, t) •  … arbitrary scale

  24. Algebraic Projective Geometry [Semple&Kneebone 52] • Homogeneous coordinates • Duality points  lines • Homography H describes any transformation • E.g.: image  image transform: x’ = Hx • All transforms can be described by 3x3 matrices • Combination of transformations: Matrix product Translation Rotation

  25. Camera Calibration (1) • Recover the 11 camera parameters: • 5 Intrinsic parameters (K: fsx, fsy, fs, u0, v0) • 6 Extrinsic parameters (M: R, t) • Calibration target: • At least 6 point correspondences  • System of linear equations • Direct (initial) solution for K and M

  26. Camera Calibration (2) • Iterative optimization • K, M, lens distortion • E.g. Levenberg-Marquart • Practical solutions require more points • Many algorithms [Tsai 87, Zhang 98, Heikkilä 00] • Overdetermined systems • Robustness against outliers • E.g. RANSAC • Refer to [Hartley, Zisserman, 2001]

  27. What can be measured ... • … with a calibrated camera • Viewing directions • Angles between viewing directions • 3D reconstruction: more than 1 view required • … with uncalibrated camera(s) • Computer Vision research of the past decade • Hierarchy of geometries: Projective – oriented projective – affine – similarity – Euclidean

  28. AgendaStructure of the SSIP Lecture • Intro, terminology, applications • 2D motion analysis • Geometry • 3D motion analysis • Practical considerations • Existing systems • Summary, conclusions

  29. 3D Motion Analysis:Location and Orientation head coord. system R t scene coord. system 6 DoF pose in real-time Extrinsic parameters in real-time

  30. 3D Motion Analysis: • Tracking technologies, terminology • Camera pose (PnP) • Stereo, E, F, epipolar geometry • Model-based tracking • Confluence of 2D and 3D • Fusion • Kalman Filter

  31. Tracking Technologies (1) • Mechanical tracking • “Magnetic tracking” • Acoustic – time of flight • “Optical”  vision-based • Compass • GPS, … External effort required ! No „self-contained“ system [Allen, Bishop, Welch. Tracking: Beyond 15 minutes of thought. SIGGRAPH’01]

  32. Tracking Technologies (2)Examples [Allen, Bishop, Welch. Tracking: Beyond 15 minutes of thought. SIGGRAPH’01]

  33. Research at EMT:Hybrid Tracking – HT Combine 2 technologies: • Vision-based • Good results for slow motion • Motion blur, occlusion, wrong matches • Inertial • Good results for fast motion • Drift, noise, long term stability • Fusion of complementary sensors ! Mimicks human cognition !

  34. Vision-Based Tracking MoreTerminology • Measure position and orientation in real-time • Obtain trajectories of object(s) • Moving observer, egomotion – “inside-out” • Stationary observer – “outside-in Tracking” • Combinations of the above • Degrees of Freedom – DoF • 3 DoF (mobile robot) • 6 DoF (head tracking in AR)

  35. Inside-out Tracking • monocular • exterior parameters • 6 DoF from  4 points • wearable, fully mobile corners blobs natural landmarks

  36. Outside-in Tracking stereo-rig IR-illumination • no cables • 1 marker/device: • 3 DoF • 2 markers: 5 DoF • 3 markers: 6 DoF devices

  37. Camera Pose Estimation • Pose estimation: Estimate extrinsic parameters from known / unknown scene  find R, t • Linear algorithms [Quan, Zan, 1999] • Iterative algorithms [Lu et al., 2000] • Point-based methods • No geometry, just 3D points • Model-based methods • Object-model, e.g. CAD

  38. PnP(1)Perspective n-Point Problem • Calibrated camera K, C = (KKT)-1 • n point correspondences scene  image • Known scene coordinates of pi, and known distances dij= ||pi – pj || • Each pair (pi,pj) defines an angle  •  can be measured (2 lines of sight, calibrated camera)  constraint for the distance ||c –pi|| pi pj dij xi xj  c

  39. PnP (2) pi pj dij xi xj  c uj ui

  40. PnP (3) • P3P, 3 points: underdetermined, 4 solutions • P4P, 4 points: overdetermined, 6 equations, 4 unknowns 4 x P3P, then find a common solution • General problem: PnP, n points

  41. PnP (4) Once the xihave been solved: • project image points  scene p’i= xi K-1ui • find a common R, t for p’i pi (point-correspondences  solve a simple system of linear equations)

  42. Stereo Reconstruction z Elementary stereo geometry in “canonical configuration” 2 h … “baseline” b Pr - Pl … “disparity” d There is just column disparity Depth computation: P(x,y,z) Pz y Cl Cr x f Pl Pr h h xl=0 x=0 xr=0

  43. Stereo (2) • 2 cameras, general configuration: Epipolar geometry X Y ur ul v ll lr Cl el er Cr

  44. Stereo (3) • Uncalibrated cameras: Fundamental matrix F • Calibrated cameras: Essential matrix E • 3x3, Rank 2 • Many Algorithms • Normalized 8-point [Hartley 97] • 5-point (structure and motion) [Nister 03]

  45. Model-Based Tracking Confluence of 2D and 3D [Deutscher, Davison, Reid 01]

  46. 3D Motion Analysis: • Tracking technologies, terminology • Camera pose (PnP) • Stereo, E, F, epipolar geometry • Model-based tracking • Confluence of 2D and 3D • Fusion • Kalman Filter • General considerations • Kalman Filter •  EMT HT project

  47. General Considerations • We have: • Several sensors (vision, inertial, ...) • Algorithms to deliver pose streams for each sensor (at discrete times; rates may vary depending on sensor, processor, load, ...) • Thus, we need: • Algorithms for sensor fusion (weighting the confidence in a sensor, sensor accuracy, ...) • Pose estimation including a temporal model [Allen, Bishop, Welch. Tracking: Beyond 15 minutes of thought. SIGGRAPH’01]

  48. Sensor Fusion • Dealing with ignorance: • Imprecision, ambiguity, contradiction, ... • Mathematical models • Fuzzy sets • Dempster-Shafer evidence theory • Probability theory • Probabilistic modeling in Computer Vision • The topic of this decade ! • Examples: CONDENSATION, mean shift

  49. Kalman Filter (1) [Welch, Bishop. An Introduction to the Kalman Filter. SIGGRAPH’01] http://www.cs.unc.edu/~welch/kalman

  50. Kalman Filter (2) • Estimate a process • with a measurement xn ... State of the process zm ... Measurement p, v ... Process and measurement noise (zero mean) A ... n x n Matrix relates the previous with the current time step B ... n x l Matrix relates optional control input u to x H ... n x m Matrix relates state x to measurement z

More Related