Projective Kalman Filter: Multiocular Tracking of 3D Locations Towards Scene Understanding

1. 1 Technical University of Catalonia, Barcelona, Spain 2 Koç University, Istanbul, Turkey MLMI 2005, Edinburgh Projective Kalman Filter: Multiocular Tracking of 3D Locations Towards Scene Understanding C. Canton1, J.R. Casas1, A.M.Tekalp2, M.Pardàs1

2. Outline • Introduction • Problem statement & Objective • Projective Kalman Filter (PKF) • Data scenario and formulation • Data association problem on P3→P2 • Results & Performance • Conclusions & Future Research • Questions

3. Introduction • Tracking 3D locations within the SmartRoom scenario towards scene understanding can provide useful information (tracking of persons, head,…)

4. Kalman tracking 3D location estimation • Drawbacks: • Two disjoint problems • Data from N cameras is regarded as one single observation • Occlusion is handled in the estimation process but not in the tracking Problem statement • Standard approaches to track 3D locations from its 2D projections on N calibrated cameras involve: 2D feature selection over the N images Correspondence search among views Initialization

5. Joint 3D location estimation and tracking • Improvements: • Unified framework • Projective nature of N observations is taken into account • Joint 3D/2D occlusion detection scheme Objective • Define a filtering scheme to track a 3D location from its N projections 2D feature selection over the N images Correspondence search among views Initialization

6. Example

7. Projection is non-linear when seen as a morphism :R3→N2 Occlusions make this hypothesis unfeasible Kalman Filter (KF) Model • When estimating a state sR3 of a discrete time process governed by the linear stochastic difference equation with a measuremement zR2xN that is Kalman filter provides the optimal solution under the conditions: • Relations between hidden and observed data are linear • w[t] and v[t] have normal distribution

8. Projective Kalman Filter (I) • Motivation: • Track a 3D location in Euclidean coordinates taking advantage of projective geometry • Model non-linearity between the hidden state s[t] and the observed data z[t] tacking into consideration the projective nature of the observations • Handle non-Gaussian impulsive noise: detect occlusion and disregard occluded data Kalman theory can be applied to track 3D locations (with a Newtonian dynamic model) taking its projections as input data.

9. An adaptive design of H[t] based on a compensation of the non-linearity from the prediction of the estimated state resolves the conflict (z=1). During Kalman filter evolution, when applying H to the state vector s[t] coordinates might not be in the image plane (z1). Projective Kalman Filter (II)Modelling non-linearity • Tackling projection non-linearity through observation matrix H:

10. Projective Kalman Filter (III)Noise model • Observation noise covariance matrix R[t] controls how reliable is an observation. An adaptive approach to handle Gaussian noise and occlusions would be: where: Criterium to set the parameter βk from the PKF scheme: DATA ASSOCIATION & OCCLUSION DETECTION

11. Data association on P3→P2 (I) • Twofold objective: • Determine the spatial correspondence of two projections generated by the same 3D feature at two consecutive time instants in the same image • Detect an occlusion in a given view and modify R[t+1] accordingly

12. Data association on P3→P2 (II) State Estimation Extrapolation Data Bounding Projection & Data Association Occlusion Detection

13. Results • Two types of data: • Synthetic: Exact algorithm evaluation and performance purposes • Real: Practial usage of this technique within a SmartRoom scenario to track the head of present people • Data specifications: • 4 Calibrated cameras • 768x576 pixels, 25 fps

14. Results on Synthetic Data (I) • First scenario: Gaussian noise • PKF outperforms KF by ~35%. Interest Region

15. Results on Synthetic Data (II) • Second scenario: Gaussian and impulsive noise (occlusions) • PKF outperforms KF when occlusions are present • Influence of occlusions is reduced by the data association process Interest Region

16. Results on Real Data (I) • Applied to track 2 people inside the SmartRoom at UPC towards scene understanding applications • Input 2D data is the top of non-overlapped foreground regions • When the 2 people are close, KF loses track but PKF keep it properly

17. Results on Real Data (II)

18. Conclusions & Future Work • Conclusions: • New scheme to track 3D locations from multiple views embeding Kalman theory and projective geometry • Model multiple projections of a 3D location into a tracking loop • Occlusion detection combining 2D/3D data • Comparable computational complexity between PKF and KF • Real-time performance • Future Work: • Comparison with Particle Filtering tracking schemes • Apply this technique to body tracking into a SmartRoom

19. The End Thank you!!!!