Visual Tracking

1. Visual Tracking CMPUT 615 Nilanjan Ray

2. What is Visual Tracking • Following objects through image sequences or videos • Sometimes we need to track a single object, sometimes a number of them • Sometimes we just track the object centroid, sometimes entire object boundary

3. Theoretical Foundation • Visual tracking is a “state” estimation problem • Bayesian inference is at the heart of visual tracking; it is called sequential Bayesian estimation • We form the posterior probability of the state, given all evidence or measurements up to the current time point • Inference is performed from the posterior density

4. Setting The Stage Some notations: Xt: unknown state we want to estimate at time point t; e.g., object centroid Zt: Measurement/observation made at time point t; e.g., image intensities The sequential estimation model assumes that we know the three probability densities: p(X0): The initial state density p(Xt|Xt-1): State transition density or motion model p(Zt|Xt): Measurement/observation/likelihood density

5. Sequential Bayesian Estimation • We want to recursively estimate the state Xt given the observations Z1:t = {Z1, Z2, …, Zt}

6. Sequential Bayesian Estimation… Filter: Prediction Previous posterior Likelihood/observation density Bayes’ Rule: Current posterior

7. Bayes’ Rule Derivation Conditional probability rule Marginal density rule Also, because measurement Zt is conditionally independent on the current state Xt: So, we have the sequential Bayes’ rule:

8. Filter Derivation Rule of marginal density Rule of conditional probability Also, note that Xt is conditionally independentXt-1 (Markovianity), so: Thus we have the filter rule:

9. Important Assumptions • Observation is conditionally independent on the current state • Current state is conditionally independent on the immediate previous state

10. Computation • Theory is all good, however we need to show people that it works in practice… • We will study Particle filter is the framework that can compute the recursive state estimation, i.e., sequential Bayesian filter • We will also study Kalman filter, popular sequential state estimation with some more assumptions

11. What is a Particle Filter? Let the particles represent the previous density So, the filter step is now: And the Bayes’ rule is now: We need to generate the current particle set from p(Xt|Zt): Particle filter

12. Factored Sampling Let h(x) = f(x)g(x) is a product of two functions, where say, g(x) is a density and f(x) is another non-negative function Factored sampling says that to represent h(x) non-parametrically by a set of particles, generate samples from g(x) and assign weights by f(x) i.e., {(s1, w1), …, (sn, wn)}, where si are generated from g(x) and wi = f(si). This is closely related to another sampling method called important sampling.

13. Conditional Density Propagation (CONDENSATION) This a product of two functions: (1) and (2) Following the principle of factored sampling, CONDENSATION generates samples from (1) And assigns weight using (2)

14. Samples From a Mixture Density is a mixture density Notice that To generate samples from the mixture density these two steps are followed:

15. CONDENSATION Algorithm

16. How to estimate the state? • OK, we generated samples, what do we d with them: estimate the current state: h is any function of the state, for example when h(x) = x the state estimation is done

17. Other PFs • To date lots of particle filters have been proposed: • Sequential importance re-sampling (SIR) • Auxiliary particle filter (APF) • Likelihood particle filter • Rao-Blackwellized particle filter • A ton others • Leading researchers in PF : ArnoudDoucet et al.

18. Some Points to Ponder about PF • The good point about PF is that it can handle very general likelihood and motion models • PF inherits a serious shortcoming from non-parametric density representation – curse of dimensionality – when the state space x is large, for example large multiple number of objects etc.