Non-Parametric Learning

1. Non-Parametric Learning Prof. A.L. Yuille Stat 231. Fall 2004. Chp 4.1 – 4.3.

2. Parametric versus Non-Parametric • Previous lectures on MLE learning assumed a functional form for the probability distribution. • We now consider an alternative non-parametric method based on window function.

3. Non-Parametric • It is hard to develop probability models for some data. • Example: estimate the distribution of annual rainfall in the U.S.A. Want to model p(x,y) – probability that a raindrop hits a position (x,y). • Problems: (i) multi-modal density is difficult for parametric models, (ii) difficult/impossible to collect enough data at each point (x,y).

4. Intuition • Assume that the probability density is locally smooth. • Goal: estimate the class density model p(x) from data • Method 1: Windows based on points x in space.

5. Windows • For each point x, form a window centred at x with volume Count the number of samples that fall in the window. • Probability density is estimated as:

6. Non-Parametric • Goal: to design a sequence of windows so that at each point x • (f(x) is the true density). • Conditions for window design: • increasing spatial resolution. (ii) many samples at each point (iii)

7. Two Design Methods • Parzen Window: Fix window size: • K-NN: Fix no. samples in window:

8. Parzen Window • Parzen window uses a window function • Example: • (i) Unit hypercube: and 0 otherwise. • (ii) Gaussian in d-dimensions.

9. Parzen Windows • No. of samples in the hypercube is • Volume • The estimate of the distribution is: • More generally, the window interpolates the data.

10. Parzen Window Example • Estimate a density with five modes using Gaussian windows at scales h=1,0.5, 0.2.

11. Convergence Proof. • We will show that the Parzen window estimator converges to the true density at each point x with increasing number of samples.

12. Proof Strategy. • Parzen distribution is a random variable which depends on the samples used to estimate it. • We have to take the expectation of the distribution with respect to the samples. • We show that the expected value of the Parzen distribution will be the true distribution. And the expected variance of the Parzen distribution will tend to 0 as no. samples gets large.

13. Convergence of the Mean • Result follows.

14. Convergence of Variance • Variance:

15. Example of Parzen Window • Underlying density is Gaussian. Window volume decreases as

16. Example of Parzen Window • Underlying Density is bi-modal.

17. Parzen Window and Interpolation. • In practice, we do not have an infinite number of samples. • The choice of window shape is important. This effectively interpolates the data. • If the window shape fits the local structure of the density, then Parzen windows are effective.