Nonparametric Density Estimation

1. Nonparametric Density Estimation Riu Baring CIS 8526 Machine Learning Temple University Fall 2007 Christopher M. Bishop, Pattern Recognition and Machine Learning, Chapter 2.5 Some slides from http://courses.cs.tamu.edu/rgutier/cpsc689_f07/

2. Overview • Density Estimation • Given: a finite set x1,…,xN • Task: to model the probability distribution p(x) • Parametric Distribution • Governed by adaptive parameters • Mean and variance – Gaussian Distribution • Need procedure to determine suitable values for the parameters • Discrete rv – binomial and multinomial distributions • Continuous rv – Gaussian distributions

3. Nonparametric Method • Attempt to estimate the density directly from the data without making any parametric assumptions about the underlying distribution • . Nonparametric Density Estimation

4. Histogram • Divide the sample space into a number of bins and approximate the density at the center of each bin by the fraction of points in the training data that fall into the corresponding bin • .

5. Histogram • Parameter: bin width • .

6. Histogram - Drawbacks • The discontinuities of the estimate are not due to the underlying density, they are only an artifact of the chosen bin locations • These discontinuities make it very difficult (to the naïve analyst) to grasp the structure of the data • A much more serious problem is the curse of dimensionality, since the number of bins grows exponentially with the number of dimensions • In high dimensions we would require a very large number of examples or else most of the bins would be empty

7. Nonparametric DE

8. Nonparametric DE

9. Nonparametric DE

10. Kernel Density Estimator

11. Kernel Density Estimator

12. k-nearest-neighbors • To estimate p(x): • Consider small sphere centered on the point x • Allow the radius of the sphere to grow until it contains k data points

13. k-nearest-neighbors • Data set comprising Nk points in class Ck, so that • Suppose the sphere has volume, V, and contains kk points from class Ck • Density Estimate Unconditional density Class Prior • Posterior probability of class membership • .

14. k-nearest-neighbors • To classify new point x • Identify K nearest neighbors from training data • Assign to the class having the largest number of representatives • Parameter, K • .

15. My thoughts • KDE and KNN require the entire training data set to be stored • Leads to expensive computation • Tweak “parameters” • KDE: bandwidth, h • KNN: K