Outline

1. Outline • Nonparametric Techniques • Density Estimation • Histogram Approach • Parzen-window method • Kn-Nearest-Neighbor Estimation • Component Analysis and Discriminants • Principal Components Analysis • Fisher Linear Discriminant • MDA 236875Visual Recognition

2. NONPARAMETRIC TECHNIQUES • So far, we treated supervised learning under the assumption that the forms of the underlying density functions were known. • The common parametric forms rarely fit the densities actually encountered in practice. • Classical parametric densities are unimodal, whereas many practical problems involve multimodal densities. • We examine nonparametric procedures that can be used with arbitrary distribution and without the assumption that the forms of the underlying densities are known. 236875Visual Recognition

3. NONPARAMETRIC TECHNIQUES • There are several types of nonparametric methods: • Procedures for estimating the density functions from sample patterns. If these estimates are satisfactory, they can be substituted for the true densities when designing the classifier. • Procedures for directly estimating the a posteriori probabilities • Nearest neighbor rule which bypass probability estimation, and go directly to decision functions. 236875Visual Recognition

4. Histogram Approach • The conceptually simplest method of estimating a p.d.f. is histogram. The range of each component xs of vector x is divided into a fixed number m of equal intervals. The resulting boxes (bins) of identical volume V are then expected and the number of points falling into each bin is counted. • Suppose that we have nisamples xj , j=1,…,nifrom class wi • Suppose that the number of vector points in the j-th bin, bj , be kj. The histogram estimate , of density function 236875Visual Recognition

5. Histogram Approach is defined as is constant over every bin bj . • Let us verify that is a density function: • We can choose a number m of bins and their starting points. Fixation of starting points is not critical, but m is important. It place a role of smoothing parameter. Too big m makes histogram spiky, for too little m we loose a true form of the density function 236875Visual Recognition

6. The Histogram Method:Example • Assume (one dimensional) data • Some points were sampled from a combination of two Gaussians: • 3 bins 236875Visual Recognition

7. The Histogram Method:Example • 7 bins • 11 bins 236875Visual Recognition

8. Histogram Approach • The histogram p.d.f. estimator is very effective. We can do it online:all we should do is to update the counters kj during the run time, so we do not need to keep all the data which could be huge. • But its usefulness is limited only to low dimensional vectors x, because the number of bins, Nb , grows exponentially with dimensionality d : • This is the so called “curse of dimensionality” 236875Visual Recognition

9. DENSITY ESTIMATION • To estimate the density at x, we form a sequence of regions R1, R2, .. • The probability for x to fall into R is • Suppose we have n i. i.d. samples x1 , … , xn drawn according to p(x) . The probability that k of them fall in R is and the expected value for k is and variance . The relative part of samples which fall into R is (k/n) is also a random variable for which When n is growing up the variance is making smaller and is becoming to be better estimator for P. 236875Visual Recognition

10. DENSITY ESTIMATION Pksharply peaks about the mean, so the k/n is a good estimate of P. • For small enough R where x is within R and V is a volume enclosed by R. Thus 236875Visual Recognition

11. Three Conditions for DENSITY ESTIMATION • Let us take a growing sequence of samples n=1,2,3... • We take regions Rn with reduced volumes V1 > V2 > V3 >... • Let kn be the number of samples falling in Rn • Let pn(x) be the nth estimate for p(x) • If pn(x) is to converge to p(x) , 3 conditions must be required: • resolution as big as possible (to reduce smoothing) • otherwise in the range Rn there will not be infinite number of points and k/n will not converge to P and we’ll get p(x)=0 • to guarantee convergence of (*). 236875Visual Recognition

12. Parzen Window and KNN •  How to obtain the sequence R1 , R2 , ..? • There are 2 common approaches of obtaining sequences of regions that satisfy the above conditions: • Shrink an initial region by specifying the volume Vnas some function of n , such as and show that kn and kn/n behave properly i.e. pn(x) converges to p(x). This is Parzen-window (or kernel ) method . • Specify kn as some function of n, such as Here the volume Vn is grown until it encloses kn neighbors of x .   This iskn-nearest-neighbor method . • Both of these methods do converge, although it is difficult to make meaningful statements about their finite-sample behavior. 236875Visual Recognition

13. PARZEN WINDOWS •  Assume that the region Rn is a d-dimensional hypercube. • If hn is the length of an edge of that hypercube, then its volume is given by • Define the following window function: • defines a unit hypercube centered at the origin. •   , if xifalls within the hypercube of volume Vn centered at x, and is zero otherwise. • The number of samples in this hypercube is given by: 236875Visual Recognition

14. PARZEN WINDOWS cont. •  Since • Rather than limiting ourselves to the hypercube window, we can use a more general class of window functions.Thus pn(x) is an average of functions of x and the samples xi. • The window function is being used for interpolation.Each sample contributing to the estimate in accordance with its distance fromx. • pn(x) must: • be nonnegative • integrate to 1. 236875Visual Recognition

15. PARZEN WINDOWS cont. •  This can be assured by requiring the window function itself be a density function, i.e., • Effect of the window size hn on p(x) • Define the function • then, we write pn(x) as the average • Since , hn affects both the amplitude and the width of 236875Visual Recognition

16. PARZEN WINDOWS cont. Examples of two-dimensional circularly symmetric normal Parzen windows for 3 different values of h.  • If hnis very large, the amplitude of is small, and x must be far from xi before changes much from 236875Visual Recognition

17. PARZEN WINDOWS cont. • In this case pn(x) is the superposition of n broad, slowly varying functions, and is very smooth "out-of-focus" estimate for p(x). • If hnis very small, the peak value of is large, and occurs near x = xi . • In this case, pn(x) is the superposition of n sharp pulses centered at the samples: an erratic, "noisy" estimate. • As hn approaches zero, approaches a Dirac delta function centered at xi, and pn(x) approaches a superposition of delta functions centered at the samples. 236875Visual Recognition

18. PARZEN WINDOWS cont. • 3 Parzen-window density estimates based on the same set of 5 samples, • using windows from previous figure •  The choice of hn (or Vn) has an important effect on pn(x) •  If Vn is too large the estimate will suffer from too little resolution •  If Vn is too small the estimate will suffer from too much statistical variability. •  If there is limited number of samples, then seek some acceptable compromise. 236875Visual Recognition

19. PARZEN WINDOWS cont. • If we have unlimited number of samples, then let Vn slowly approach zero as n increases, and have pn(x) converge to the unknown density p(x). Examples • Example 1: p(x) is a zero-mean, unit variance, univariate normal density. Let the widow function be of the same form: •  Let where is a parameter • pn(x) is an average of normal densities centered at the samples: 236875Visual Recognition

20. Examples cont. • Generate a set of normally distributed random samples. • Vary n and h1. 236875Visual Recognition

21. Examples cont. • The results depend both on n and h1. • For n =1, pn(x) is merely a single Gaussian centered about the first sample, which has neither the mean nor the variance of the true distribution. • For n =10 and h1 =0.1, the contributions of the individual samples are discernible. This is not the case for h1 =1 and h1 =5.  • Example 2: • Let be the same as in Example 1. but let the unknown density be a mixture of uniform and a triangle density. • The case n =1 tells more about the window function than it tells about the unknown density. • For n=16, none of the estimates is good. •  For n =256, and h1=1, the estimates are beginning to appear acceptable. 236875Visual Recognition

22. Examples cont. Parzen window estimates of a bimodal distribution 236875Visual Recognition

23. Examples cont. • Classification • To make a classification we should: • Estimate the density for each category using Parzen-window method. • Classify a test point by the label corresponding to the maximum posterior. • The decision regions for a Parzen-window classifier depend upon the choice of window function. 236875Visual Recognition

24. Examples cont. Small h: more complicated boundaries. Large h: Less complicated boundaries. 236875Visual Recognition

25. Examples cont. • For this data, a small h would be appropriate for the upper points, while a large h for the lower region. • No single window width is ideal overall. • In general, the training error can be made arbitrarily low by making the window width sufficiently small. • Remember, the goal of creating a classifier is to classify novel patterns, and a low training error does not guarantee a small test error. • Advantages: • Generality: same procedure can be used for unimodal normal and bimodal mixture. •  We do not need to make assumption about the distribution ahead of time. • With enough samples, we are assured of convergence to an arbitrarily complicated target density 236875Visual Recognition

26. Examples cont. • Disadvantages: • Number of samples needed may be very large (much larger than would be required if we knew the form of the unknown density) . •  Severe requirements for computation time and storage. •  The large number of samples grows exponentially with the dimensionality of the feature space ("curse of dimensionality") •  Sensitivity to the choice of the window size: • Too small: most of the volume will be empty, and the estimate pn(x)will be very erratic. • Too large: important variations may be lost due to averaging. • It may be the case that a cell volume appropriate for one region of the feature space might be entirely unsuitable in a different region. 236875Visual Recognition

27. Kn-Nearest-Neighbor Estimation • To estimate p(x) from n training samples, we center a cell about x and let it grow until it captures kn samples. where kn is some specified function of n. • These samples are the kn nearest-neighbors of x. • If the density is high near x, the cell will be relatively small good resolution. 236875Visual Recognition

28. Kn-Nearest-Neighbor Estimation • If we take we want when for be a good estimate of probability that a point will fall into the cell. But kn should grow sufficiently slow that the size of the cell captured kn samples will shrink to zero. Thus is necessary and sufficient for pn(x) to converge to p(x). If and assume that pn(x) is good approximation for p(x) than . Thus but with determined by the nature of the data 236875Visual Recognition

29. Comparison of Density Estimators • Parzen window estimates require the storage of all the observations and n evaluations of the kernel function for each estimate, which is computationally expensive! • Nearest neighbor also require the storage of all the observations. • Histogram estimates do not require storage for all the observations, they require storage for description of the bins. But for simple histogram the number of the bins grows exponentially with dimension of the observation space. 236875Visual Recognition

30. Classification with K-NN and Parzen window: Estimation of a posteriori probabilities • The k-nn (and Parzen window) techniques can be used to estimate the a posteriori probabilities from a set of n labeled samples. •  Suppose that we place a cell of volume V around x and capture k samples: • ki are labeled wi •  k-ki have other labels. •  A simple estimate for the joint probability is • A reasonable estimate for is 236875Visual Recognition

31. Estimation of a posteriori probabilities cont. • The estimate of the a posteriori probability that is the state of nature is merely the fraction of the samples within the cell that are labeled . •  For minimum error rate, we select the category most frequently represented within the cell. • In this classifier at the learning stage there is nothing be done (only to get the sample points), and at the classification stage simply calculate frequencies of different classes. This classifier is called Instant Based Learning . 236875Visual Recognition

32. Classification With Nearest Neighbor Rule • Let denote a set of n labeled prototypes and let be the prototype nearest to a test point x . Then The Nearest Neighbor Rule: assign the label of • This rule is suboptimal, but when the number of prototypes is large its error is never worse then twice the Bayes rate. This rule partition the feature space into cells consisting of all points closer to a given training point x’ than to any other training points. All points in such a cell are thus labeled by the category of the training point – a so called Voronoi tessellation of the training space. 236875Visual Recognition

33. Classification With Nearest Neighbor Rule Voronoi cells 236875Visual Recognition

34. Classification With the k-Nearest Neighbor Rule The k-nn query starts at the test point and grows a spherical region until it encloses k training samples, and it labels by a majority vote of these samples. • Algorithm: • For each sample point • Compute Distance (sample point, test • point) • Sort the distances • Inspect the k smallest distances • Label test point by a majority vote. 236875Visual Recognition

35. Classification With the k-Nearest Neighbor Rule Question. If a posteriori probabilities P(wi|x) , i=1,2 for two classes are known, for example what is a probability of choosing a class w1 with the Bayes, the nearest neighbor, the k-nn classifiers? 236875Visual Recognition

36. Classification With the k-Nearest Neighbor Rule. • Answer • Bayes: always w1 • NN: P(w1|x) • K-nn: 236875Visual Recognition

37. Component Analysis and Discriminants • How to reduce excessive dimensionality? Answer: Combine features. • Linear methods project high-dimensional data onto lower dimensional space. • Principal Components Analysis (PCA) - seeks the projection which best represents the data in a least-square sense. • Fisher Linear Discriminant - seeks the projection that best separates the data in a a least-square sense. 236875Visual Recognition

38. Principal Components Analysis (PCA) • How to represent nd-dimensional vector samples {x1,.., xn} by a single vector x0 ? Find x0 that minimize squared error correction function • The solution is sample mean • This is zero-dimensional representation of the data set. • One-dimensional representation by projecting the data onto a line running through the sample mean reveals variability in the data. 236875Visual Recognition

39. PCA cont. • Let e be a unit vector in a direction of the line. The equation of the line • Representing xk by find “optimal” set minimizing criterion function : from we find The least square solution: project the vector xkonto the line in the direction of e, passing through the sample mean. 236875Visual Recognition

40. Scatter matrix • How to find the best direction for e ? • Define a scatter matrix S. It is (n-1) times sample covariance matrix • Substituting (3) into (2) we find 236875Visual Recognition

41. Scatter matrix • Vector e that minimizes J1 also maximizes . So we find e,which maximize subject to constraint || e ||=1. Let be Lagrange multiplier. Differentiating with respect to e: By setting the result to zero we see that is e an eigenvector of : So to maximize take maximal 236875Visual Recognition

42. Scatter matrix • The result is easily extended to d’ dimensional projection: • The criterion function • is minimized when vectors are the eigenvectors having the largest eigenvalues. • The coefficients are principal components. • Transformation to the resulting set of axes is called a Karhunen-Loeve or Hotelling transform 236875Visual Recognition

43. Error function • If: error which is made by dropping the last terms is • This is a sum of lowest eigenvalues. 236875Visual Recognition

44. PCA – the algorithm • Input: • Take • Output: • Algorithm: • Compute the mean of the training set • Compute the scatter matrix S. • Find eigenvectors of S and corresponding eigenvalues: • Choose eigenvectors, and for each sample point compute 236875Visual Recognition

45. PCA – for clustering • Example: • Note that center of the new coordinate system was shifted to the sample mean. • Now, if we need to cluster the data, we’d better do it along the direction(s) with the highest variance – that is, along the principal component(s). • Above we clearly see 2 clusters along e1, while e2 doesn’t provide much information about the structure of the data. 236875Visual Recognition

46. PCA limitations • Suppose we have a 5 – dimensional data, and after applying PCA we found that the eigenvalues of the covariance matrix are 300,278, 260,12,10. • Clearly, we can represent the most information about the data by 3 first eigenvectors; we say that the effective dimensionality of the data is 3. • But what about this example: • Here the effective dimension is 1, but PCA detects 2 “equal” dimensions. • The reason – PCA is a linear method, unable to detect nonlinear structure. 236875Visual Recognition

47. PCA limitations – cont. • On the left, 2-D data has 3 clusters. Also from the 2 eigenvectors produced by PCA l1 is the largest, l2 isn’t far behind, so we did not reduced dimensionality. Besides, these are not the directions along which clusters are easily detected. • On the right, 3-D data has 3 clusters (each one “lives” in a different subspace). Here, too, the eigenvalues, found by PCA help neither to reduce dimensionality nor to detect the clusters. 236875Visual Recognition

48. PCA limitations – cont. • Another problem with PCA: The scale of all the dimensions must be the same (if values of the first component are in 0-1000000 while those of the second are in 50-60, PCA will almost completely disregard the second dimension, even so that it may be more important 236875Visual Recognition

49. Fisher Linear Discriminant • We have nd-dimensional samples x1,.., xn, n1 in a subset , labeled w1 and n2 in a subset , labeled w2 . • Find direction of line w , that maximally separate the data. • Let a difference of sample means be a measure of separation of projected points 236875Visual Recognition

50. Fisher Linear Discriminant cont. • Project samples onto w. n samples are divided into the subsetsand • Let be the sample mean • The sample mean for projected points • Distance between the projected means is 236875Visual Recognition