Curse of Dimensionality

1. Curse of Dimensionality Prof. NavneetGoyal Dept. Of Computer Science & Information Systems BITS - Pilani

2. Curse of Dimensionality!! • Poses serious challenges ! • Important factor influencing the design on pattern recognition techniques • Mixture of oil, water & gas(homogeneous , annular & laminar) • Each data point is a point in a 12-dimensional space. • 100 points along only two dimensions, x6 & x7 • x – predict its class? Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer

3. Curse of Dimensionality!! • Unlikely that it belongs to the blue class! • Surrounded by lot of red points • Also, many green points nearby • Intuition: identity of the x should be determined strongly by nearby points and less strongly by more distant points • How can we turn this intuition into a learning algorithm? Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer

4. Curse of Dimensionality!! • Make grid lines! • Use majority voting • Problems?? Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer

5. Curse of Dimensionality • No. of cells grow exponentially with D • Need exponentially large no. of training data points • Not a good approach for more than a few dimensions! Reference: Christopher M Bishop: Pattern Recognition & Machine Leaning, 2006 Springer

6. Curse of Dimensionality • Solutions?? • Dimensionality Reductions • Develop Algorithms that are not affected by Curse of Dimensionality

7. Curse of Dimensionality Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler • Problems: • running time • over-fitting • number of samples required

8. Running Time Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler • Complexity (running time) increases with dimension d! • A lot of methods have at least O(nd2) complexity(n=no. of samples) • For eg.: estimation of covariance matrix • With large d, O(nd2) complexity may be too costly

9. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler Suppose we want to use the nearest neighbor approach with k = 1 (1NN) Suppose we start with only one feature This feature is not discriminative, i.e. it does not separate the classes well Use 2 features 1NN method needs a lot of samples, i.e. Samples have to be dense To maintain the same density as in 1D (9 samples per unit length), how many samples do we need?

10. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler We need 92 samples to maintain the same density as in 1D

11. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler When we go from 1 feature to 2, no one gives us more samples, we still have 9 This is way too sparse for 1NN to work well

12. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler Things go from bad to worse if we decide to use 3 features If 9 was dense enough in 1D, in 3D we need 93=729 samples!

13. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler In general, if n samples is dense enough in 1D Then in d dimensions we need nd samples! ndgrows really really fast as a function of d Common pitfall: If we can’t solve a problem with a few features, adding more features seems like a good idea However the number of samples usually stays the same The method with more features is likely to perform worse instead of expected better

14. Number of Samples Reference: CS434a/541a: Pattern Recognition - Prof. Olga Veksler For a fixed number of samples, as we add features, the graph of classification error: Thus for each fixed sample size n, there is the optimal number of features to use