Last lecture summary

1. Last lecture summary

2. Information theory • quantifies information, information is inherently linked with uncertainty and surprise. • Consider a random variable and ask how much information is received when a specific value for this variable is observed. • The amount of information can be viewed as the ‘degree of surprise’ on learning the value of .

3. One variable • Shannon information • units: depend on the base of the log • Average information – Shannon entropy

4. Two variables and • Quantify the remaining entropy of a random variable given that the value of is known. • Conditional entropy of a random variable given that the value of other random variable is known –

5. Two variables and • Uncertainty associated with the variable is given by its entropy . • Once you measure the value of , the remaining entropy of a random variable is given by the conditional entropy . • Mutual informatoion – the reduction in uncertainty about as a consequence of the observation of

6. Decision trees branch leaf Intelligent bioinformatics The application of artificial intelligence techniques to bioinformatics problems, Keedwell

7. Supervised • Used both for • classification – classification tree • regression – regression tree • Advantages • computationally undemanding • clear, explicit reasoning, sets of rules • accurate, robust in the face of noise

8. How to split the data so that each subset in the data uniquely identifies a class in the data? • Perform different tests • i.e. split the data in subsets according to the value of different attributes • Measure the effectiveness of the tests to choose the best one. • Information based criteria are commonly used.

9. information gain • Measures the information yielded by a test x. • Reduction in uncertainty about classes as a consequence of the test x? • It is mutual information between the test x and the class. • gain criterion: select a test with maximum information gain • biased towards tests which have many subsets

10. New stuff

11. Gain ratio • Gain criterion is biased towards tests which have many subsets. • Revised gain measure taking into account the size of the subsets created by test is called a gain ratio.

12. J. Ross Quinlan, C4.5: Programs for machine learning (book) “In my experience, the gain ratio criterion is robust and typically gives a consistently better choice of test than the gain criterion”. • However, Mingers J.1 finds that though gain ratio leads to smaller trees (which is good), it has tendency to favor unbalanced splits in which one subset is much smaller than the others. 1 Mingers J., ”An empirical comparison of selection measures for decision-tree induction.”, Machine Learning 3(4), 319-342, 1989

13. Continuous data • How to split on real, continuous data? • Use threshold and comparison operators , , , (e.g. “if then Play” for Light variable being between 1 and 10). • If continuous variable in the data set has values, there are possible tests. • Algorithm evaluates each of these splits, and it is actually not expensive.

14. Regression tree Regression tree for predicting price of 1993-model cars. All features have been standardized to have zero mean and unit variance. The R2 of the tree is 0.85, which is significantly higher than that of a multiple linear regression fit to the same data (R2 = 0.8)

15. Pruning • Decision tree overfits, i.e. it learns to reproduce training data exactly. • Strategy to prevent overfitting – pruning: • Build the whole tree. • Prune the tree back, so that complex branches are consolidated into smaller (less accurate on the training data) sub-branches. • Pruning method uses some estimate of the expected error.

16. Support Vector Machine(SVM)

17. supervised binary classifier (SVM) • also works for regression (SVMR) • two main ingrediences: • maximum margin • kernel functions

18. Linear classification methods • Decision boundaries are linear. • Two class problem • The decision boundary between the two classes is a hyperplane (line, plane) in the feature vector space.

19. Linear classifiers denotes +1 denotes -1 x2 How would you classify this data? x1

20. Linear classifiers denotes +1 denotes -1 Any of these would be fine.. ..but which is best?

21. Linear classifiers denotes +1 denotes -1 How would you classify this data? Misclassified to +1 class

22. Linear classifiers denotes +1 denotes -1 Define the margin of a linear classifier as the width that the boundary could be increased by before hitting a datapoint.

23. Linear classifiers denotes +1 denotes -1 Support Vectors are the datapoints that the margin pushes up against The maximum margin linear classifier is the linear classifier with the, um, maximum margin. This is the simplest kind of SVM (called an LSVM) Linear SVM

24. Why maximum margin? • Intuitively this feels safest. • Small error in the location of boundary – least chance of misclassification. • LOOCV is easy, the model is immune to removal of any non-support-vector data point. • Only support vectors are important ! • Also theoretically well justified (statistical learning theory). • Empirically it works very, very well.

25. How to find a margin? • Margin width, can be shown to be . • We want to find maximum margin, i.e. we want to maximize . • This is equivalent to minimizing . • However not every line with high margin is the solution. • The line has to have maximum margin, but it also mustclassify the data.

26. Source: wikipedia

27. Quadratic constrained optimization • This leads to the following quadratic constrained optimization problem: • Constrained quadratic optimization is a standard problem in mathematical optimization. • A convenient way how to solve this problem is based on the so-called Lagrange multipliers .

28. Constrained quadratic optimization using Lagrange multipliers leads to the following expansion of the weight vector in terms of the input examples : ( is the output variable, i.e. +1 or -1) • Only points on the margin (i.e. support vectors xi) have αi > 0. does not have to be explicitly formed dot product

29. Training SVM: find the sets of the parameters and . • Classification with SVM: • To classify a new pattern , it is only necessary to calculate the dot product between and every support vector. • If the number of support vectors is small, computation time is significantly reduced.

30. Soft margin • The above described margin is usually refered to as hard margin. • What if the data are not 100% linearly separable? • We allow error ξi in the classification.

31. Soft margin CSE 802. Prepared by Martin Law

32. Soft margin • And we introduced capacity parameter C - trade-off between error and margin. • C is adjusted by the user • large C – a high penalty to classification errors, the number of misclassified patterns is minimized (i.e. hard margin). • Decrease in C: points move inside margin. • Data dependent, good value to start with is 100

33. Kernel Functions

34. Linear classifiers have advantages, one of them being that they often have simple training algorithms that scale linearly with the number of examples. • What to do if the classification boundary is non-linear? • Can we propose an approach generating non-linear classification boundary just by extending the linear classifier machinery? • Of course we can. Otherwise I wouldn’t ask.

35. transform

37. Input space is one dimensional with the dimension . • Feature space is two dimensional with dimensions (coordinates) . • Feature space is generated from input space by the feature function X2

38. Nomenclature • Input objects are contained in the input space . • The task of classification is to find a function that for each assigns a value from the output space . • In binary classification the output space has only two elements:

39. Nomenclature contd. • A function that maps each object to a real value is called a feature. • Combining features results in a feature mapping and the space is called feature space.

40. The way of making a non-linear classifier out of a linear classifier is to map our data from the input space to a feature space using a non-linear mapping • Then the discriminant function in the space is given as

41. So the feature mapping maps a point from 1D input space (its position is given by the coordinate x) into 2D feature space . • In this space the coordinates of the point are . • In feature space the problem is linearly separable. • It means, that this discriminant function can be found:

42. Example • Consider the case of 2D input space with the following mapping into 3D space: • In this case, what is ? features

43. The approach of explicitly computing non-linear features does not scale well with the number of input features. • For the above example the dimensionality of the feature space is roughly quadratic in the dimensionality of the original space . • This results in a quadratic increase in memory and in time to train the classifier. • However, the step of explicitly mapping the data points from the low dimensional input space to high dimensional feature space can be avoided.

44. We know that the discriminant function is given by • In the feature space it becomes • And now we use the so called kernel trick. We define kernel function

45. Example • Calculate the kernel for this mapping. • So to form the dot product we do not need to explicitly map the points and into high dimensional feature space. • This dot product is formed directly from the coordinates in the input space as.

46. Kernels • Linear (dot) kernel • This is linear classifier, use it as a test of non-linearity. • Or as a reference for the classification improvement with non-linear kernels. • Polynomial • simple, efficient for non-linear relationships • d – degree, high d leads to overfitting

47. O. Ivanciuc, Applications of SVM in Chemistry, In: Reviews in Comp. Chem. Vol 23 Polynomial kernel d = 2 d = 5 d = 10 d = 3

48. Gaussian RBF Kernel σ = 1 σ = 10 O. Ivanciuc, Applications of SVM in Chemistry, In: Reviews in Comp. Chem. Vol 23

49. Kernel functions exist also for inputs that are not vectors: • sequential data (characters from the given alphabet) • data in the form of graphs • It is possible to prove that for any given data set there exists a kernel function imposing linear separability ! • So why not always project data to higher dimension (avoiding soft margin)? • Because of the curse of dimensionality.

50. SVM parameters • Training sets the parameters and . • SVM has another set of parameters called hyperparameters. • The soft margin constant C. • Any parameters the kernel function depends on • linear kernel – no hyperparameter (except for C) • polynomial – degree • Gaussian – width of Gaussian