Line ar machines márc. 9.

1. Linear machinesmárc. 9.

2. Decison surfaces We focus now on the decision surfaces Linear machines = linear decision surface Non-optimal solution but tractable model

3. Decision surface for Bayes classifier with Normal densites(i = esete)

4. Decision tree and decision regions

5. Linear discriminant function twocategoryclassifier: choose1if g(x) > 0elsechoose2if g(x) <0 If g(x) = 0 thedecision is undefined. g(x)=0 definesthedecisionsurface Linearmachine = lineardiscriminantfunction: g(x) = wtx + w0 w weightvector w0constantbias

7. c lineardiscriminantfunction: iis predictedifgi(x) > gj(x)  j  i; i.e. pairwisedecisionsurfacesdefinesthedecisionregions More than 2 categories

9. It is proved that linear machines can only define convex regions, i.e. concave regions cannot be learnt. Moreover the decision boundaries can be higher order surfaces (like elliptoids)… Expression power of linear machines

10. Homogen coordinates

11. Training linear machines 10

12. Lineáris gépek tanulása Searching for the values of w which separates classes Usually a goodness function is utilised as objective function, e.g. 11

13. Two categories - normalisation 12 ifyibelongs to ω2replace yiby -yi then search forawhichatyi>0 (normalised version) Thereisn’t anyuniquesolution.

14. Iterative optimalisation The solutionminimalisesJ(a) Iterative improvement ofJ(a) 13 a(k+1) Stepdirection Learningrate a(k)

15. Gradient descent 14 Learning rate is a function of k, i.e. it describes a cooling strategy

16. Gradient descent 15

17. 16 Learning rate?

18. 17 Perceptron rule

19. 18 Perceptron szabály Y(a): the set of training samples misclassified by a IfY(a)is emptyJp(a)=0;else Jp(a)>0

20. Perceptron rule UsingJp(a)in the gradient descent: 19

21. 20 Misclassifiedtraining samplesbya(k) Perceptron convergence theorem: If the training dataset is linearly separable the batch perceptron algorithm finds a solution in finete steps.

22. 21 η(k)=1 online learning Stochasticgradientdesent: Estimatethegradientbasedon a fewtraingingexamples

23. Online vs offline learning • Online learningalgorithms: • The modell is updatedbyeachtraininginstance (orby a small batch) • Offline learningalgorithms: • The trainingdataset is processedas a whole • Advantages of online learning: • Update is straightforward • The trainingdatasetcan be streamed • Implicit adaptation • Disadvantagesof online learning: • - Itsaccuracymigth be lower

24. Not linearly separable case Changethelossfunction, itshouldcounteachtrainingexample e.g. thedirecteddistancefromthedecisionsurface 23

25. SVM

26. Which one to prefer? 25

27. Margin: the gap around the decision surface. It is defined by the training instances closest to the decision survey (support vectors) 26

28. 27

29. Support Vector Machine(SVM) SVM is a linear machine where the objective function incorporates the maximalisation of the margin! This provides generalisation ability 28

30. SVM Linearlyseparablecase

31. Linear SVM: linearly separable case Trainingdatabase: Searchingfor w s.t. or 30

32. Note the size of the margin by ρ Linearly separable: We prefer a unique solution: argmax ρ= argmin Linear SVM: linearlyseparablecase 31

33. Linear SVM: linearlyseparablecase 32 Convexquadraticoptimisationproblem…

34. The form of thesolution: bármely t-ből xt támasztóvektor iff Linear SVM: linearlyseparablecase 33 Weightedavearge of traininginstances onlysupportvectorscount

36. SVM notlinearlyseparablecase

37. Linear SVM: not linearly separable case ξslackvariableenablesincorrectclassifications („softmargin”): ξt=0 iftheclassification is correct, elseitisthedistancefromthemargin 36 C is a metaparameterforthetrade-offbetweenthemarginsize and incorrectclassifications

39. SVM non-linearcase

40. Generalised linear discriminant functions E.g. quadratic decision surface: Generalised linear discriminant functions: yi: Rd →Rarbitrary functions g(x) is not linear in x, but is is linear in yi (it is a hyperplane in the y-space)

41. Example

44. Non-linear SVM 43

45. Non-linear SVM 44 Φis a mapping into a higher dimensional (k) space: There exists a mapping into a higher dimensional space for any dataset where the dataset will be linearly separable in the new space.

46. The kernel trick g(x)= The calculation of mappingsintohighdimensionalspacecan be omitedifthe kernel of tox can be computed 45

47. Example: polinomial kernel 46 K(x,y)=(x y) p d=256 (original dimensions) p=4 h=183 181 376 (high dimensional space) on the other hand K(x,y) is known and feasible to calculate while the inner product in high dimensions is not

48. Kernels in practice 47 • No rule of thumbs for selecting the appropiate kernel

49. The XOR example 48

50. 49 The XOR example