Evaluation

1. Evaluation David KauchakCS 158 – Fall 2019

2. Admin Assignment 3 • ClassifierTimer class Reading

3. So far… • Throw out outlier examples • Remove noisy features • Pick “good” features • Normalize feature values • center data • scale data (either variance or absolute) • Normalize example length • Finally, train your model!

4. What about testing? training data (labeled examples) pre-process data learn model/ classifier “better” training data

5. What about testing? pre-process data test data classify model/ classifier prediction How do we preprocess the test data?

6. Test data preprocessing • Throw out outlier examples • Remove noisy features • Pick “good” features • Normalize feature values • center data • scale data (either variance or absolute) • Normalize example length Which of these do we need to do on test data? Any issues?

7. Test data preprocessing • Throw out outlier examples • Remove irrelevant/noisy features • Pick “good” features • Normalize feature values • center data • scale data (either variance or absolute) • Normalize example length Remove/pick same features Do these Do this Whatever you do on training, you have to do the EXACT same on testing!

8. Normalizing test data For each feature (over all examples): Center: adjust the values so that the mean of that feature is 0: subtract the mean from all values Rescale/adjust feature values to avoid magnitude bias: • Variance scaling: divide each value by the stddev • Absolute scaling: divide each value by the largest value What values do we use when normalizing testing data?

9. Normalizing test data For each feature (over all examples): Center: adjust the values so that the mean of that feature is 0: subtract the mean from all values Rescale/adjust feature values to avoid magnitude bias: • Variance scaling: divide each value by the stddev • Absolute scaling: divide each value by the largest value Save these from training normalization!

10. Normalizing test data training data (labeled examples) learn model/ classifier mean, stddev, max,… pre-process data test data classify model/ classifier prediction

11. Features pre-processing summary • Throw out outlier examples • Remove noisy features • Pick “good” features • Normalize feature values • center data • scale data (either variance or absolute) • Normalize example length Many techniques for preprocessing data Which will work well will depend on the data and the classifier Try them out and evaluate how they affect performance on dev data Make sure to do exact same pre-processing on train and test

12. Supervised evaluation Data Label Training data 0 0 1 Labeled data 1 0 1 Testing data 0

13. Supervised evaluation Data Label Training data 0 0 1 Labeled data train a classifier model 1 0 1 Testing data 0

14. Supervised evaluation Data Label 1 0 Pretend like we don’t know the labels

15. Supervised evaluation Data Label 1 1 1 0 model Classify Pretend like we don’t know the labels

16. Supervised evaluation Data Label 1 1 1 0 model Classify Pretend like we don’t know the labels Compare predicted labels to actual labels

17. Comparing algorithms 1 1 Data Label model 1 model 2 Is model 2 better than model 1? 1 0 1 0

18. Idea 1 Predicted Evaluation Label 1 1 score 1 0 1 model 2 better if score 2 > score 1 Predicted Label model 2 model 1 1 1 score 2 0 0 When would we want to do this type of comparison?

19. Idea 1 Predicted Evaluation Label 1 1 score 1 0 1 compare and pick better Predicted Label model 2 model 1 1 1 score 2 0 0 Any concerns?

20. Is model 2 better? Model 1: 85% accuracy Model 2: 80% accuracy Model 1: 85.5% accuracy Model 2: 85.0% accuracy Model 1: 0% accuracy Model 2: 100% accuracy

21. Comparing scores: significance Just comparing scores on one data set isn’t enough! We don’t just want to know which system is better on this particular data, we want to know if model 1 is better than model 2 in general Put another way, we want to be confident that the difference is real and not just due to random chance

22. Idea 2 Predicted Evaluation Label 1 1 score 1 0 1 model 2 better if score 2 + c > score 1 Predicted Label model 2 model 1 1 1 score 2 0 0 Is this any better?

23. Idea 2 Predicted Evaluation Label 1 1 score 1 0 1 model 2 better if score 2 + c > score 1 Predicted Label model 2 model 1 1 1 score 2 0 0 NO! Key: we don’t know the variance of the output

24. Variance Recall that variance (or standard deviation) helped us predict how likely certain events are: How do we know how variable a model’s accuracy is?

25. Variance Recall that variance (or standard deviation) helped us predict how likely certain events are: We need multiple accuracy scores! Ideas?

26. Repeated experimentation Data Label Training data 0 0 1 Rather than just splitting once, split multiple times Labeled data 1 0 1 Testing data 0

27. Repeated experimentation Data Label Data Label Data Label 0 0 0 0 0 0 1 1 1 Training data … 1 1 1 0 0 0 1 1 1 = train = development

28. n-fold cross validation repeat for all parts/splits:train on n-1 parts evaluate on the other break into n equal-sized parts Training data … … … … split 3 split 1 split 2

29. n-fold cross validation evaluate score 1 score 2 … … … score 3 split 3 split 1 split 2 … …

30. n-fold cross validation better utilization of labeled data more robust: don’t just rely on one test/development set to evaluate the approach (or for optimizing parameters) multiplies the computational overhead by n (have to train n models instead of just one) 10 is the most common choice of n

31. Leave-one-out cross validation n-fold cross validation where n = number of examples aka “jackknifing” pros/cons? when would we use this?

32. Leave-one-out cross validation Can be very expensive if training is slow and/or if there are a large number of examples Useful in domains with limited training data: maximizes the data we can use for training Some classifiers are very amenable to this approach (e.g.?)

33. Comparing systems: sample 1 Is model 2 better than model 1?

34. Comparing systems: sample 2 Is model 2 better than model 1?

35. Comparing systems: sample 3 Is model 2 better than model 1?

36. Comparing systems What’s the difference?

37. Comparing systems Even though the averages are same, the variance is different!

38. Comparing systems: sample 4 Is model 2 better than model 1?

39. Comparing systems: sample 4 Is model 2 better than model 1?

40. Comparing systems: sample 4 Model 2 is ALWAYS better

41. Comparing systems: sample 4 How do we decide if model 2 is better than model 1?

42. Statistical tests Setup: • Assume some default hypothesis about the data that you’d like to disprove, called the null hypothesis • e.g. model 1 and model 2 are not statistically different in performance Test: • Calculate a test statistic from the data (often assuming something about the data) • Based on this statistic, with some probability we can reject the null hypothesis, that is, show that it does not hold

43. t-test Determines whether two samples come from the same underlying distribution or not ?

44. t-test Null hypothesis: model 1 and model 2 accuracies are no different, i.e. come from the same distribution Assumptions: there are a number that often aren’t completely true, but we’re often not too far off Result: probability that the difference in accuracies is due to random chance (low values are better)

45. Calculating t-test For our setup, we’ll do what’s called a “pair t-test” • The values can be thought of as pairs, where they were calculated under the same conditions • In our case, the same train/test split • Gives more power than the unpaired t-test (we have more information) For almost all experiments, we’ll do a “two-tailed” version of the t-test Can calculate by hand or in code, but why reinvent the wheel: use excel or a statistical package http://en.wikipedia.org/wiki/Student's_t-test

46. p-value The result of a statistical test is often a p-value p-value: the probability that the null hypothesis holds. Specifically, if we re-ran this experiment multiple times (say on different data) what is the probability that we would reject the null hypothesis incorrectly (i.e. the probability we’d be wrong) Common values to consider “significant”: 0.05 (95% confident), 0.01 (99% confident) and 0.001 (99.9% confident)

47. Comparing systems: sample 1 Is model 2 better than model 1? They are the same with: p = 1

48. Comparing systems: sample 2 Is model 2 better than model 1? They are the same with: p = 0.15

49. Comparing systems: sample 3 Is model 2 better than model 1? They are the same with: p = 0.007

50. Comparing systems: sample 4 Is model 2 better than model 1? They are the same with: p = 0.001