Evaluating Classifiers

1. Evaluating ClassifiersReading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

2. Evaluating Classifiers • What we want: classifier that best predicts unseen data • Assumption: Data is “iid” (independently and identically distributed)

3. Accuracy and Error Rate • Accuracy = fraction of correct classifications on unseen data (test set) • Error rate = 1 − Accuracy

4. How to use available data to best measure accuracy? Split data into training and test sets. But how to split? Too little training data: Don’t get optimal classifier Too little test data: Measured accuracy is not correct

5. One solution: “k-fold cross validation” • Each example is used both as a training instance and as a test instance. • Split data into k disjoint parts: S1, S2, ..., Sk. • For i = 1 to k Select Sito be the test set. Train on the remaining data, test on Si, to obtain accuracy Ai . • Report as the final accuracy.

6. Evaluating classification algorithms “Confusion matrix” for a given class c Actual Predicted (or “classified”) Positive Negative (in class c) (not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c) FalsePositiveTrueNegative

7. Example: “A” vs. “B” Confusion Matrix Assume “A” is positive class ActualPredicted Positive Negative Positive Negative Results from Perceptron: Instance Class Perception Output 1 “A” −1 2 “A” +1 3 “A” +1 4 “A” −1 5“B”+1 6“B” −1 7“B” −1 1 “B” −1 Accuracy:

8. Evaluating classification algorithms “Confusion matrix” for a given class c Actual Predicted (or “classified”) Positive Negative (in class c) (not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c) FalsePositiveTrueNegative Type 2 error Type 1 error

9. Precision and Recall All instances in test set Positive Instances Negative Instances

10. Precision and Recall All instances in test set Positive Instances Instances classified as positive Negative Instances

11. Precision and Recall All instances in test set Positive Instances Instances classified as positive Instances classified as negative Negative Instances Recall = Sensitivity = True Positive Rate

12. Example: “A” vs. “B” Assume “A” is positive class Results from Perceptron: Instance Class Perception Output 1 “A” −1 2 “A” +1 3 “A” +1 4 “A” −1 5“B”+1 6“B” −1 7“B” −1 1 “B” −1

13. Creating a Precision vs. Recall Curve Results of classifier

14. (“sensitivity”) (1  “specificity”)

16. Creating a ROC Curve Results of classifier

17. Precision/Recall versus ROC curves http://blog.crowdflower.com/2008/06/aggregate-turker-judgments-threshold-calibration/

20. There are various algorithms for computing AOC (often built into statistical software) – we won’t cover this in this class.

21. How to create a ROC curve for a perceptron • Run your perceptron on each instance in the test data, without doing the sgn step: • Get the range [min, max] of your scores • Divide the range into about 200 thresholds, including −∞ and max • For each threshold, calculate TPR and FPR • Plot TPR (y-axis) vs. FPR (x-axis)