COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk

1. Classification and evaluation COMP 4332 Tutorial 5 March 11 CHEN Zhao zchenah@ust.hk

2. Outline • Classification process • Evaluate models • Automating experiments

3. Classification process Classifier parameters Training Phase Training data Classifier Training Model Testing Phase Predicted scores/labels Testing data Model

4. What determines a model? • 1. Training Data • Different preprocessing methods lead to different training data • GIGO: Garbage in, garbage out • 2. Classifier type • SVM/Decision Tree/KNN/Naïve Bayes/etc. • 3. Parameter of the classifier • Kernel in SVM • Regularization coefficient in Linear Family • K in KNN

5. Evaluation: Is my model good? • Evaluation metric: accuracy, AUC, precision-recall • Evaluation options: • Train-Test Split • Use train • Use test • Random split • Cross Validation • 3/5/10 folds • Leave-one-out

6. Evaluation for KDDCUP 2009 • Training data released first, test data quite late. • Submit at most 5 times per day to get AUC score on 10% of them. • Train on training data, and test on training data? • Split training data into two datasets: one for training and one for testing. • Cross validation.

7. How AUC is calculated? All are classified as positive, so tp/pos = 1 • Sort the predicted scores: • -- score of i-th sample, and • -- true label (-1 or 1) of the i-th sample • For each , use it as a threshold: • For samples 1 to i, classify them as negative (-1) • For sample i+1 to n, classify them as positive (+1) • Calculate Sensitivity = tp/pos, Specificity = tn/neg and a point in the curve is obtained. • Calculate the area under the curve. https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf

8. The Result Table >Thousands of combinations of * Preprocessing * Classifier type * Parameter

9. Automate the experiments! • Case: • Change in LibLinear with 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 100.0, 1000.0 (13 values) • Do it 13 times in your command line? • Or write a script?

10. DEMO: evaluation.pycross validation for evaluation • Cross validation • Average AUC score for different C in LibLinear

11. A few tips on programming • Use command line tools of the classifiers and execute them in a script. • Weka can also be used as command line tools. • LibSvm/LibLinear/FEST/sgd/etc. all command line style • Store your result gradually in case of crash in the middle. • Python’s logging module/just plain text files • Make your experiments repeatable. • Set the seed of random number generators

12. Assignment 2 • Task: Do cross validation on training set. • Reference: tutorial 5 and its example code. • Report – As concise as possible: • what you do, • what you get (result), • and findings/analysis • Deadline: 26 March 2014 • Submission:emailtozchenah@ust.hk

13. Project 1 Schedule • Project 1 Results Deadline: 25 March 2014 • Assignment 2 Deadline: 26 March 2014 • Do cross validation on training set • Project 1 Report Deadline: 31 March 2014 • Report and what you have tried and more importantly what you have found • Preprocessing steps • Classifiers • Ensemble methods