Introduction to Supervised Machine Learning Concepts

1. Introduction to SupervisedMachine Learning Concepts PRESENTED BY B. Barla Cambazoglu⎪ February 21, 2014

2. Guest Lecturer’s Background

3. Lecture Outline • Basic concepts in supervised machine learning • Use case: Sentiment-focused web crawling

4. Basic Concepts

5. What is Machine Learning? • Wikipedia: “Machine learning is a branch of artificial intelligence, concerning the construction and study of systems that can learn from data.” • Arthur Samuel: “Field of study that gives computers the ability to learn without being explicitly programmed.” • Tom M. Mitchell: “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.”

6. Unsupervised versus Supervised Machine Learning • Unsupervised learning • Assumes unlabeled data (the desired output is not known) • Objective is to discover the structure in the data • Supervised learning • Trained on labeled data (the desired output is known) • Objective is to generate an output for previously unseen input data

7. Supervised Machine Learning Applications • Common • Spam filtering • Recommendation and ranking • Fraud detection • Stock price prediction • Not so common • Recognize the user of a mobile device based on how he holds and moves the phone • Predict whether someone is a psychopath based on his twitter usage • Identify whales in the ocean based on audio recordings • Predict in advance whether a product launch will be successful or not

8. Terminology • Instance • Label • Feature • Training set • Test set • Learning model • Accuracy Toy problem: To predict the income level of a person based on his/her facial attributes.

9. Instances

10. Categorical Labels

11. Numeric Labels $3K $4K $2K $5K $7K $8K $11K $1K $9K $12K

12. Features Blonde No White No Male 5cm Bald No White Yes Male 0cm White NoBlackYes Male 3cm Dark YesWhiteNo Female 12cm

13. Training Set Blonde No White No Male 5cm Bald No White Yes Male 0cm White NoBlackYes Male 3cm Dark YesWhiteNo Female 12cm

14. Test Set Dark No White No Female 14cm Dark No White Yes Male 6cm Dark NoBlackNo Male 6cm Dark YesWhiteNo Female 15cm

15. Training and Testing Test instance Set of training instances Testing Prediction Training Model

16. Accuracy Actual labels Predicted labels Accuracy = # of correct predictions / total number of predictions = 2 / 4 = 50%

17. Precision and Recall • In certain cases, there are two class labels and predicting a particular class correctly is more important than predicting the other. • A good example is top-k ranking in web search. • Performance measures: • Recall • Precision

18. Some Practical Issues • Problem: Missing feature values • Solution: • Training: Use the most frequently observed (or average) feature value in the instance’s class. • Testing: Use the most frequently observed (or average) feature value in the entire training set. • Problem: Class imbalance • Solution • Oversampling: Duplicate the training instances in the small class • Undersampling: User fewer instances from the bigger class

19. Majority Classifier • Training: Find the class with the largest number of instances. • Testing: For every test instance, predict that class as the label, independent of the features of the test instance. Class Size 13 8 4 Testing Prediction Model

20. k-Nearest Neighbor Classifier k = 3 • Training: None! (known as a lazy classifier). • Testing: Find the k instances that are most similar to the test instance and use majority voting to decide on the label.

21. Decision Tree Classifier • Training: Build a tree where leaves represent labels and branches represent features that lead to those labels. • Testing: Traverse the tree using the feature values of the test instance. Black White Not black Yes Black No

22. Naïve Bayes Classifier P( | ) = 0.40 P( | ) = 0.65 P( | )= 0.78 • Training: For every feature value v and class c pair, we compute and store in a lookup table the conditional probability P(v | c). • Testing: For each class c, we compute:

23. Other Commonly Used Classifiers • Support vector machines • Boosted decision trees • Neural networks

24. Use Case:Sentiment-Focused Web Crawling G. Vural, B. B. Cambazoglu, and P. Senkul, “Sentiment-focused web crawling”, CIKM’12, pp. 2020-2024.

25. Problem • Early discovery of the opinionated content in the Web is important. • Use cases • Measuring brand loyalty or product adoption • Politics • Finance • We would like to design a sentiment-focused web crawler that aims to maximize the amount of sentimental/opinionated content fetched from the Web within a given amount of time.

26. Web Crawling • Subspaces • Downloaded pages • Discovered pages • Undiscovered pages

27. Sentiment-Focused Web Crawling • Challenge: to predict the sentimentality of an “unseen” web page, i.e., without having access to the page content.

28. Features • Assumption: Sentimental pages are more likely to be linked by other sentimental pages. • Idea: Build a learning model using features extracted from • Textual content of referring pages • Anchor text on the hyperlinks • URL of the target page

29. Labels • Our data (ClueWeb09-B) lacks ground-truth sentiment scores. • We created a ground-truth using the SentiStrength tool. • Assigns a sentiment score (between 0 and 8) to each web page as its label. • A small scale user-study is conducted with three judges to verify the suitability of this ground-truth. • 500 random pages sampled from the collection. • pages are labeled as sentimental or not sentimental. • Observations • 22% of the pages are labeled as sentimental. • High agreement between judges: the overlap is above 85%.

30. Learner and Performance Metric • As the learner, we use the LibSVM software in the regression mode. • We rebuild the prediction model at regular intervals throughout the crawling process. • As the main performance metric, we compute the total sentimentality score accumulated after fetching a certain number of pages.

31. Evaluated Crawlers • Proposed crawlers • based on the average sentiment score of referring page content • based on machine learning • Oracle crawlers • highest sentiment score • highest spam score • highest PageRank • Baseline crawlers • random • indegree-based • breadth first

32. Performance