1 / 46

Learning from Multi-topic Web Documents for Contextual Advertisement

Learning from Multi-topic Web Documents for Contextual Advertisement. KDD 2008. Outline. Introduction Sensitive Content Detection Sentiment Classification and Detection & Opinion Mining Experiments Conclusion. 1. Introduction (1/4). Contextual advertisement

avye-pugh
Download Presentation

Learning from Multi-topic Web Documents for Contextual Advertisement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning from Multi-topic Web Documents for Contextual Advertisement KDD 2008

  2. Outline • Introduction • Sensitive Content Detection • Sentiment Classification and Detection & Opinion Mining • Experiments • Conclusion

  3. 1. Introduction (1/4) • Contextual advertisement • A popular advertising paradigm where web page owners allow ad platforms to place ads on their pages that match the content of their sites • Problems: • The huge variety of content that can appear on a single web page • e.g. news sites, blogs, etc • Advertisers do not want to show their ads on pages with content like violence, pornography etc. (sensitive content) • They may also not wish to advertise on pages which contain negative opinion about their products

  4. 1. Introduction (2/4) • Objective • Not only to tell if a document has some targeted content in it, but also to label the parts of the document where the content is present • Sub-document classification • Classifier train on entire pages using page-level labels and test on individual blocks

  5. 1. Introduction (3/4) • Challenges: • Pages can contain unwanted parts • e.g., navigation panes, text advertisements, etc • Pages may also contain information on multiple topics • To collect large amounts of broad coverage single-topic training data, pre-clean and hand-label the blocks are difficult and expensive

  6. 1. Introduction (4/4) • This paper using multiple-instance learning (MIL) techniques – MILBoost to improve the performance of traditional methods (Naive-Bayes and Decision tree) • To train sub-document classifiers using only page level labels • The problems of sensitive content detection and opinion/sentiment classification for advertising can be considered as 2-class and multi-class classifying • In sentiment detection, a Naive-Bayes based MILBoost detector performs as well as the best block detector trained with block-level labels

  7. Outline • Introduction • Sensitive Content Detection • Sentiment Classification and Detection & Opinion Mining • Experiments • Conclusion

  8. 2. Sensitive Content Detection (1/3) • Sensitive content categories • e.g., crime, war, disasters, terrorism, pornography, etc • The various sensitive categories are grouped as one class labeled as “sensitive” • As long as a web page contains such content blocks, it will be marked as sensitive and the ad display will be turned off • The available training web pages are labeled at the page-level • The labels only tell whether a page contains sensitive content somewhere in it or not

  9. 2. Sensitive Content Detection (2/3)

  10. 2. Sensitive Content Detection (3/3) • If using the entire page, traditional classification methods run the risk of learning everything on the page as “sensitive” • To avoid this problem, a classifier that can accurately identify the parts of the page that contain the targeted content is needed • Better still is a classifier that can integrate the two tasks of locating and learning • Multiple Instance Learning framework

  11. 2.1. Multiple Instance Learning Boosting (1/8) • Multiple Instance Learning (MIL) is a variation of supervised learning where labels of training data are incomplete • Traditional methods where the label of each individual training instance is known • In MIL the labels are known only for groups of instances • Bag: a web page • Instance: a block of text

  12. 2.1. Multiple Instance Learning Boosting (2/8) • 2-class classification (sensitive or non-sensitive) • A bag is labeled positive if at least one instance in that bag is positive • A bag is labeled negative if all the instances in it are negative • The goal of MIL algorithm is to produce a content detector at the sub-document (block) level without having the block labels in the training data • This can save significant amount of money and effort byavoiding labeling work at the block level

  13. 2.1. Multiple Instance Learning Boosting (3/8) • Why MILBoost: • The state of the art traditional algorithms use boosting • We needed a framework to accurately measure the added effectiveness of the MIL framework • MILBoost has been successfully applied to a similar problem • Training a face detector to detect multiple faces in pictures when only picture level labels are available

  14. Negative ? Positive ? Negative Negative ? Positive ? ? ? Positive 2.1. Multiple Instance Learning Boosting (4/8) Positive Training initial classifier Positive Negative Positive Positive Negative Positive Negative Negative Negative

  15. 2.1. Multiple Instance Learning Boosting (5/8) • For each instance of bag , the probability of an instance is positive is given by where is the weighted sum of the output of each classifier in ensemble with t steps • is the output score of the instance generated by the classifier of ensemble

  16. 2.1. Multiple Instance Learning Boosting (6/8) • The probability that the bag is positive is a “noisy OR” • Under this model the likelihood assigned to a set of training bags is where is the label of bag i

  17. 2.1. Multiple Instance Learning Boosting (7/8) • Following the AnyBoost approach, the weight on an instance is given as • Each round of boosting is a search for a classifier which maximum where is the score assigned to the instance of bag i by the weak classifier (for a binary classifier ) • The parameter is determined using a line search to maximum

  18. 2.1. Multiple Instance Learning Boosting (8/8)

  19. Outline • Introduction • Sensitive Content Detection • Sentiment Classification and Detection & Opinion Mining • Experiments • Conclusion and Future Work

  20. 3. Sentiment Classification and Detection & Opinion Mining • Sentiment/opinion mining fromreview pages or blogs • A page may contain one or more topics • It is common to label reviews as “positive” or “negative” • Reviews are often not as polar or one sided as the label indicates • Blog review sites or discussion forums usually feature many people expressing varied opinions about the same product • These “mixed” opinions may act as noise during the training of traditional classification methods

  21. 3.1 Multi-target MILBoost Algorithm (1/6) • To apply MILBoost to the multi-topic detection task, it needs to be extended to a multi-class scenario • The “positive” and “negative” opinions can be treated as the target classes and the “neutral” class as the null class • A bag is labeled as belonging to class k if it contains at least one instance of class k • A bag can be multi-labeledsince it may contain instances from more than two different target classes • To deal with multi-labels • Creating duplicates of a bag with multiple labels • Assigning a different label to each duplicate

  22. 3.1 Multi-target MILBoost Algorithm (2/6) • Suppose we have 1 . . .K target classes and class 0 is the null class • For each instance of bag , the probability that belongs to class k (k {1, 2, . . . ,K}) is given by a softMax function, where is the weighted sum of the output of each classifier in the ensemble with t steps • is the output score for class k from instance generated by the classifier of the ensemble

  23. 3.1 Multi-target MILBoost Algorithm (3/6) • The probability that a page has label k is the probability that at least one of its content block has label k • Assuming that the blocks are independent of each other, the probability that a bag belongs to any target class k (k > 0) is • The probability that a page is neutral (or belongs to the null class 0) is the same as the probability that all the blocks in the page are neutral (“noisy OR” model)

  24. 3.1 Multi-target MILBoost Algorithm (4/6) • The log likelihood of all the training data can be given as • The weight on each instance for next round of training is given as • For the null class,

  25. 3.1 Multi-target MILBoost Algorithm (5/6) • Combining weak classifiers • Once the (t + 1)th classifier is trained, the weight on the classifier can be obtained by a line search to maximize the log likelihood function • Choice of classifier • In experiments, Naive Bayes and decision trees are used

  26. 3.1 Multi-target MILBoost Algorithm (6/6) • Testing • The new page is divided into blocks and the block level probabilities are computed using the classifier • The page level probabilities are obtained by combining the block level probabilities using noisy-OR • The block and page level labels are calculated using thresholds on the probabilities

  27. Outline • Introduction • Sensitive Content Detection • Sentiment Classification and Detection & Opinion Mining • Experiments • Conclusion

  28. 4.1 Sensitive Content Detection (1/5) • The data set contains 2,000 web pages which are labeled at the page level by human annotators • The label for each web page is binary, either sensitive or non-sensitive • There is no labeling done at the text block level • The evaluation has to be done at the web page level • Two popular base classifiers were used to build the MILBoost ensemble, decision trees and Naive Bayes • Both the MILBoost and the non-MILboost versions were run through 30 boosting iterations which end up with an ensemble of 30 classifiers • Area Under the ROC Curve (AUC) was used to evaluate the effectiveness of the various detectors

  29. 4.1 Sensitive Content Detection (2/5) Significantly outperforms both base classifiers and traditional boosted versions The MILBoost version achieved almost the same performance as the boosted page-classifier MILBoost further improves this performance by another 8.2%

  30. 4.1 Sensitive Content Detection (3/5) Althoug the AUC is about the same, the MILBoosted system is almost consistently better than the boosted page-classifier at the early part, where usually the operation point exists. This “early lift” brings practical advantage to the MILBoosted system.

  31. 4.1 Sensitive Content Detection (4/5) • Naive Bayes vs Decision Trees • Naive Bayes performed much better than decision trees in this task • The decision tree ensemble uses only about 700 keywords while NB theoretically uses the whole vocabulary, which is about 20,000 • The bigger feature set enables NB to generalize better at the testing stage

  32. 4.1 Sensitive Content Detection (5/5) • A Sensitive Content Detection Demo

  33. 4.2.1 Sentence Level Sentiment Detection (1/2) • The subjectivity dataset from the Cornell movie review data repository is used as the data set • 10000 “objective” and “subjective” sentences are labeled • These sentences were extracted from 3000 reviews, which are labeled at the review-level as well • A review is a “page” and a sentence is a “block” • The MILBoost detector is trained with the review data only using page-level labels, and then evaluated at the sentence-level with sentence level labels • Traditional page-level classifiers using boosted NB and decision trees are also built as benchmark algorithms for comparison • A page-level classifier using support vector machines (SVM) is also trained to compare the performance

  34. 4.2.1 Sentence Level Sentiment Detection (2/2) Train the classifiers with the reviews using page-level labels Train the classifiers with the reviews using sentence-level labels The highest performance in all algorithms using page-level labels, and it is comparable with the best sentence detector trained with sentence-level labels The SVM did not do as well as the NB classifiers for sentence classification trained either with the page- or with the sentence-level label. MILBoost improves the performance by about 10% over boosted decision trees

  35. 4.2.2 Multi-class Sentiment Detection (1/3) • Sentiment detection is naturally a three-class problem with “positive”, “negative” and “neutral” as class labels • The “positive” and “negative” classes are the target classes and the “neutral” class is the null class in the MILBoost setup • In these tasks, only built a multi-class MIL system based on Naives Bayes • The evaluation can only be done at the page-level

  36. 4.2.2 Multi-class Sentiment Detection (2/3) MILBoost based system improves upop the boosted Naive Bayes classifier The performance using SVM is comparable to the MILBoost system

  37. 4.2.2 Multi-class Sentiment Detection (3/3)

  38. 4.2.3 When does MIL improve on traditional methods? An Analysis Experiment (1/3) • This paper hypothesized before that multiple-instance learning should improve learning of traditional techniques when the amount of mixed content is high • The experiments were run on a car review dataset which contained 113,000 user reviews from MSN Autos • The objective of the learning task to identify negative opinions in review texts • These experiments want to show that as the amount of mixed content increases, MIL based approach can help traditional techniques improve

  39. 4.2.3 When does MIL improve on traditional methods? An Analysis Experiment (2/3) • This data set had an overall review rating score from 0-10 • Assume that if the rating score is 6 or below, there will be some negative opinions in the review text • Further split the negative reviews into two subsets, one with rating scores from 0 to 3 (data 0-3)and the other with ratings from 4 to 6 (data 4-6) • Presumably, the percentage of negative sentences in “data 0-3” will be much higher than that in “data 4-6” • If hypothesis hold right, MIL based techniques should give a bigger boost in “data 4-6”

  40. 4.2.3 When does MIL improve on traditional methods? An Analysis Experiment (3/3) With good quality training data, MILBoost does not give much advantage over traditional methods; however, if the training data has a high ratio of mixed content, then MILBoost does provide significant advantages There are three times as many pages in “data 4-6” as in “data 0-3” and the entire class distribution is highly biased towards positive with positive to negative ratio of 5:1 The MILBoost based system did not improve much over the regular boosted system Statistically significant improvement over traditional classifiers

  41. Outline • Introduction • Sensitive Content Detection • Sentiment Classification and Detection & Opinion Mining • Experiments • Conclusion

  42. 5. Conclusion • This paper explored sub-document classification for contextual advertisement applications where the desired content appears only in a small part of a multi-topic web document • The sub-document classifiers are trained when only page level labels are available • It showed that the MILBoost system improve the performance of the traditional classifiers in such tasks, especially when the percentage of mixed content is high • These systems provide good quality block level labels for free, leading to significant savings in time and cost on human labeling at the block level

  43. END

  44. AdaBoost

  45. Multi-labelled Document

More Related