Paul Kantor Rutgers May 14, 2007

1. Bayesian Models, Prior Knowledge, and Data Fusion for Monitoring Messages and Identifying Authors. Paul Kantor Rutgers May 14, 2007

2. Outline • The Team • Bayes’ Methods • Method of Evaluation • A toy Example • Expert Knowledge • Efficiency issues • Entity Resolution • Conclusions

3. Many collaborator • Principals • Fred Roberts • David Madigan • Dave D. Lewis • Paul Kantor • Programmers • Vladimir Menkov • Alex Genkin • Now Ph.Ds • Suhrid Balakrishnan • Dmitriy Fradkin • Aynur Dayanik • Andrei Anghelescu • REU Students • Ross Sowell • Diana Michalek • Jordana Chord • Melissa Mitchell

5. Overview of Bayes • Personally I • go to Frequentist Church on Sunday • shop at Bayes’ the rest of the week

6. Making accurate predictions • Be careful to not over fit the training data • Way to avoid this: use a prior distribution • 2 types: Gaussian and Laplace Laplace

7. If you use Gaussian Prior • For Gaussian prior(ridge): Every feature enters the models

8. If you use a LaPlace Prior • For Laplace prior(Lasso): Features are added slowly, and require stronger evidence

9. Success Qualifier AUC =average distance, from the bottom of the list, of the items we’d like to see at the top. Null Hypothesis: Distributed as the averagesum of P uniform variates.

10. Test Corpus • We took ten authors who were prolific authors between 1997 and 2002 and who had papers which were easy to disambiguate manually so that we could check the results of BBR. We then chose six KINDS OF features from these people‘s work to be used in training and testing BBR in the hopes that a specific Kind of features might prove more useful in identifying authors than another • Keywords • Co-Author Names • Addresses (words) • Abstract • Addresses (n-grams) • Title

12. Domain Knowledge and Optimization in Bayesian Logistic Regression Thanks to Dave Lewis David D. Lewis Consulting, LLC

13. Outline • Bayesian logistic regression • Advances • Using domain knowledge to reduce the need for training data • Speeding up training and classification • Online training

14. Logistic Regression in Text and Data Mining • Classification as a fundamental primitive • Text categorization: classes = content distinctions • Filtering: classes = user interests • Entity resolution: classes = entities • Bayesian logistic regression • Probabilistic classifier allows combining outputs • Prior allows creating sparse models and combining training data and domain knowledge • Our KDD-funded BBR and BMR software now widely used

15. A Lasso Logistic Model (category “grain”)

16. Using Expert Knowledge in Text Classification • What do we know about a category: • Category description (e.g. MESH- MEdical Subject Headings) • Human knowledge of good predictor words • Reference materials (e.g. CIA Factbook) • All give clues to good predictor words for a category • We convert these to a prior on parameter values for words • Other classification tasks, e.g. entity resolution, have expert knowledge also

17. Constructing Informative Prior Distributions from Domain Knowledge in Text ClassificationAynur Dayanik, David D. Lewis, David Madigan, Vladimir Menkov, and Alexander Genkin, January 2006

18. Corpora • TREC Genomics. Presence or absence of certain mesh headings • ModApte “top 10 categories” (Wu and Srihari) • RCV1 A-B ……see next slide…...

19. Categories. We selected a subset of the Reuters Region categories whose names exactly matched the names of geographical regions with entries in the CIA World Factbook (see below) and which had one or more positive examples in our large (23, 149 document) training set. There were 189 such matches, from which we chose the 27 with names beginning with the letter A or B to work with, reserving the rest for future use.

20. Some experimental results • Documents are represented by the so-called tf.idf representation. • Prior information can be used either to change the variance, or set the mode (an offset or bias). Those results are shown in red. Lasso with no prior information is shown in black.

21. Bio Articles ModApte RCV1-v2 Lasso 54.2 84.1 62.9 Var/TFIDF 55.2 84.6 70.8 Mode/TFIDF 53.3 83.6 64.5 Ridge 26.3 82.9 42.2 Var/TFIDF 52.2 83.8 68.9 Mode/TFIDF 41.9 83.1 62.9 Large Training Sets(3700 to 23000 examples)

22. Bio Articles ModApte RCV1-v2 Lasso 29.6 42.7 52.1 Var/TFIDF 34.3 61.3 50.7 Mode/TFIDF 36.4 58.5 51.5 Ridge 18.8 27.1 23.0 Var/TFIDF 35.7 61.5 53.0 Mode/TFIDF 33.9 62.1 48.8 Tiny Training Sets(5 positive examples + 5 random examples)

23. Findings • For lots of training data, adding domain knowledge doesn’t help much • For little training data, it helps more, and more often. • It is more effective when used to set the priors, than when used as “additional training data”.

24. Speeding Up Classification • Completed new version of BMRclassify • Replaces old BBRclassify and BMRclassify • More flexible • Can apply 1000’s of binary and polytomous classifiers simultaneously • Allows meaningful names for features • Inverted index to classifier suites for speed • 25x speedup over old BMRclassify and BBRclassify

25. Online & Reduced Memory Training • Rapid updating of classifiers as new data arrives • Use training sets too big to fit in memory • Larger training sets, when available, give higher accuracy

26. KDD Entity Resolution Challenges • ER1b: Is this pair of persons who have been possibly renamed, both really the same person? • ER2a: Which of these persons, in the author list, is using a pseudonym?

27. KDD Entity Resolution Challenges • ER1b: Is this pair of persons who have been possibly renamed, both really the same person? (NO) • Smith and Jones • Smith Jones and Wesson • ER2a: Which of these persons, in the author list, is using a pseudonym? YES This one!

28. Conclusions drawn from KDD-1 • On ER1b several of our submissions topped the rankings based on accuracy (the only measure used). Best: dimacs-er1b-modelavg-tdf-aaan • probabilities for all document pairs from 11 CLUTO methods and 1 Doc. Sim. model; summed; Vectors included some author address information

29. And …. • dimacs-er1b-modelavg-tdf-noaaan was second • no AAAN info in the vectors. • Third:dimacs-er1b-modelavg-binary-noaaan • combining many models, no binary representation. • CONCLUSION: Model averaging (alias: data fusion, combination) is better than any of the parts.

30. Conclusions on ER2a • On ER2a measures were accuracy, squared error, ROC area (AUC) and cross entropy. • Our : dimacs-er2a-single-X; 3rd (accuracy) ; 4th by AUC. • trained binary logistic regression, using binary vectors (with AAAN info). • probability for “no replacement” = product of conditional probabilities for individual authors. Some post-processing using AAAN info. • No information from the text itself

31. And …... • Omitting the AAAN post-processing (dimacs-er2a-single), somewhat worse (4th by accuracy and 6th by AUC). • Every kind of information helps, even if it is there by accident.

32. ER2: Which authors belong:The affinity of authors to each other (naïve)

33. ER2: Which authors belong?More sophisticated model • Update the probability that y is an author of D yielding, after some work, the formula:

34. Read More at: • Simulated Entity Resolution by Diverse Means: DIMACS Work on the KDD Challenge of 2005Andrei Anghelescu, Aynur Dayanik, Dmitriy Fradkin, Alex Genkin, Paul Kantor, David Lewis, David Madigan, Ilya Muchnik and Fred Roberts, December 2005, DIMACS Technical Report 2005-42

35. Read more at • http://www.stat.rutgers.edu/~madigan/PAPERS/tc-dk.pdf

36. Streaming algorithms • L1 • Good performance • Algorithms for Sparse Linear Classifiers in the Massive Data Setting. S. Balakrishnan and D. Madigan. Journal of Machine Learning Research, submitted, 2006.

37. Summary of findings • With little training data, Bayesian methods work better when they can use general knowledge about the target group • To determine whether several “records” refer to the same “person” there is no “magic bullet” and combining many methods is feasible and powerful • to detect an imposter in a group, “social” methods based on combing probabilities are effective (we did not use info in the papers)