Consensus Relevance with Topic and Worker Conditional Models

1. Consensus Relevance with Topic and Worker Conditional Models Paul N. Bennett, Microsoft Research Joint with Ece Kamar, Microsoft Research Gabriella Kazai, Microsoft Research Cambridge

2. Motivation for Consensus Task • Recover actual relevance of a topic-document pair based on noisy predictions from multiple labelers. • Obtain a more reliable signal from the crowd and/or benefit from scale (expert quality from inexperienced assessors). • Variety of proposed approaches in the literature and in competition. • Supervised: Classification models. • Semi-supervised: EM-style algorithms. • Unsupervised: majority vote.

3. Common Axes of Generalization Documents Compute consensus for “new documents” on known topics. Relevance Observed in Training Relevance Not Observed in Training Observed In Training Use rules or observed worker accuracies on other topics/documents to compute consensus on new topics and documents. Topics Not Observed In Training Compute consensus on new topics for documents with known relevance on other topics. Note hidden axis of observed workers.

4. Our Approach • Supervised • Given gold truth judgments on a topic set and worker responses, learn a consensus model to generalize to new documents on same topic set. • Must be able to generalize to new workers. • Want a well-founded probabilistic method • Need to handle major sources of worker error. • Worker skill/accuracy. • Topic difficulty. • Needs to handle correlation in labels. • Correlation expected because of underlying label. • Note: will use “assessor” for ground truth labeler and “worker” for noisy labelers.

5. Basic Model The probability of relevance should depend on the document, worker response vector, and topic: where is a particular topic, is a particular document, is the event that is relevant to topic and is the response of worker k for the i,jthpair abbreviate

6. Exchangeability Related Assumptions • Given two identical sets of voting history, we assume two workers have the same response distribution. • Whether or not a worker’s opinion is elicited is not informative. • The ordering of responses/elicitation is not informative.

7. Relevance Conditional Independence • Assume conditional independence of worker response given document relevance. • implies workers have comparable accuracies across tasks. • Assume one topic independent prior on relevance • Referred to as naïve Bayes. Probability of relevance across all topics. Probability of a random worker’s response given relevance (across all topics).

8. Topic and Relevance Conditional Independence • Assume response conditionally independent given topic and relevance. • Implies workers have comparable accuracy within a topic, but varying across topics. • Assume topic dependent prior on relevance. • Referred to as nB Topic. Probability of relevance for this topic. Probability of a random worker’s response given relevance for this topic.

9. Worker and Relevance Conditional Independence • Each worker has a particular skill/accuracy in making relevance judgments. • This can be estimated by aggregating a history of accuracy across all tasks. • Responses are independent conditional on historical accuracy and relevance. • Referred to as nB Worker. Probability of relevance across all topics. Probability of this worker’s response given relevance (across all topics).

10. Evaluation • Which Label • Gold: evaluate using expert assessor’s label as truth. • Consensus: evaluate using consensus of participants’ responses as truth. • Other Participant: evaluate using a particular participant’s responses as truth. • Methodology • Use development validation as test to decide what method to submit. • Split development train into 80/20 train/validation by topic-docID pair (i.e. for a given topic all responses for a docID were completely in/out of the validation set.

11. Development Set • Skew and scarcity of development set, made model selection challenging. • Chose nB Topic since only method that outperformed the baseline (predicting most common class).

12. Results • Methods that report probabilities did better on probability measures in almost all cases and almost always improve on decision theoretic threshold. • Outlier’s performance in Log loss and conversion to accuracy implies poorly calibrated wrt decision threshold, but likely good overall. • Our method best on probability measures and near top in general.

13. Conclusions • Simple topic and relevance conditional assumption model produces • Best performance on probability measures on gold set. • Nearly best performance on accuracy. • Topic-level effects explain the majority of variability in judgments (on this data and over set of submissions). • Future: • Worker-relevance on test set • Worker-topic-relevance conditional independence model • Method performance versus best/median individual worker (sufficient data to evaluate?)

14. Thoughts for Future Crowdsourcing Tracks • Is consensus independent of elicitation? • Can consensus be studied independent of the design for worker response collection? • Probably okay if development and test sets are collected with the same methodology. • Likely collection design impact factors worth analyzing. • Number of gold standard in “training set” on topic • Number of labels per worker • Number of labels per item • Number of worker responses on observed items • Stability of topic-conditional prior of relevance

15. Questions?