1 / 19

Differences Between Observed and Latent Confidence in Rank Ordering

Differences Between Observed and Latent Confidence in Rank Ordering. Brent J. Miller Mark Steyvers University of California, Irvine. Getting Expert Information. Getting Expert Information. Delphi Method. Iterative Collaboration. Independent Rank-Ordering Tasks.

loe
Download Presentation

Differences Between Observed and Latent Confidence in Rank Ordering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Differences Between Observed and Latent Confidence in Rank Ordering Brent J. Miller Mark Steyvers University of California, Irvine

  2. Getting Expert Information

  3. Getting Expert Information Delphi Method Iterative Collaboration

  4. Independent Rank-Ordering Tasks Order these U.S. Presidents, by term of office: 1st Participant 2nd Participant 3rd Participant Thomas Jefferson Thomas Jefferson Thomas Jefferson John Adams John Adams George Washington John Adams James Monroe James Monroe James Madison George Washington James Monroe James Madison George Washington James Madison

  5. Iterative Rank-Ordering Tasks Order these U.S. Presidents, by term of office: 1st Participant 2nd Participant 3rd Participant George Washington Thomas Jefferson George Washington James Madison John Adams Thomas Jefferson Thomas Jefferson James Monroe James Madison James Madison John Adams James Monroe James Monroe George Washington John Adams

  6. Rank-Ordering Experiment • 100 Subjects • 17 Ranking Questions • Chronological Events (Holidays, U.S. Presidents) • Physical Measures (Landmass, City Pop., State Locations) • Ordinal Information (Ten Amendments, Ten Commandments) • 2 Conditions

  7. Rank Ordering – Independent & Iteratives 17 Questions from University Students x 4 17 Questions via Mechanical Turk

  8. Thurstonian Model for Rank Ordering A B C σ1 X13 X11 X12 Participant1 Y1: A < B < C

  9. Thurstonian Model for Rank Ordering A B C σ2 X13 X11 X12 Participant1 Y1: A < B < C X23 X21 X22 Participant2 Y2 : A < C < B

  10. Iterative Collaboration Position 33 34 35 36 37 38 39 Participants

  11. Thurstonian Model for Item Representation A B C σ11 σ12 σ13 X13 X11 X12 Participant1 Y1: A < B < C

  12. Rank-Ordering Experiment • 50 Subjects • 8 Ranking Questions • Chronological Events (Holidays, U.S. Presidents) • Physical Measures (Landmass, City Pop., State Locations) • Ordinal Information (Ten Amendments, Ten Commandments) Independent (8) Iterative (8) 7 5 6 4 1 8 3 2 7 5 6 4 1 8 3 2

  13. Independent vs. Iterative Aggregate Performance Independent Iterative

  14. Independent Subject Generative Model i Items μi µi ≡ Ground truth σ2ij ~ InvGamma(.2,1) xi ~ Norm(µi,σ2ij) Yj ~ Rank(xj) yj xij σij j Subjects

  15. Iterative Subject Generative Model cut μij µi ≡ Ground truth σ2ij ~ InvGamma(.2,1) Xij-1 ~ Norm(µi,σ2ij-1) Yj-1 ~ Rank(xj-1) σi,j-1 σij yj yj-1 xi,j-1 xij i Items j Subjects

  16. Simulated Aggregate Performance

  17. Reduction in Iterative Error by Item Confidence Confidence for Items with Error > 4 Tau No Change

  18. Conclusions • Item-level knowledge may allow partial expertise between participants, allowing for improvement in group aggregation. • Subject item-level confidence appears to support the validity of our proposed model. • We should be able to use subjects’ confidence as an estimate of underlying item-level expertise.

  19. Differences Between Observed and Latent Confidence in Rank Ordering Danke!

More Related