300 likes | 416 Views
Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs. Tyler Lu and Craig Boutilier University of Toronto. Introduction. New communication platforms can transform the way people make group decisions.
E N D
Vote Elicitation with Probabilistic Preference Models: Empirical Estimation and Cost Tradeoffs Tyler Lu and Craig Boutilier University of Toronto
Introduction New communication platforms can transform the way people make group decisions. How can computational social choice realize this shift? People Choices Computational Social Choice Consensus
Introduction • Computational social choice • Aggregate full preferences (rankings) • Mostly study rank-based schemes (Borda, maximin, etc…) • Rank-based voting schemes rarely used in practice Problem:Cognitive and communication burden Our approach (recent work): Elicit just the right preferences to make good enough group decisions This work: Multi-round elicitation and probabilistic preference models to further reduce burdens Bob Cindy Alice 1 1 > 2
Outline • Preliminaries • Multi-round Probabilistic Vote Elicitation • Methodology and Analysis for One-round • Experimental Results
Preliminaries • Voters N = {1..n}; alternatives/items A = {a1…am} • Vote vi is a ranking of A • Complete profile v = (v1, …, vn) Bob Alice Cindy 1 voting rule r 2 3
Score-based Rules • Many rules have score-based interpretation • Surrogate for “total group satisfaction” • E.g. Borda, Bucklin, maximin, Copeland, etc… • Associates a score for each item given full rankings s(a, v) • Winner has highest score Bob Alice Cindy Borda scores s( , v) = 7 1 s( , v) = 6 2 3 s( , v) = 5
Partial Preferences • Partial vote pi is a partial order of A • Represented as a (consistent) set of pairwise comparisons • Higher order: top-k, bottom-k, … • Easy for humans to specify • Partial profile p Alice > How to make decision with partial preferences? > >
Decision with Partial Preferences • Possible and necessary co-winners [Konczak, Lang’05] • Recently: minimax regret (MMR) [Lu, Boutilier’11] • Provides worst-case guaranteeon score loss w.r.t. true winner • Small MMR means good enough decision • Zero MMR means decision is optimal
Minimax Regret Adversarial Best response
Vote Elicitation • MMR: good choices with “right” partial votes • How to minimize amount of partial preference queries to make good decision? • MMR-based incremental elicitation [Lu, Boutilier’11] • Problem: must wait for response before next query
Incremental Elicitation Woes • Each query is a (voter, pairwise comparison)pair • Exploits MMR, depends on all previous responses … > ? … ? > Bob annoyed at having to come back to answer query “interruption cost” YES NO Elicitor
Our Solution:Multi-Round Batching • Send queries to many voters in each round MMR ≤ ε Recommendation: Round: 2 Give your next top 1 Round: 1 Give your top 2 Elicitor 1. 1. 1. 2. 2. 2. 3. 3. 3. Interruption cost reduced
Multi-Round Probabilistic Vote Elicitation • Query class: “rank top-5”, “is A > B?”, etc… • Single request of preferences from voter • Have different cognitive costs • In each round π selects a subset of voters, and corresponding queries • Can be conditioned on previous round responses • Function ω, selects winner and stops elicitation • How to design elicitation protocol with provably good performance? • Worst-case not useful (for common rules) • Use probabilistic preference models to guide design
Multi-Round Probabilistic Vote Elicitation • Distribution P over vote profiles • Induced distribution over runs of protocol (π, ω) • Can define distribution over performance metrics Quality of winner: Max regret, expected regret Amount of information elicited: equivalent #pairwise comparisons, or bits. Number of rounds of elicitation Tradeoffs! Depends on what costs are important.
One-Round Protocol • Query type: top-k • “Rank your top-k most preferred” • Simple top-kheuristics [Kalech et al’11] • Necessary and possible co-winners • No theoretical guarantees on winner quality • Don’t provide guidance on good k • No tradeoff between winner quality and k
Probably Approximately Correct (PAC) One-Round Protocol • Any rank-based voting rule • Any distribution P over profiles • What is a good k? • p[k] are partial votes after eliciting top-k k*: smallest k, with prob. ≥ 1 - δ, MMR(p[k]) ≤ ε • As long as we can sample from P, we can find “approximately” good k… • Samples can come from historical datasets, surveying, or generated from learned distribution
Probably Approximately Correct One-Round Protocol General Methodology • Input: sample of vote profiles: v1, …, vt • MMR accuracy ε > 0 • MMR confidence δ > 0 • Sampling accuracy ξ > 0 • Sampling confidence η > 0 Find best the smallest k with
Probably Approximately Correct One-Round Protocol Theorem: if sample size then for any P, with probability 1 - η, we have • ≤ k* • P[ MMR(p[ ]) ≤ ε] ≥ 1 - δ- 2ξ
Practical Considerations • Sample size from theorem typically unnecessarily large • Empirical methodology can be used heuristically • Can generate histograms of MMR for profile samples from runs of elicitation • Can “eyeball” a good k • Can “eyeball” tradeoffs with MMR
Experimental Results • First experiments with Mallows distribution • Rankings generated i.i.d. • Unimodal, with dispersion parameter • t = 100 profiles (for guarantees, use bounds for t) • Borda voting • Simulate runs of elicitation • Measure max regret and true regret • Normalize regret by number of voters
Experimental Results x-axis is MMR per voter
Experimental Results Sushi 10 alternatives 50 profiles, each with 100 rankings
Experimental Results Dublin North 12 alternatives 73 profiles, each with 50 rankings
Concluding Summary • Model of multi-round elicitation protocol • Highlights tradeoffs between quality of winner, amount of information elicited, and #rounds • Probabilistic preference profiles to guide design and performance instead of worst-case • One-round, top-k elicitation • Simple, efficient empirical methodology for choosing k • PACguarantees and sample complexity • With MMR solution concept, enables probabilistic and anytime guarantees previous works cannot achieve
Future Work • Multi-round elicitation, top-k or pairwise comparisons • Fully explore above tradeoffs (associative different costs) • Assess expected regret and max regret