Conjoint Adaptive Ranking Database System ( CARDS )

1. Good, Fast, Cheap and Easy? ConjointAdaptive Ranking Database System (CARDS) Ely Dahan Michael Yee, John Hauser & Jim Orlin EXPLOR Award Winning Presentation – September 22, 2004

2. The Problem • Current methods require manyquestions for few answers • Respondents must rate products they don’t like • Simplifying rules to narrowchoices not typically captured • Respondents make mistakesdue to fatigue, causing inconsistency • Is there a better way?

3. Smart Phone Example

4. Small Service Phone Brand Mini Keyboard Flip Example: Smart Phone • Respondent: Alex Bell • How does Alex choosea smart phone? 5 6 7 Utility Scores: Alex makes tradeoffs 9 10

5. Small Service Phone Brand Mini Keyboard Flip Example: Smart Phone • Respondent: Alex Bell • How does Alex choosea smart phone? 1 2 4 Process of Elimination:Focus on key features 8 16

6. Consider this tough task:Rank 32 Smart Phones based on your preferences Are wesurprised thatrespondents becomefatigued and makemistakes!? There are a billion, billion, billion, billion, ways for a respondent to rank 32 smartphones!

7. Prior Research on Adaptive Questioning • Johnson (1987, 1991) & Orme and King (‘02) Sawtooth ACA • Huber and Zwerina (1996), Aggregate utility balance • Arora and Huber (2001), Aggregate customization • Sandor and Wedel (2001), Aggregate + prior beliefs • Louviere, Hensher, and Swait (2000), Aggregate CBC Prior Research on Fast & Frugal Rules • Tversky (1969, 1972), lexicographic semi-order, elimination by aspects • Dawes and Corrigan (1974), unit weights, linear models • Montgomery and Svenson (1976), 2-stage processing • Thorngate (1980), efficient decision heuristics • Shugan (1980), cost of thinking (pair wise comparisons) • Johnson, Meyer, et. al. (1984, 1989), protocol anal., choice models can fail • Roberts and Lattin (1991, 1997), two-stage w/greedy • Gigerenzer and Goldstein (1996), Take the Best & others • Bettman, Luce, Payne (1996, 1998), Accuracy vs. effort, lexicography • Martignon and Hoffrage (2002), fast and frugal is robust

8. Two new ideas: • IDEA 1: We can now measure Alex’s process of elimination • IDEA 2: We can help Alex avoid inconsistent answers

9. IDEA 1: Phone Brand Mini Keyboard Flip Customer Insight:Respondents may be using a simpleprocess of eliminationto narrow choices for consideration “I will only considerflip phones, with mini-keyboards, from Blackberry”

10. IDEA 1: Customer Insight:Respondents may be using a simpleprocess of elimination How hard is it to identify each respondent’s simplifying rule? AGB

11. IDEA 1: How can we identify each respondent’sprocess of elimination ? • Tougher than it seems, because they may be using one of a huge number of possible rules • We solved this problem with a new computer technique (speedy) • We tested our theory and it works!

12. The big benefit of identifying respondents’process of elimination ? • Good Accuracy, customer insight • Fast 1 minute for them, quick for us • Cheap Pack more into the same study • Easy Reduce drudgery

13. Process of elimination Benefit: Cheap RankSome RankAll 7 minutes 2 minutes 2 7 Fast Could you use 5 extra minutes of survey time?

14. Benefit: Easy kind of fun okay about right long Ranksome Rankall Somewhatinteresting

15. How do we know if theprocess of eliminationidea is Good?

16. Holdout sample for rankings Pretty GOOD

17. Holdout sample for first choice Pretty GOOD

18. IDEA 2: Avoiding inconsistent answers The consistency criterion,a new approach Reduce response error by “guiding” respondents towards consistent answers Each choice must be 100% consistent with at least one set of utility scores

19. IDEA 2: Consistency Show product features Click on favorite cards Inconsistentcards just“disappear” GetUtilityscores Save lotsof clicks Keeping people consistent: Conjoint Adaptive Ranking Database System (CARDS)

20. IDEA 2: Consistency 10 9 7 6 5 Small Service Phone Brand Mini Keyboard Flip How do we keep people consistent? Imagine we knew Alex Bell’s utility scores… We would know how he would rank all 32 phones

21. IDEA 2: Consistency How do we keep people consistent? Imagine we knew every possible set of scores… Each set of utility scores are consistent witha unique ranking of all 32 phones Surprise: Consistent rankings are atiny percentage of the possible answers Eliminate rankings not on the consistent list

22. The big benefit of keeping respondentsconsistent ? • Good Accuracy, consistent • Fast minutes for them, quick for us • Cheap Pack more into the same study • Easy 50% to 75% effort reduction

23. Consistency Benefit: Easy 7 cards 17 cards Without consistency With consistency Consistency reduced effort 73%!

24. Extra benefits of consistency • Scalable • Utility Scores as you go • Emphasizes likes • Measures uncertainty

25. How do we know if consistency is Good?

26. Holdout sample for rankings Pretty GOOD

27. Holdout sample for first choice Just OKAY

28. Key Takeaways: • Good Predictive; Customer insight • Fast For them and for us • Cheap Pack more into one study • Easy Reduce drudgery & mistakes

29. Thank you for this exciting award! edahan@ucla.edu Good, Fast, Cheap, Easy demonstrations:http://orc-pumba.mit.edu/~myee/CARDS/conjoint.php Use email: nodebug7@ http://wow.mit.edu