310 likes | 416 Views
Changes in Probabilistic Causal Models. or: Do we stink at probability or what? James Baker PSY446. Participation. Mostly NU undergrads Some NU grad students Early data collection error / internet preteens filtered 67 data records 53 finished samples 13 of type 0 14 of type 1
E N D
Changes in Probabilistic Causal Models or: Do we stink at probability or what? James Baker PSY446
Participation • Mostly NU undergrads • Some NU grad students • Early data collection error / internet preteens filtered • 67 data records • 53 finished samples • 13 of type 0 • 14 of type 1 • 13 of type 2 • 13 of type 3
Gametype 0 (control) • A: Display "33%" Prob: .35 Price: $3 • B: Display "66%" Prob: .68 Price: $6 • C: Display "97%" Prob: .95 Price: $10 • No changes
Responses to gametype 0 • I can get 20 right if I always choose the most expensive... • I didn't really know how the computer was operating. I just figured that if the probability was high I would follow that suggestion. • I felt like 'B' was more than 66 percent, so after trying the other two, I stuck with B. • I chose to go with query B every time, because, on average, this would return 22 correct answers. Query A would do the same. And, I think, query C would return about 19, on average. Unless my math is wrong (which is very possible)...consistently choosing A or B is the best bet. • so i guess i got more 'correct' answers when i just played many times with low probablility of winning- but again- it's a little too late in the day for me to actually strategize.
Responses to gametype 0 • I think the computer worked on the probability given, but when you moved to a higher cost value, it decreased the likelihood that the probability given would work at a lower cost value. (Therefore, if you ask at $3, then move to $6, $3 will be less likely to work in subsequent rounds.) I mostly stayed with the $6 option, only moving to $10 when I wanted a correct answer and $3 when I wanted $6 to be more likely, which seemed to work. • I spent nearly all my money on the 97% question. • i would stick with c until it stopped being correct, then move to b for a while until it stopped working then go back to c, etc
Results-0 • > 90% A’s: 3 • > 75% B’s: 4 • > 70% C’s: 4 • 30% < B and C < 60%: 2 • Conclusion: Tendency to pick one favorite
Gametype 1 • A: Display "30%" Prob: .32 Price: $3 • B: Display "40%" Prob: .22 Price: $4 • C: Display "50%" Prob: .52 Price: $5 • Equally good choices • B ramps from .22 to .92 • Hypothesis: People will like C the best, might detect something off about B.
Responses to gametype 1 • If the computer was trying to pick up on my strategy, it would be pretty easy to learn the strategy. I picked C every time. Although it was the most expensive option, it had the highest probability of a right answer. So I picked it each time. • i would stick with c until it stopped being correct, then move to b for a while until it stopped working then go to a, then start over • totaly random • Honestly, I have no idea how the computer was operating. I think asking the same thing, then changing to something less likely, then going back to the original query, makes it more likely to get the correct answer, so I continued to follow that strategy. • Probablility shouldn't necessarily play into what results occur because it is still a tossup and 40% still seemed to produce positive results about 50 % of the time..
Responses to gametype 1 • I figured it was random, so I always picked A which was had the highest return rate. • I didn't make any changes; selecting C everytime seemed to be working. • It seems that 50% of a chance would be better odds despite the cost. • i figured i would last the longest if i kept choosing C • The odds for this game were that I would get twenty queries right, so I always went with the most expensive query because it was quicker.
Results-1 • > 95% A’s: 2 • > 70% B’s: 1 • > 70% C’s: 6 • Mixed: 4 • Among mixed: 57 A, 61 B, 77 C • Conclusion: We like round numbers?
Gametype 2 • A: Display "35%" Prob: .37 Price: $3 • B: Display "65%" Prob: .67 Price: $5 • C: Display "95%" Prob: 1.0 Price: $7 • C starts as best choice • C ramps from 1.0 to .3 • Hypothesis: People will start on C, then change to B/A when C gets worse.
Responses to gametype 2 • Same logic as last time (divided $200 by the cost/query and then multiplied by the odds of that query returning a "correct answer"). I predicted 27 correct answers for choice C, which would have been the most. • considering that the correct answer changed (I haven't figured out the pattern yet) when I stopped getting right answers I would switch. • Option C maximizes expected profit in the long run. Hence, I believe in following it should return in the best outcome in the long-run. • I just tried the 95% to play it safe. • i didn't change my strategy at all, i just kept going for what i thought was the most consistent, c every time. • stick with highest odds
Responses to gametype 2 • I think the computer worked on the probability given, but when you moved to a higher cost value, it decreased the likelihood that the probability given would work at a lower cost value. (Therefore, if you ask at $3, then move to $5, $3 will be less likely to work in subsequent rounds.) I mostly stayed with the $5 option, since it was a better than 50% chance and it cost an average amount. • just played C as long as I could (in writing, it seemed like the best deal) • The correct answer was not returned at anywhere near 95%, so I don't know how I'm supposed to guess which query to go with. • I stayed with the highest probability option • It seemed that the computer started returning wrong answers for C more than 5% of the time, even when I would vary my answers.
Results-2 • >90% B: 1 • >75% C: 10 • 50 < C < 60: 2 • The last two started with strings of C • Conclusion: Very slow to change mental model from samples
Gametype 3 • A: Display "30%" Prob: .10 Price: $1 • B: Display "60%" Prob: .35 Price: $6 • C: Display "90%" Prob: .70 Price: $10 • A appears to be best, but isn’t • A ramps from .1 to .6 • Hypothesis: People will start on A, discover it’s bad, switch to C/B, move around a bit, perhaps find A again when it’s good
Responses to gametype 3 • I stuck mostly with the cheapest one this time becuse it only cost $1. I was able to score pretty high by being able to ask such cheap questions. • When the $1 started winning I just kept going with those. I assumed that the probabilities set out in the beginning of the task were going to be followed so I just tried to mix up the type of answer I chose and was more patient with A and B. • Same strategy as always. Loyally played the odds. I only predicted 60 correct answers...but I guess I got rewarded for hanging in there. Vegas here I come! • i chose a alot since it was so cheap • i thought that there might have been different questions eventually- since there weren't i decided to stick with the most likely 'correct' answer from the computer.
Responses to gametype 3 • the cheapest query gave the best odds by about a factor of 3. I got even more right than the odds predicted. • tried A, but it failed 10 in a row; so then I tested C, and I was convined it was legit, so I plkayed it utnil the end, when I could only afford B • I tried the riskier ones when I got low on money. • seemed as though ninety percent wasn't actually appropriate.. it seemed more like 66%
Results • > 80% A: 5 • > 90% B: 1 • > 70% C: 0 • Mixed: 7 • Of A’s, 2 were ALL As and 3 had other selections in the middle section. • Conclusion: More churn with bad outcomes, again slow to change model of A
Aggregate Results • Tendency to pick one favorite • We like round numbers? • Very slow to change mental model from samples • More churn with bad outcomes, again slow to change model of A • Disconnect between comments and unconscious trends?
Confounding Factors • Display of percentages reinforcing static mental model
Fin James Baker
Backup • Gametype 0 • 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. • 55 choices: 50 A's, 0 B's, 5 C's. .91% A, .00% B, .09% C. • 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. • 32 choices: 3 A's, 25 B's, 4 C's. .09% A, .78% B, .12% C. • 30 choices: 6 A's, 15 B's, 9 C's. .20% A, .50% B, .30% C. • 24 choices: 4 A's, 3 B's, 17 C's. .17% A, .12% B, .71% C. • 35 choices: 5 A's, 29 B's, 1 C's. .14% A, .83% B, .03% C. • 20 choices: 0 A's, 0 B's, 20 C's. .00% A, .00% B, 1.00% C. • 33 choices: 0 A's, 33 B's, 0 C's. .00% A, 1.00% B, .00% C. • 27 choices: 1 A's, 16 B's, 10 C's. .04% A, .59% B, .37% C. • 60 choices: 55 A's, 4 B's, 1 C's. .92% A, .07% B, .02% C. • 33 choices: 0 A's, 33 B's, 0 C's. .00% A, 1.00% B, .00% C. • 66 choices: 66 A's, 0 B's, 0 C's. 1.00% A, .00% B, .00% C.
Backup • Gametype 1 • 42 choices: 3 A's, 4 B's, 35 C's. .07% A, .10% B, .83% C. • 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. • 47 choices: 7 A's, 21 B's, 19 C's. .15% A, .45% B, .40% C. • 51 choices: 20 A's, 15 B's, 16 C's. .39% A, .29% B, .31% C. • 49 choices: 5 A's, 35 B's, 9 C's. .10% A, .71% B, .18% C. • 47 choices: 16 A's, 3 B's, 28 C's. .34% A, .06% B, .60% C. • 50 choices: 14 A's, 22 B's, 14 C's. .28% A, .44% B, .28% C. • 65 choices: 63 A's, 0 B's, 2 C's. .97% A, .00% B, .03% C. • 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. • 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. • 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. • 41 choices: 2 A's, 2 B's, 37 C's. .05% A, .05% B, .90% C. • 40 choices: 0 A's, 0 B's, 40 C's. .00% A, .00% B, 1.00% C. • 66 choices: 66 A's, 0 B's, 0 C's. 1.00% A, .00% B, .00% C.
Backup • Gametype 2 • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 35 choices: 7 A's, 9 B's, 19 C's. .20% A, .26% B, .54% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 29 choices: 0 A's, 2 B's, 27 C's. .00% A, .07% B, .93% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 31 choices: 2 A's, 5 B's, 24 C's. .06% A, .16% B, .77% C. • 40 choices: 2 A's, 37 B's, 1 C's. .05% A, .93% B, .03% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 29 choices: 1 A's, 0 B's, 28 C's. .03% A, .00% B, .97% C. • 33 choices: 2 A's, 12 B's, 19 C's. .06% A, .36% B, .58% C.
Backup • Gametype 3 • 115 choices: 98 A's, 17 B's, 0 C's. .85% A, .15% B, .00% C. • 38 choices: 12 A's, 18 B's, 8 C's. .32% A, .47% B, .21% C. • 200 choices: 200 A's, 0 B's, 0 C's. 1.0% A, .00% B, .00% C. • 35 choices: 2 A's, 33 B's, 0 C's. .06% A, .94% B, .00% C. • 119 choices: 106 A's, 9 B's, 4 C's. .89% A, .08% B, .03% C. • 84 choices: 68 A's, 7 B's, 9 C's. .81% A, .08% B, .11% C. • 39 choices: 18 A's, 7 B's, 14 C's. .46% A, .18% B, .36% C. • 32 choices: 4 A's, 21 B's, 7 C's. .12% A, .66% B, .22% C. • 44 choices: 24 A's, 6 B's, 14 C's. .55% A, .14% B, .32% C. • 200 choices: 200 A's, 0 B's, 0 C's. 1.0% A, .00% B, .00% C. • 33 choices: 10 A's, 10 B's, 13 C's. .30% A, .30% B, .39% C. • 28 choices: 8 A's, 2 B's, 18 C's. .29% A, .07% B, .64% C. • 26 choices: 4 A's, 6 B's, 16 C's. .15% A, .23% B, .62% C.
Backup • Gametype 0 • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • B,B,B,A,A,A,B,B,B,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, • C,C,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A,A,B,B,B,B,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,B,B,C,C,C,C,C,C,C,A,A,A,B,A, • B,A,A,A,B,B,B,B,B,B,B,B,B,B,B,C,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, • B,B,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,C,C,C,C,B,B,C,C,A, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,A,A,A,A,A,A,A,A,A,B,C,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,A, • B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,
Backup • Gametype 1 • A,B,C,C,C,C,B,A,B,C,A,C,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,C,C,C,B,C,A,C,C,B,B,A,C,C,C,B,A,A,A,C,C,C,C,C,C,C,B,B,B,A,C,B,A,C,B,B,B,B,B,B,B,B,B,B,B,B,B, • B,A,A,A,A,A,A,A,A,A,A,A,A,A,B,B,B,B,B,B,B,C,C,C,C,C,C,C,C,A,A,A,A,A,B,B,B,B,B,C,C,C,C,C,C,C,B,B,A,A,C, • A,A,A,B,B,B,B,C,C,C,C,C,C,C,C,C,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A • A,A,A,A,A,C,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,B, • B,B,B,B,B,B,B,B,B,B,C,C,C,C,C,C,B,B,B,A,A,A,B,B,B,B,C,C,C,C,A,A,A,A,A,A,A,A,A,C,C,B,B,B,B,B,A,C,C,A, • C,C,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,A,A,B,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,
Backup • Gametype 2 • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,A,A,C,A,C,C,C,C,C,B,B,B,B,B,A,A,A,A,C,C,C,C,C,C,B,B,B,B, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,B,C,C,C,C,C,C,C,C,B, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,B,B,A,B,B,A,C,C,B,C,C, • B,B,B,B,B,B,B,B,B,B,B,A,B,B,B,B,B,B,B,C,B,B,B,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A, • C,C,C,A,C,C,C,C,C,B,B,B,C,C,C,B,B,B,B,B,B,C,C,C,B,C,C,C,B,C,C,B,A,
Backup • Gametype 3 • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, • B,C,A,A,A,B,B,B,A,B,B,C,C,C,C,B,B,B,B,B,A,A,A,A,A,A,A,A,B,B,B,B,B,B,B,C,C,C, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, • B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A, • B,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,B,B,B,B,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,A,A,B,B,A,A,A,A,C,C,A,B,B,B,A,C,A,A,A,A,A,A,C,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,B,A, • C,B,B,A,C,C,C,C,C,B,A,C,C,C,C,C,C,B,B,B,A,C,C,A,A,A,A,A,A,A,A,A,B,A,A,A,A,A,A, • C,B,C,C,C,C,C,C,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,B,A,A,A,A, • B,B,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A,A,A,A,A,C,C,C,C,C,C,C,C,C,B,B,A,A,A,A,A,A,A,A,A,A, • A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A,A, • A,A,A,A,A,A,A,A,A,A,C,C,C,B,B,B,B,B,B,B,C,C,C,C,C,C,C,C,C,C,B,B,B, • C,B,B,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,C,A,A,A,A,A,A,A,A, • C,B,B,C,C,C,B,C,A,C,B,C,A,C,C,B,C,C,C,C,C,C,C,B,A,A