1 / 46

on human and model- based decision making Chris Snijders

on human and model- based decision making Chris Snijders. www.chrissnijders.com /eth2012. Overview of course content on a lecture-by-lecture basis Inspirational material Assignments. Chris Snijders c.c.p.snijders@gmail.com Eindhoven University of Technology

kyna
Download Presentation

on human and model- based decision making Chris Snijders

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. on human and model-baseddecision makingChris Snijders

  2. www.chrissnijders.com/eth2012 • Overview of course content on a lecture-by-lecture basis • Inspirational material • Assignments

  3. Chris Snijders c.c.p.snijders@gmail.com Eindhoven University of Technology Background in mathematics(game theory / econometrics) PhD in Sociology, nowintoDecision Making www.chrissnijders.com/me

  4. Passing the course … • Presence and participation • Create a “CaseFile” based on the SuperCrunchers book (individually or in groups of 2) + evaluate others’ work • Write assignment about your own “Super Cruncher” idea + evaluate others’ work

  5. Overview of today • Somefamousexamples • The sciencebehindit • Computers as decision makers

  6. Case: Cook county hospital • Emergency Department • 250.000 patients per year • many persons without insurance • not enough rooms, overworked staff • 1996: Brendan Reilly director • (see Gladwell, 2005)

  7. Problem 1: acute chest pain Diagnose through: blood pressure, stethoscope: fluid in the lungs, how long have you been experiencing pain, how does it feel precisely, where does it hurt, does it always hurt or only when you exercise, have you had heart problems before, how about your cholesterol, do you have diabetes, let's look at your ECG, are there any heart problems in the family, do you use drugs, how old are you, are you in shape, do you smoke, do you drink, check appearance: stressed, overweight, .... High risk : 8 Medium risk : 12 Go home 30 p/day

  8. Reilly finds Goldman: obv 10,000 cases Only 4 things matter ECG Blood pressure Fluid in your lungs "unstable angina"

  9. Great! So let's do that! Or not... Implementation: … physicians protest … A test: 20 cases were given to several physicians Hardly any agreement between physicians!

  10. Reilly tests Goldman’s idea physician Goldman’s scheme vs 82% 95%

  11. A literature check … Clinical versus statistical prediction For instance (zie Grove et al., 2000) • Survival probabilities in medical procedures • Probability of recidivism • Probability of success of starting firms • Choice of job candidates • Diagnosing schizofrenia • Predicting school success • …

  12. The results … Over 160 studies When given the same info, the number of cases in which the expert wins = ??

  13. 0

  14. Models beat Humans(quiteoften) How canthisbe?

  15. ... we have some clues ... Our memory fools us (Wagenaar) “Dealing with probabilities / Base rate neglect”(Bar-Hillel) We emphasize the improbable’ (Stickler) Confirmation bias (Edwards, Wason) Mental sets (Redelmayer, Tversky) Hindsight bias (Fischhoff) Cognitive dissonance (Festinger)

  16. And there are more of these "Mental Floating Frankfurters"

  17. Restriction 1: “Mental sets” Connect the 9 dots with at most 4 straight lines, without lifting your pen from the paper. • • • • • • • • •

  18. Restriction 2: Memory “Where were you, when …” Shuttle Columbia Crew Lost Feb. 1, 2003

  19. Restriction 3: the “availability heuristic” Which is more likely, a plane crash or a car crash?

  20. Restriction 4: dealing with probabilities Suppose: a manager has a good intuition in business: • when a problem will arise: he gets a gut-feeling that something is wrong with probability 90% • when no problem will arise: he gets a gut-feeling that something is wrong with probability 10% On average, there is a problem in 5% of the transactions. The manager starts a transaction, and he gets a gut-feeling that something might be wrong. What is the probability that something is indeed wrong?

  21. Restriction 4: dealing with probabilities A murder has been committed. The only evidence available is DNA, found at the murder scene. DNA-research shows a match with your DNA. The probability that two persons are diagnosed as having the same DNA is about 1 in 100.000. How likely is it that you are the murderer?

  22. Restriction 5: overconfidence TrivialPursuit:estimatehowmanyquestionsyouanswercorrectly Estimates are generally too high ... and this gets worse with expertise!

  23. Restriction 6:Finding non-existent patterns

  24. Restriction 7: the noble art of finding a broken leg

  25. Restriction 8: where is the feedback?

  26. Restriction 9: Hindsight bias http://www.hss.cmu.edu/departments/sds/media/pdfs/fischhoff/HindsightEarlyHistory.pdf

  27. A list of biases …

  28. Decision making = Store, retrieve, combine

  29. Wherewere we? End of a big set of reasonswhyhumans (even expert humans) are oftenoutperformedby computer models.

  30. The competition: • “Naturalistic decision making” • Fast and frugal heuristics (covered only to some extent)

  31. Intuition at its finest http://dsc.discovery.com/tv-shows/dirty-jobs/videos/chicken-sexer.htm

  32. Reilly finds Goldman: obv 10,000 cases Only 4 things matter ECG Blood pressure Fluid in your lungs "unstable angina"

  33. AND? are we usingthistoday?

  34. NO!

  35. Back to the course ... ... the sciencebehindmany more questionsthatyoucanask in relationtosuch topics • This is aninnovation adoption processthatneedsto take standard hurdles. • Can we findconsistenciesacross topics? • Which kind(s) of crunchers are more likelytobeadopted? • etc... idea implementation

  36. Diagnosing Actinic Keratosis The power of intuitive judgment vs The rigor of statistical prediction

  37. Typical questions in the area When and why do the models win? Can we use the experts’ knowledge somehow? When are the models used, and when not (and why is that)? What can/should you do when you want to have a model-based solution? What prevents people from using models?

  38. About the Supercruncher book

  39. The Supercrunchers book Intro Who's doing your thinking for you? Creating your own data with the flip of a coin Government by chance Evidence based medicine Experts vs equations Why now? Are we having fun yet? On the web site [CaseFile] [Example] [Issue] [Method]

  40. “Super Crunching”, what is that? • Using (lotsof) data to predict something (think Twitter, Blogs, Airmiles, …) that we normally cannot predict • Using data to predict something that humans normally tend to predict • Experts vs models • Experiment • “Natural experiments”

  41. show website (if I had notdonethatbefore)

  42. To do • Read the book – cover to cover, asap • Think about the different cases you encounter, try to uncover general patterns • Upcoming assignment will be to create a “casefile” for one of the topics in the book: check for topics that interest you … • Next lecture at 17:00 (it’s a colloquium), and tomorrow at 13:00 again.

  43. Top 6 of statistically speaking total bogus professions Weather predictors 2. Sports predictors 3. “Profilers” 4. Art critics 5. Wine experts 6. Stock market experts

More Related