Issues in Assessment, Part I PSYC 4500: Introduction to Clinical Psychology Brett Deacon, Ph.D. September 24, 2013

1. Issues in Assessment, Part I PSYC 4500: Introduction to Clinical PsychologyBrett Deacon, Ph.D.September 24, 2013

2. Announcements • Research participation is now open, pretest is closed • Exam #1 will be handed back on Thursday • Forthcoming response papers: • This Thursday: Garb & Boyle • Next Thursday: Hunsley, Lee, & Wood

3. Response Paper Questions for Garb & Boyle (2003) Article (due this Thursday) • Questions 1, 2, and 3: Describe three factors that make it difficult for therapists to learn from their own clinical experience

4. Response Paper Questions for Hunsley, Lee, & Wood (2003) Article (due next Thursday) • Questions • 1. Why do you think these tests are so commonly used by practicing psychologists? • 2. Should these tests be taught to clinical psychology graduate students? • 3. Is there sufficient justification for using the Rorschach, TAT, projective drawings, or anatomically detailed dolls in forensic settings?

5. What’s in Store for Part 2 of This Class? • Issues in assessment (e.g., the accuracy of clinical judgment) • Clinical interviewing • Fundamentals of psychological testing • Intellectual assessment • Personality assessment

6. Thinking Like a Scientist • Thus far, you have learned about the essential features of science and the relevance of science to clinical practice • What does it look like to “think like a scientist” in approaching clinical decision-making? • Diagnosis • Case conceptualization • Treatment planning • Other predictions

7. Prediction • Common examples of predictions psychologists make: • Would this patient benefit from treatment? Which kind? Should I see the patient myself or make a referral? • Is this patient going to attempt suicide? • Is this child sufficiently intelligent to succeed in a gifted program? • Will this criminal reoffend if released? • Will this applicant succeed in graduate school?

8. Prediction • Key questions for this week: • On what basis should we arrive at our predictions? How should we integrate our subjective judgments with empirical findings? • How accurate are judgments based on clinical judgment, vs. sticking to empirical findings, vs. their combination?

9. Prediction via Intuition vs. Empirical Data: A Real-World Case Example • Moneyball: http://www.youtube.com/watch?v=AiAHlZVgXjk • What is the take-home message of this movie, and its relevance to the work of a clinical psychologist?

10. Prediction – Case Example • Women who hired a hit man to kill her husband • Psychological evaluation to establish mitigating factors and argue against the likelihood of reoffending • What actually happened while she was in jail

11. Measuring Predictive Accuracy 4 possible outcomes of a prediction Actually Occurs Does Not Occur Predicted to Occur True Positive False Positive False Negative True Negative Predicted Not To Occur

12. Prediction – Case Example 4 possible outcomes Actually calls hit-man Does not call hit-man Predicted to call hit-man True Positive False Positive False Negative True Negative Predicted not to call hit-man

13. Prediction • Methods of evaluating predictive accuracy • Sensitivity – probability of a true positive • Specificity – probability of a true negative • Overall hit rate - % of all accurate predictions

14. Prediction - Example Scenario • 100 patients in a clinical trial had a 50% response rate to antidepressant medication • Researchers are interested in predicting who will respond to medication • Using a combination of psychological and biological measures, researchers try to predict each individual’s antidepressant response

15. What is the Sensitivity? Actual Responder Actual Non-Responder Predicted Responder 40 20 Predicted Non-Responder 10 30

16. What is the Specificity? Actual Responder Actual Non-Responder Predicted Responder 40 20 Predicted Non-Responder 10 30 40/50 Sensitivity (80%)

17. What is the Overall Hit Rate? Actual Responder Actual Non-Responder Predicted Responder 40 20 Predicted Non-Responder 10 30 40/50 Sensitivity (80%) 30/50 Specificity (60%)

18. Are These Figures Impressive? Actual Responder Actual Non-Responder Predicted Responder 40 20 Predicted Non-Responder 10 30 40/50 Sensitivity (80%) 30/50 Specificity (60%) 70/100 Hit Rate (70%)

19. Prediction and Base Rates • Predictive accuracy is highly dependent on the base rate of the outcome being predicted • Common base rate phenomena in clinical psychology: positive response to therapy, relapse after drug treatment, developing depression after acquiring a terminal illness • Low base rate phenomena: suicide, homicide, developing a psychotic disorder

20. The Base Rate Problem • A major reason for predictive errors is using high base rate predictors to predict low base rate outcomes • Examples of low-base rate outcomes: • Suicide • School shootings • Airplane hijacking

21. The Base Rate Problem • Examples of high base rate predictors: • Depression • Stress • No friends • Hates job • History of antisocial behavior • Substance abuse • Views pornography • “Suspicious” ethnicity (for racial profiling)

22. The Base Rate Problem: An Example • Airport security screening • What is the outcome TSA screeners are trying to predict? What is its base rate? • Which variables are used to predict this outcome? What are their base rates? + =

23. An Example: Predicting Suicide • Completed suicide base rate in depressed patients = 5/1000 (0.5%) • Suppose you’ve developed a predictive formula with 80% sensitivity and 80% specificity • When applied to 1,000 depressed patients...

24. Predicting Suicide Actual Suicide No Actual Suicide Predicted Suicide 4 199 No Predicted Suicide 1 796

25. Predicting Suicide Actual Suicide No Actual Suicide Predicted Suicide 4 199 No Predicted Suicide 1 796 4/5 Sensitivity (80%) 796/955 Specificity (83.3%)

26. Predicting Suicide • How useful is your suicide prediction test? • Based on these results, how might you use it? • If overall hit rate is the goal, what’s the best prediction?

27. Predicting Suicide • But wait a minute… • However, 199 non-suicides were false positives • Only 4 of the 203 predicted suicidal patients (2%) actually committed suicide • If used to determine hospitalization, 199/203 (98%) would be wrongly hospitalized

28. Predicting Suicide Actual Suicide No Actual Suicide 4/203 Positive Predictive Power Predicted Suicide 4 199 796/797 Negative Predictive Power No Predicted Suicide 1 796 4/5 Sensitivity 796/955 Specificity

29. Predicting Suicide • Sensitivity vs. positive predictive power • % of positive outcomes correctly predicted • % of predicted positive outcomes that were correct • Specificity vs. negative predictive power • % of negative outcomes correctly predicted • % of predicted negative outcomes that were correct

30. Prediction • Predictions are influenced by the costs of true and false positives and negatives • Situations in which the cost of a wrong prediction is high, thereby leading toward a bias in favor of false positives • Predicting suicide • Predicting homicide • Predicting terrorists at the airport • Predicting reoffending in sex offenders

31. The Problem with Clinical Prediction • The enterprise of prediction is often different for scientists and clinicians • Scientists (e.g., social, developmental, and cognitive psychologists) search for principles that apply to people in general • Clinical psychologists try to understand what caused what for an individual person

32. The Problem with Clinical Prediction • Most psychological principles that apply to people in general don’t apply to every single person • Group-level statistics provide us with probability estimates that can be applied to individual cases • We can use them to derive predictions based on what is most likely to happen even though we usually can’t know for sure

33. A Statistics Primer • 63% of Fox News viewers believe President Obama was not born in the United States. What are the odds that any Fox News viewer, selected at random from the general population of Fox News viewers, believes that President Obama was not born in the United States?

34. A Statistics Primer • The survival rate for a particular type of cancer is 80%. Your friend has this cancer. What is your friend’s probability of survival?

35. A Statistics Primer • But aren’t we treating your friend as “just a number?” • Are we “dehumanizing” her? • Your friend is a unique individual! She is optimistic, mentally tough, and has a great support network. So her probability of survival must be higher than 80%. Right?

36. A Statistics Primer • Who are the people among whom the 80% probability of survival was established? • Are they unique individuals, too? Or are they a homogeneous group of faceless people who differ fundamentally from your friend?

37. A Statistics Primer • Exposure therapy is more effective than antidepressant medication in the treatment of OCD. Nevertheless, the therapist decides to refer an OCD patient for medication as the initial treatment strategy the patient has a family history of OCD, suggesting a more biologically-based condition. Is this a good idea?

38. A Statistics Primer • The psychology GRE is the best predictor of success in graduate school. A particular applicant, despite strong qualifications in other areas, has a low psychology GRE. Relative to applicants with higher scores, how is this applicant likely to perform in graduate school?

39. A Statistics Primer • But what if the applicant, despite the low psychology GRE score, has a wealth of excellent research experience? • What if the applicant’s letter writers say the GRE score doesn’t reflect her ability? • What if the applicant sincerely promises to try extra hard in graduate school?

40. A Statistics Primer • How does your insurance company decide the cost of your car insurance premium? • How does your insurance company decide the cost of your life insurance premium?

41. Methods of Combining Data • Statistical (actuarial) method – use equation to integrate data • Use shared characteristics with others as predictors

42. Methods of Combining Data • Clinical method – subjectively integrating data • Relies on training, knowledge, experience, and intuition

43. The Problem with Clinical Prediction • Many clinicians are unwilling to apply group-level statistics to individual cases (i.e., engage in statistical prediction). • “Group level statistics don’t matter because this person is a unique individual.” • This is another way of saying, “I’m free to disregard scientific research findings in favor of my own judgment.”

44. Question • What is wrong with the idea that when predicting outcomes for a specific individual, group-level statistics are no longer relevant since each person is unique?

45. The Problem with Clinical Prediction • OK, so you think this person is a unique individual. Rather than incorporate findings derived from group-level research, you want to make a decision based on the facts and your impressions of this unique person. • Here’s the rub: exactly how are you supposed to combine all this information integrate in order to arrive at a prediction?

46. Combining Data in Prediction • An example: predicting potential to succeed in clinical psychology graduate school • Data: gender, age, ethnicity, overall GPA, psychology GPA, last 2 years GPA, GRE V & Q scores, GRE psychology subtest scores, research experience, clinical experience, personal statement, letters of recommendation, applicant’s interview (answers to questions, impressions from students and faculty, appearance, manner, etc.) • How should these data be combined to determine admission offers?

47. An Interesting Question… • Should graduate programs screen out applicants based on the results of a statistical equation, as opposed to the usual process of subjective evaluation of application materials by faculty members? Why or why not?

48. Methods of Combining Data • Statistical (actuarial) method – use equation to integrate data • Use shared characteristics with others as predictors

49. Methods of Combining Data • Clinical method – subjectively integrating data • Relies on training, knowledge, experience, and intuition

50. The Problem with Clinical Prediction • Many clinicians are reluctant to apply group-level statistics to individual cases (i.e., engage in statistical prediction). • “Group level statistics don’t matter because this person is a unique individual.” • “My own judgment is a more reliable guide than findings from scientific research.” • “Relying solely on statistics does not allow me to have any input.”