Research Methods Fall 2011

1. Research MethodsFall 2011

2. Science Helps Avoid Bias Biases confound our judgment Overconfidence Confirmation bias Self-fulfilling prophecies Belief perseverance Illusory correlations Availability heuristic Conjunction fallacy Seeing patterns when there are none 2

3. 1. Our clinical judgment is biased biases in perception and interpretation are pervasive (e.g., blinding, grading, replication effect sizes lower) when figuring out causes of problems when figuring out what is effective for individuals change occurs for many reasons so difficult to detect specific causes with the naked eye hard to see subtle, delayed, or slow change resulting from specific causes “common sense” is often wrong harmful treatments can seem helpful Is a Research Class Necessary? 3

4. Early in therapy, my BPD client got worse. Caused by therapy? What could be other causes? After 8 months, her behavior improved. …soon after her parents threatened to stop paying for therapy What could be other causes? BPD Case Study 4

5. 2. Lists of ESTs is overly simplistic research does tell us what is effective NOW for my particular client in this particular setting largely ignores moderators minimally addresses principles of change (mediators) too many treatments on the list Is a Research Class Necessary? 5

6. Numerous therapists practice unvalidated and sometimes discredited methods Astrology, Tarot cards, palm reading Homeopathic remedies Primal Scream Therapy Sensory deprivation therapy Rebirthing Therapy Thought Field Therapy Facilitated Communication (autism) EMDR (SDPA article) Pseudo-Science is Easy to Believe 6

7. Science provides a systematic and (relatively) reliable approach to figuring out causes of important problems and change: Why Do I Care about Research? 7

8. Guidelines for physical health: Exercise, omega-3, anti-oxidants, vitamin D3 Red wine and alcohol healthy Flossing could make you live longer Breast cancer - soy and estrogen replacement Amalgam fillings Low calorie diets Cholesterol Would you get surgery from a doctor whose practice wasn’t based on reliable evidence? Why Do I Care about Research? 8

9. Why a Research Class? 1. Dissertation (quicker, easier) 2. Research is fun! 3. Better understand disorders (causes) 4. Prediction of clinical outcomes 5. Improve effectiveness with clients • evidence-based practice • what needs to change (causes) • how to change it (causes) 6. Publish (to get internships & postdocs)

10. Cannot please everyone! Testing is a drag! Try to balance pace Some students feel overwhelmed whereas others have said they have learned nothing new need solid grasp of stats Labeling/terminology is important This Class is Demanding! 10

11. Plaque on teeth correlated with plaque in arteries, therefore flossing could make you live longer (on TV news show) how to choose a topic - bring in an article the ideal experiment - a time machine RG mom said she was been worse since start of therapy RG better because of coercion "low calorie" foods can lead to weight gain (like a placebo effect) Garlic causes insomnia 11

17. It is problematic to ignore research Research findings can easily be misinterpreted or misused (be careful relying on experts or over-valuing statistical significance) Not all research is created equal

18. Research Findings Can Easily Be Misinterpreted 1. Many sources of bias (many subtle) 2. Confounds make results ambiguous 3. Results do not generalize

19. Ways to Determine What Works Clinical observation and intuition Treatment research can reduce bias and ambiguity 19

20. Three basic designs Observational/correlational studies Non-randomized manipulations Randomized experiments Correlational studies never randomize!! Not all Research Designs are Equally Persuasive 20

21. Too many causes to untangle Hard to isolate a specific cause (poor internal validity) Non-randomized Studies Often Yield Ambiguous Answers Regarding Cause and Effect 21

22. Studying Cause and Effect 1. Have a clear causal theory (if possible) 2. Causes must precede effects “cause” = independent variable (IV) “effect” = dependent variable (DV) OR the IV precedes (and predicts) the DV

23. Internal Validity ofNon-Randomized Studies Internal validity improves when you rule out confounds; for example, you improve internal validity when: • you include gender as covariate • you exclude men • you match non-randomized groups on gender • the supposed cause precedes and correlates with the effect

24. Cross-sectional (one time data collection) correlations among current events, experiences, behaviors, and constructs retrospective: some measures rely on memory for prior events, behaviors, etc. Prospective (longitudinal) Correlational Studies 24

25. PUT IN SLIDE ON PROSPECTIVE STUDY OF SHAME PREDICTING SUBSEQUENT SELF-INJURY, WHILE COVARYING BASELINE SELF-INJURY Prospective Correlational Studies 25

26. Percent Eventual Suicide of Persons at High Risk for Suicide Who Obtain Treatment vs. Refuse Treatment Motto (1976)

27. Percent Suicide for Contacted vs. Non-Contacted High Suicide Risk Persons Who Refuse Further Treatment Motto (1976) *=p<.05

28. If you believe that non-randomized studies are sufficient to evaluate treatment efficacy, then you have to admit that treatment-as-usual increases suicide among high risk individuals. If you don’t want to make that conclusion, then you need experimental research. 28

29. Non-randomized Studies Often Yield Different Findings than RCTs • suicide treatment study • estrogen therapy study • critical incident stress debriefing • cholesterol studies yields consistent findings

30. Research Validity What conclusions can be made with what degree of confidence? 1. Is the IV really the IV? DV? 2. Is I.V. really a true cause of D.V.? (internal validity) • Alternative interpretations of findings? • Does the intervention work? 3. Why did the “cause” lead to the effect? • How does the I.V. cause the D.V.? • Why does intervention work (mediators) 4. For whom are the causes truly causes (what populations and settings)?

31. Confound = Confuse Are the results “confused”? Is the I.V. confounded with another variable, and could this third variable be the main cause of the I.V. and the D.V.? Is the IV-DV relationship spurious? Does the I.V. cause the D.V. for the specific hypothesized reasons or are supposedly non-essential parts the primary causes?

32. What are Confounds? Tell your Grandma! 1. State the IV-DV relationship and why you think the relationship exists 2. Identify a confound • However, the IV-DV relationship “could be due to ___” 2a. State the IV-confound relationship 2b. State the confound-DV relationship 2c. Therefore, the IV-DV relationship may simply be due to the confound.

33. Confounds Example: Why is hot weather associated with ice cream sales? Causal link or spurious correlation? Example: If darker-skinned people commit more crimes, what could be the reason? Causal link or spurious correlation? Example: Why did CBT group end up with less depression than group who got supportive counseling? Causal link? Or is outcome difference due to some other difference between the groups? Example: Why do hairier players score more goals? Causal link or spurious correlation?

34. Mediation 7. Treatment leads to changes in outcome (direct effect) 8. The direct effect for treatment diminishes (in magnitude and significance) when the mediator is entered in the analysis. Treatment is effective because it changes ME. ME = stressful events or negative cognitions 34

35. Time confounds Maturation confound Cognitive training to young people Cognitive training to old people natural cognitive decline could mask benefit Internal Validity 35

36. Hairiness is associated with scoring more goals on the mixed-gender soccer team because having more hairs keeps muscles warmer. However, that may be due gender, strength, and speed. Men are more often stronger and faster players. Stronger and faster players can score more. Therefore, hairiness per se may not cause the more effective performance. psych medications is likely to be an important internal validity confound in a study of bipolar disorder. If the control group does not have equal amounts and types of meds then difference between bipolar participants and participants in the control group could simply be caused by the difference in medications. Bipolar participants may have worse memory simply because they have more toxic medications in their body that cause memory impairment because they cause brain atrophy or heavy sedation. We have to acknowledge this possibility and hopefully we can rule it out by controlling for these variables. Homework #1 36

37. TWO MOODLE SUBMISSIONS ARE REQUIRED for this assignment. http://www.dbtsandiego.com/form11 1) Identify one specific plausible mediator. Describe the mediator clearly and completely. 2) Identify one specific plausible threat to internal validity? Describe the confound clearly and completely, including the direction of the effect. 3) Identify one specific plausible moderator of the relationship between the independent and dependent variables, and explain the moderation effect clearly and completely, including the direction of the associations. Study 1: Consider a single-group correlational study to assess the effects of level of childhood sexual abuse on adult interpersonal violent behavior, both measured as continuous variables. All participants had at least some history of abuse during childhood, and level of abuse was a combination of frequency and severity of prior abuse episodes, ranging from a single instance of an older child touching the participant's genitals over clothing, up to multiple rapes involving intercourse with an adult stranger and threats of violence. Level of violence was a combination of frequency and severity, ranging from a single instances of verbal cruelty or destroying the property of others, up to multiple physical assaults or murder. Study 2: Consider a two-group study designed to test the hypothesis that having a history of childhood sexual abuse (binary independent variable: none versus any) increases risk of physically assaulting another person (binary dependent variable: never versus at least one). The percentage of adult participants who have physically assaulted another person at least once will be compared between the abused and non-abused groups. Homework #1 37

38. Confounds vs. Mediators The label depends on your theory of the causal process intrinsic or extrinsic part of your I.V.? If there is no main effect of the IV. mediators cannot be examined (overall) there can be moderator effects crossing regression lines mediators can be examined for subgroups 38

39. Internal Validity: Is the I.V. really a true cause of the D.V.? Construct Validity: Why did the “cause” lead to the effect? Depends on your theory Is explanatory variable an intrinsic part of I.V.? Internal vs. Construct Validity 39

40. Construct Validity Finding out: • Why does a I.V. lead to a D.V.? • Why does a manipulation cause a change? • Why is an intervention effective?

41. Two Types of Construct Validity Construct validity of IV some intrinsic part of the IV accounts for the IV-DV correlation or group difference Causal sequence (mediation) the IV => mediator => DV the mediator occurs after the IV the mediator occurs before the DV 41

42. Construct Validity Examples: • Ice Cream

43. Mediation 7. Treatment leads to changes in outcome (direct effect) 8. The direct effect for treatment diminishes (in magnitude and significance) when the mediator is entered in the analysis. Treatment is effective because it changes ME. ME = stressful events or negative cognitions 43

44. First state your specific theory of why the IV is related to the DV (intermediate cause) IV => M => DV Mediation requires: 1. IV is correlated with mediator 2. mediator is correlated with DV 3. The IV-DV correlation is reduced when the mediator is added into the prediction equation 4. A mediation test shows the IV-DV correlation reduction is statistically significant Mediation 44

45. Correlational mediation tests do not prove causal direction All pathways could be true: IV => M => DV M => IV => DV DV => M => IV DV => IV => M Mediation 45

46. Construct Validity Confounds 1. State your specific theory of why the IV is related to the DV (mediator) 2. Identify a competing theory of the IV-DV relationship (C.V. confound), e.g., how the relationship could be due to other intrinsic parts of the IV or DV (which may be more generic or broader characteristics of the IV or the DV. 3. Identify variables that share the confounds include the variable(s) as covariate(s) have those characteristics in control group 46

47. Validity vs. Confounds 47

48. Confound Examples Do BPD patients have more shame than normal controls? Does shame lead to self-injury? Does CBT reduce depression (compared to wait list control condition)? Does HRV biofeedback plus exposure lead to more reduction in fear than exposure alone? What are the IV and DV construct validity confounds? 48

49. Confound Examples Independent variables: Chinese (human, race, culture, Asian) shame (emotion, depression, anxiety, guilt) BPD (disorder, personality disorder, history of abuse, suicidality) CBT (support, hope, commitment, payment, problem-solving, new thinking, beh. activation) 49

50. Designing a Control Group Figure out all the variables (characteristics or experiences) that are likely to influence your dependent variable other than your independent variable (confounds) Include most important confounds in the control group Figure out all the components of your independent variable (population, disorder, manipulation, treatment, etc.) Decide what part of the independent variable you want to evaluate (broad or specific?) and include all other part(s) of the I.V. in the control group 50