1 / 56

Quasi-experiments

Quasi-experimental designs Experimental designs for “studies in nature”. Quasi-experiments. Naturally occurring events / case studies. Single group interventions. Experiments with non -equivalent groups. Time series designs. Quasi-experiments: Existing groups. .

aldona
Download Presentation

Quasi-experiments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quasi-experimental designs Experimental designs for “studies in nature”. Quasi-experiments Naturally occurring events / case studies Single group interventions Experiments with non-equivalent groups Time series designs Research Ethics

  2. Quasi-experiments: Existing groups  Naturally occurring events / case studies Group Naturally occurring event or social change Observe Single group interventions Experiments with non-equivalent groups Time series designs Research Ethics

  3. Naturally occurring events • Independent variable: • Often assessed after the event; e.g., natural disaster • Researcher has no control over strength / type of event • Participant selection • No control over who is exposed to the event • Some control over selection of sample (e.g., via targeted sampling) • Outcome (dependent) variables: • No control with archival data • Some control with surveys • Retrospective measures can help clarify findings or actually test a hypothesis. Group Naturally occurring event or social change Observe Week 12-13, quasi-experimental designs.

  4. Naturally occurring events • Typical use: Surveys or measures after an event. • Heuristic value: generating hypotheses for later study or confirm controlled data in “real world” setting. • Internal / External validity: • No control over selection of people into the event. • Potentially no control over selection into measurement group. • No control group; uncontrollable event, or other groups may not “need” the intervention (e.g., therapy) Group Observe Naturally occurring event or social change Possible random selection, or Convenience sample. • Event not controlled / manipulated. • Not a true Independent Variable May or may not have control over measures (e.g., surveys v. archival measures). Week 12-13, quasi-experimental designs.

  5. EventStudy question (“Predictor”) (“Outcome”) Naturally occurring events • Natural disaster / stressor • 3-mile island Stress -> immune system • S.F. earthquake Stress & coping • Crime / trauma • Iraq service, • 9 / 11 / 01 PTSD & treatment • Historical event • 9/11 & air travel ban Contrails & climate change • Economic contraction Voting patterns • Publicity / cultural event • Info. re: Hormone replacement Health behavior E X A M P L E Week 12-13, quasi-experimental designs.

  6. Naturally occurring events: Retrospective designs Using retrospective (measured) variables we can: • Clarify our interpretation of outcomes • Test hypotheses: Retrospective Event Outcome variable(s) (“Predictor variable”) variable earthquake stress & coping crime / trauma mental health historical event voting patterns cultural event health behavior Social support Psych. history Personal attitudes Demographics Week 12-13, quasi-experimental designs.

  7. Naturally occurring events Observation + retrospective data San Francisco earthquake & coping Event: - Major earthquake, 1984 Question: - Does social support “buffer” stress? Sampling frame: - Randomly selected survey participants Outcomes: - Standardized mental health scales - Self-reports of stress Retrospective: - population norms on outcomes - ancillary measures, e.g., social support Group Naturally occurring event E X A M P L E Findings: - High rates of stress Rx, - Social support ‘buffers’ stress Week 12-13, quasi-experimental designs.

  8. Naturally occurring events Health effects of bereavement Event: - Loss of spouse Question: - Does social support affect health? Sampling frame: - Hospital records, self-selected spouses Outcomes: - Blood draws for immune markers - Standardized mental health scales - Occupational functioning Retrospective: - Population norms on MH scales - Archivaldata: occupation & illness Findings: - Long-term immune suppression - Social support ‘buffered’ stress - Bereavement > other stressors E X A M P L E Week 12-13, quasi-experimental designs.

  9. Single group intervention studies Consumer Reports psychotherapy survey Research questions: • Does psychotherapy “work” from consumer view? • Who gets therapy / what does it consist of? • Do consumer responses vary by type of therapy? Research approach: • One shot case study / survey Sampling frame: • Any therapy or psychological service user • No real information re: population of therapy users. Sampling procedure: • 4,100 Consumer reports readers responding to “in magazine” mail-back survey form E X A M P L E Week 12-13, quasi-experimental designs.

  10. Consumer reports survey, 2 • Experimental Controls • Evaluate by gender, type of treatment, medications. Negatives: • Selection bias • no control over who got therapy (self-selection) • of those who got therapy, no control over who returned a survey (secondary self-selection) • Cursory outcome measure: “satisfaction” Positives: • Huge, national sample • Wholly anonymous, 3rd party data collection • “Real world” assessment of product quality E X A M P L E Week 12-13, quasi-experimental designs.

  11. Consumer reports survey, 3. Findings: • People who got more treatment (> 6 months) did better. • For general ψ health MH specialists did best, • For patients’ presenting problem(s) all specialists did about the same. marriage counselors worst. E X A M P L E Week 12-13, quasi-experimental designs.

  12. Naturally occurring events: Summary • No control over independent variable(s), only partial control over measurement: • An experiment is not possible • There cannot be a control group • “Pre-” measures not possible or practical • Virtue: • Assess naturally occurring or uncontrollable socially or politically important events • Provides “real world” look at processes that are typically studied in experiments • Archival data can help interpret the findings / “control” some alternate interpretations. • Liability: • lack of control group creates multiple threats to internal validity • No pre-measure makes interpretation (e.g., of change…) difficult. Week 12-13, quasi-experimental designs.

  13. Quasi-experiments: Existing groups Naturally occurring events / case studies  Single group interventions Experiments with non-equivalent groups Time series designs Research Ethics

  14. One group pre-test — post-test Group Observe1 Intervention or event Observe2 Selected or convenience sample. Baseline Assessment May or may not have control over measures (e.g., surveys v. archival measures). Outcome Assessment Typically controllable, but may be archival. Event or intervention May or may not be controllable by researcher, e.g., policy change. • Educational & social environments • Political or health policy change • Not feasible to have a control group • System-wide intervention / social change (school, public health campaign..) Uses: Week 12-13, quasi-experimental designs.

  15. Observe1 Confound Observe2 Key design feature: no control group. Group Observe1 Intervention or event Observe2 Threats to internal validity (confounds): Historical / cultural events occur between baseline & follow-up. • History Individual maturation or growth occurs between baseline & follow-up. • Maturation People respond to being measured or being a measured a second time. • Reactive measures Extreme scores at baseline “regress” to a more moderate level over time. • Statistical regression People leave the experiment non-randomly (i.e., for reasons that may affect the results…). • Mortality / drop-out Week 12-13, quasi-experimental designs.

  16. Single group intervention studies Effects of HIV testing on sexual risk. Question: - Does HIV testing lead people to be sexually safer? Event: - HIV testing & counseling Sampling frame: - Participants in testing centers Study structure: - Baseline retrospective interview at testing session - Follow-up interview 3 months later Quasi-controls: - Population characteristics to predict between-group differences Outcomes: - Self-reports of sexual risk E X A M P L E Week 12-13, quasi-experimental designs.

  17. HIV testing, 2 Findings: - Significant shifts toward safety - Few demographic predictors of change Threats to internal validity - Self-selection into testing group E X A M P L E - Mortality: non-random drop-out(?) - History: general shift in norms & behavior during study period - Instrument change; people answer more conservatively during a follow-up interviews Week 12-13, quasi-experimental designs.

  18. Single group intervention studies Example 2: “No Child left behind” testing. Question: - Do standardized tests & strict accounting facilitate learning? Intervention: - Standardized testing becomes integral to school evaluation. Sampling frame & - Longitudinal data across multiple Study structure: years in target school grades. - No control group possible. Quasi-controls: - Population characteristics to predict between-group differences Outcomes: - Standardized test scores E X A M P L E Week 12-13, quasi-experimental designs.

  19. Example: One group pre- post-, education, 2 Education reform & test scores. Findings: • Modest, statistically significant increase in scores • Usual demographic predictors of change; more affluent, better schools.. Internal validity?: • Reactive measures; teachers & students do better when measured; (they also cheat; see Houston Miracle article) • Instrumentation: kids get better at taking standardized tests, teacher better at teaching them • History: General cultural shift • Education more prominent in city • More affluent families sending kids to public schools E X A M P L E Week 12-13, quasi-experimental designs.

  20. One group pre- post- designs; Summary • One group pre- post- test design useful where: • An experiment is not possible • There cannot be a control group • Researchers have control over measurement and the independent variable • Virtues: • provide data on naturally occurring socially or politically important events • Pre-measure allows researcher to interpret change & examine status of groups at baseline. • History • maturation • statistical regression • reactive measures • mortality / drop-out • Liability: lack of control group creates multiple threats to internal validity: Week 12-13, quasi-experimental designs.

  21. Quasi-experiments: Existing groups Naturally occurring events / case studies Single group interventions  Experiments with non-equivalent groups Time series designs Group Observe1 Intervention or event Observe2 Group Observe1 Observe2 Contrast group Research Ethics

  22. Non-equivalent two-group designs Groups are not equivalent at baseline, due to.. • Self-selection • Non-random assignment • Use of existing groups • Participants not blind #1; Static Group Design Intervention or event Observe1 Group1 (No baseline) Contrast group Observe1 Group2 • Assessments may or may not be controlled • Survey or interviews • Archival / existing data, e.g., clinic records, grades • Intervention or event may or may not be controlled by researcher; • Existing program • Experimental intervention • Naturally occurring event Week 12-13, quasi-experimental designs.

  23. Existing groups Existing groups: Single self-selected group; no comparison possible • users of psychotherapy (or any product) • members of group or cult [contrast with demographically matched controls?] Two or more groups, with self-selection and / or "non-blind" assignment • Psychological interventions: therapy v. wait list, etc. Two or more groups, no random assignment • Comparing schools / cities / existing groups… Week 12-13, quasi-experimental designs.

  24. Non-equivalent designs; pre- post- #2 Two Group Pre- Post- Design Non-equivalent groups • Self-selection • Non-random assignment • Use of existing groups • Participants not blind Group Observe1 Intervention or event Observe2 Group Observe1 Observe2 Contrast group Intervention & Assessments often controlled by researcher in these designs. Similar to true experimental design, except for non-equivalent groups • Observation1 used to • Assess equivalence of groups at baseline • Assess change: the key outcome • Test for threats to internal validity: • Reactive measures • History, mortality effects • Regression effects Week 12-13, quasi-experimental designs.

  25. True experiments Quasi-experiments True vs. Quasi- experiments Randomly selected from target population Randomly assigned; groups identical at baseline. Procedures =for exp. & control groups. Complete control over the IV. Complete control over measures. • Unbiased assignment to groups • Participant and experimenter blind Convenience sample? Existing group? Self-selection? Self-selection? Non-random assignment? Existing grps? Not blind? Self-selected / existing groups ≠ procedures? Naturally occurring IV? Archival measures? Existing assessment?

  26. Quasi-experiments E X A M P L E Safer sex intervention for drug using, risky MSM • Multi-frame targeted sampling of gay/bisexual men • Intervention group: 6 90-min. group clinical sessions • Control group: 6 90-min. general group discussion sessions • Men randomized at first group meeting • Structured risk / attitude assessment at baseline, 3-, 6-, & 12-month follow-ups. Week 12-13, quasi-experimental designs.

  27. Quasi-experiment example E X A M P L E Safer sex intervention for drug using, risky MSM Sample selection • We do not have a sampling frame for this sub-group of MSM • We necessarily use a multi-frame convenience sample • We ask men to call for enrollment, so there is self-selection into the sample Week 12-13, quasi-experimental designs.

  28. Quasi-experiment example E X A M P L E Safer sex intervention for drug using, risky MSM Group assignment • The groups are randomly assigned. (Not self-selected!) • We try to convince participants that each arm of the intervention (control vs. treatment) are equal, but they still cannot be blind. • Of course the interventionists cannot be blind. Week 12-13, quasi-experimental designs.

  29. Quasi-experiment example E X A M P L E Safer sex intervention for drug using, risky MSM Procedures • The procedures are highly standardized, so they are equivalent across group. • All assessments are done with computer interviews, so the measurement procedures are also equivalent. Week 12-13, quasi-experimental designs.

  30. Quasi-experiment example E X A M P L E Safer sex intervention for drug using, risky MSM Treatment • We designed the experimental and control interventions, so we have complete control. • (Different if we were assessing a “naturally occurring” therapy or health program.) Week 12-13, quasi-experimental designs.

  31. Quasi-experiment example E X A M P L E Safer sex intervention for drug using, risky MSM Bottom line • A randomized controlled trial of a behavioral intervention has both true- and quasi-experimental features. • Groups cannot be perfectly equivalent • Interpretation of findings have to take into account: • Sampling methods • Non-blind participants & interventionists Week 12-13, quasi-experimental designs.

  32. True v. quasi-experimental designs • Emphasize internal validity • Assess cause & effect (in relatively artificial environment) • Test clear, a priori hypotheses • Emphasize external validity • Describe “real” / naturally occurring events • Clear orexploratory hypotheses • Participants randomly assignedto exp. or control groups • Participants & experimenter Blindto assignment • Non-equivalent groups • Existing groups • Non-random assignment • Participants not blind • Self-selection • Full control may not be possible • May not be able to manipulate the independent variable • Partial control of procedures & measures • Control study procedures • Manipulate independent variable • Control procedures & measures Week 12-13, quasi-experimental designs.

  33. Non-equivalent control group design, condoms Question:Does condom ed. & distribution: - increase safety - increase sexual activity Study structure: - NY = intervention schools, Chicago are contrast schools. - Baseline sexual health programming 9 mo. Follow-up Intervention: - Condom education & distribution in High School health classes Sampling frame: - Schools in New York & Chicago - Schools matched for SES, race, size E X A M P L E Outcomes: - Clinical measures: STIs - Self-reports: sexual activity & safety Week 12-13, quasi-experimental designs.

  34. Examples: Non-equivalent groups, condoms, 2 Condom distribution, cont. Findings: • NY (intervention) students; lower STI rate, safer sex • NY and Chicago students; similar levels of sexual activity • Thus; sexual health classes appeared to increase safety without increasing sexual activity. Internal validity?: • Reactive measures; Study is not blind; NY students know they are the intervention group • Non-equivalent groups: Possible differences between cities = unmeasured confounds E X A M P L E Week 12-13, quasi-experimental designs.

  35. Non-equivalent 2 group designs:Summary • Most common quasi-experimental approach. • Used where: • Some form of control or contrast group is possible • Groups cannot be equivalent: • Participants cannot be blindre: group assignment • Random assignment not possible • Must use existing groups • Participants self-select into (or out of) groups. • Virtue: • Study natural / “real world” interventions • Contrast group lessens major threats to internal validity • Liability: non-equivalent groups = possible confound. Week 12-13, quasi-experimental designs.

  36. Quasi-experiments: Existing groups Naturally occurring events / case studies Single group interventions Experiments with non-equivalent groups  Time series designs Group Measure1 M2 M3 M4 M5 M6… Intervention or event

  37. Interrupted time series design Group Measure1 M2 M3 M4 M5 M6… Intervention or event • Test effect of intervention or event on ongoing series of measurements. • Intervention may be experimental or observed • Policy shift, e.g., educational policy • Uncontrolled event; e.g., 9/11/01, Media event • Assessments may be experimental or archival • Successive cross-sectional surveys • Traffic data, clinic or crime reports, test scores Week 12-13, quasi-experimental designs.

  38. Time series designs • Threats to internal validity: • Sensitive to very local history • Single group possibly prey to confound • Advantage for internal validity • Eliminates carryover effects of repeated measurement • Sests maturation, history, reactive measurement, etc Group Measure1 M2 M3 M4 M5 M6… Intervention or event • Multiple baseline • Demonstrate highly stable effect • long-term crime rates • disease prevalence • economic performance… • Show steady rate of change • Hypothesis; tested by: • Shift in stable rate after intervention • Increase / decrease in rate of change after intervention Week 12-13, quasi-experimental designs.

  39. Example of interrupted time series:Shift in Baboon culture. Core question: Do baboon troops develop and transmit a learned “culture”? Baseline: Long-term observational data on aggressiveness in a specific baboon troop. E X A M P L E • Intervention: • Tuberculosis outbreak due to infected food. • Dominant / aggressive males fed first • are selectively infected • are naturally culled from troop • Naturally occurring event in >20yr. ongoing field study. Week 12-13, quasi-experimental designs.

  40. Baboon culture: findings Quasi-controls: Parallel data from other baboon troops. • Outcome measures:Standardized indices of aggression & dominance behavior • Core finding: • With dominant males gone, remaining males showed more cooperative behavior • Enhanced cooperation was transmitted across generation, showing learned “culture”. E X A M P L E Week 12-13, quasi-experimental designs.

  41. Example: Interrupted time series data The “Magic Johnson effect” on HIV testing Question: Does a celebrity or “role model” getting HIV affect others…? Data: Archival records of HIV tests reported to CDC, collected monthly • Data show stable baseline over multiple observations • Timing of intervention precise relative to data collection Intervention: Magic reports infection on national TV. • Uncontrollable, “naturally occurring” event • Tests hypothesis re: modeling effects in health behavior Core Finding: Initial spike in testing rates, followed by leveling off at higher base rate. • Initial increase expected • Hypothesis tested by longer-term shift in testing rates E X A M P L E Week 12-13, quasi-experimental designs.

  42. Example of time-series data: “Magic” / HIV effect. Time-series data showing shift in HIV testing after Magic’s announcement. Magic’s Announcement Initial spike E X A M P L E New, higher base rate Low & variable baserate of testing Multiple (monthly) measures. Tesoriero, J.M., Sorin, M.D., Burrows, K.A., LaChance-McCullough, M.L. (1995). Harnessing the heightened public awareness of celebrity HIV disclosures: “Magic” and “Cookie” Johnson and HIV testing. AIDS Education and Prevention, 232-250. Week 12-13, quasi-experimental designs.

  43. Multiple time series study Groups typically formed by blocking variable measured post-hoc; • Health claims in NYC v. other cities post- 9/11/01 • Younger v. older voting patterns post- Iraq invasion • Heterosexual v. gay HIV testing rates post- Magic Johnson media event. Multiple time series data Group 1 Measure1 M2 M3 M4 M5 M6… Group 2 Measure1 M2 M3 M4 M5 M6… Intervention or event • Hypothesis; tested by interaction of blocking variable by repeated measure: • Is shift in stable rate ( rate of change) greater in one group than another? Week 12-13, quasi-experimental designs.

  44. Blocking variables Testing blocking variables in the HIV testing time-series data. • Core questions: • Heterosexuals and Ethnic minorities had low HIV testing rates • Perceive HIV as a “white gay” problem? • They may lack resources or venues for testing. • Will having a prominent African-American Heterosexual disclose HIV+ status may change those perceptions? • Hypotheses: • Heterosexuals will respond more strongly than will gay/bisexual men. • African-American and Latino men and women will respond most strongly. E X A M P L E Week 12-13, quasi-experimental designs.

  45. Blocking variables: sexual orientation, 1. Testing blocking variables: Gay / IDU data. Risky men & injection drug users: High baseline, high variability. Gay / bisexual men: less variable, but low baseline. E X A M P L E Risky men & IDUs: slight increase, high variability. Gay & bisexual men: no change. Week 12-13, quasi-experimental designs.

  46. Blocking variables: sexual orientation, 2. Testing blocking variables: Heterosexuals In contrast to gay / bisexual men or IDUs, heterosexual show an initially low baserate. Followed by a large spike after the announcement E X A M P L E And a much higher new baseline. The hypothesis that heterosexuals would be more affected by the “Magic” announcement was supported by the interaction of Time x Sexual Orientation. Week 12-13, quasi-experimental designs.

  47. Blocking variables: ethnic differences Testing blocking variables: Ethnic differences. African-Americans and Hispanics show low baseline and a high spike after the announcement E X A M P L E Both groups go back toward their baseline shortly post-announcement. Week 12-13, quasi-experimental designs.

  48. Blocking variables: ethnic differences, 2. Testing blocking variables: Ethnic differences. • HIV testing among Whites was similar to African-Americans & Hispanics at baseline, • They showed stable, much higher testing rate after Magic’s HIV announcement. E X A M P L E Week 12-13, quasi-experimental designs.

  49. Summary: Blocking variables in time series data E X A M P L E A series of measures before & after an event allows us to clearly identify patterns of behavior, and to test group differences (via blocking variables). The hypothesis that ethnic groups would differ was supported by interaction of Time x the blocking variable of ethnicity (but in a direction that was not predicted: Whites showed more change). Week 12-13, quasi-experimental designs.

  50. Time series designs: Summary • Time series is most common with archival data: existing, standard records collected for other purposes. • Used where: • The hypothesis concerns changes in long-term trends • Typically an experiment cannot be run • Simple practicality or cost, e.g., health care issues • Ethics; crime rates, rates of domestic violence, etc. • The target events are not controllable. • Virtue: • Study natural / “real world” processes or interventions • Blocking variables – comparing time trends across groups -- lessens major threats to internal validity • Liability: lack of control = possible confound. Week 12-13, quasi-experimental designs.

More Related