C ausal inference from multiple studies, critical assessment of assumptions

Causal inference from multiple studies, critical assessment of assumptions Presented by: Bethany Harmon, Jenny Jackson, and Jill Pawlowski H 615 November 22, 2013

Overview • Generalized Causal Inference: Methods for Multiple Studies (Ch. 13) • Multistudy programs of research • Summaries of multiple studies • Narrative reviews • Quantitative reviews • Critical Assessment of Assumptions (Ch. 14) • Reflection Activity (10 minutes)

Choosing a psychotherapist • Imagine for a moment you are experiencing a difficult emotional time in your life… • Because you are an informed, intelligent person, you do some reading on psychotherapy and discover that many different approaches are available. • These assorted styles of psychotherapy, although they stem from different theories and employ different techniques, all share the same basic goal: to help you change your life in ways that make you a happier, more productive, and effective person http://www.edmondschools.net/Portals/3/docs/Terri_McGill/READ-TherapistChoice.pdf

Choosing a psychotherapist • Which type of therapy should you choose? • What you would really like to know is: • Does psychotherapy really work? • If it does, which method works best? • While many comparison studies have been done, most of them: • support the method used by the psychologists conducting the study • were rather small • are spread over a wide range of books and journals • This makes afully informed judgment extremely difficult. What should you do?

You have 2 options

Generalized causal inference: methods for multiple studies • Single studies typically lack large, heterogeneous samples across persons, settings, times, treatments, and outcome measures; rarely use diverse methodologies • Multiple studies usually have greater variation in all of the above = allows for better tests of generalized causal inference http://library.downstate.edu/EBM2/2100.htm

Generalizing from multiple studies • How is this accomplished? • Multistudy programs of research • Summaries of multiple studies

Multistudy programs of research • Phased models • Same topic, same researcher/lab • Research proceeds in phases, from basic research discovery to application in real world settings • Directed programs of experiments • One researcher/one lab over time, or multiple researchers/multiple labs at same time • Aim is to systematically investigate over many experiments the explanatory variables (moderators, mediators, constructs) that may account for an effect, gradually refining generalizations

Multistudy programs of research • Phased models • Example: phases of cancer research • Pre-phase: basic research on the effects of a treatment on cancer cells in test tubes and on animals • Phase I: review existing basic and applied research to determine testable hypothesis about new cancer agent • Phase II: method development to ensure accurate and valid research procedures and technology are available to study the agent • Phase III: intervention efficacy trials • Phase IV: intervention effectiveness studies • Phase V: field studies aimed at entire communities to determine public health impact

Multistudy programs of research • Phased models • Excel at exploring generalized causal inference • Rely on purposive sampling of persons, settings, times, outcomes, and treatments • Address the 5 principles of causal explanation: • Surface similarity: using human cancer cell lines to create comparable tumors on animals • Ruling out irrelevancies: deliberately making patient characteristics diverse • Making discriminations: identifying which kind of cancers are most/least affected • Interpolation/extrapolation: varying drug doses to see how toxicity and clinical response may vary • Causal explanation: developing models of how a drug acts on human cancer cells so that the potency of that action can be increased

Multistudy programs of research Advantages Disadvantages • Control over aspects of generalized causal inference from study to study • Pursuit of precisely the questions that need answering at any given time

Summaries of multiple studies • Same topic, different researchers • Allows more precise estimates of effects than could be had from single studies • Better exploration of how causal relationships might change over variation in study features • Helps clarify nature of relationship, boundaries, behavior within those boundaries, and explanations • Wider knowledge base often results in more credibility than claims based on single studies

Narrative reviews of experiments • Describe existing literature using narrative descriptions without attempting quantitative syntheses of study results • Reviewer relies on statistical significance levels to draw conclusions about intervention effectiveness • Summary of votes suggests whether treatment works • Also allows examination of potential moderators of the generalizability of intervention effects

Narrative reviews combining experimental and nonexperimental research • Field experiments plus evidence from surveys, animal studies, and basic laboratory studies • Purpose is to match evidence across studies to a pattern that would be consistent with and explain an effect and clarify its generalizability • Useful when direct experimental manipulation of the treatment would be unethical or impossible with humans • Results can be controversial

Narrative reviews Advantages Disadvantages Difficult to keep straight all of the relationships between effects and potential moderators Traditionally rely on box score summaries of results from significance tests of outcomes Imprecise descriptions of study results Even more complex when trying to examine relationships among outcomes and potential moderators Overwhelming! • Hypothesis generation • Thick description of a literature • Theory development with qualitative categories and relationships among variables

Quantitative reviews of existing research • Glass (1976) coined the term meta-analysis • AKA “Mega-Silliness”, “Blossoming Nose Blemish”, and “Classic of Social Science” • http://garfield.library.upenn.edu/classics1983/A1983QF87200001.pdf • Meta-analysis converts each study’s outcomes to a common effect size metric, such that different outcomes have the same means and standard deviations. • The common effect size metric can then be more readily averaged across studies

Essential steps of meta-analysis • Problem identification and literature review • Aim: develop clear research question and framework of inclusion criteria for studies in the meta-analysis • Studies included in review must address the question of interest and be relevant to the treatments, units, settings, measures, and times outlined in the guiding problem formation • May also have methodological criteria

Essential steps of meta-analysis • Coding of studies • Meta-analyses use a common coding scheme to quantify study characteristics and results • Codes should reflect the researcher’s hypotheses • For example, in a review of interventions for childhood obesity, codes might be developed based on whether the intervention included… • Behavioral, educational, environmental, diet, and/or physical activity features • Parental or family involvement • Outcome measures (e.g. BMI, change in diet or physical activity behaviors) • Codes often include characteristics of the study report, participants, intervention, intervention process, and methodology

Essential steps of meta-analysis • Computing effect sizes • Various study outcomes must be converted to a common metric before outcomes can be meaningfully compared over studies • Two most appropriate effect size measures for meta-analyses of experiments: • Standardized mean difference statistic (d) – for continuous outcome measures di = Xti – Xciwhere Xti = mean of treatment group in ith study siXci = mean of comparison group in ith study si = pooled standard deviation of the two groups • Odds ratio (o) – for dichotomous outcome measures oi = AD BC

Essential steps of meta-analysis • Analyzing meta-analytic data • Theoretically, meta-analytic effect sizes are analyzed just like any other social or behavioral data, using both descriptive and inferential statistics with univariate and multivariate techniques • Unusual statistical features • Desirability of weighting effect size estimates by a function of study sample size • Use of tests for homogeneity of effect sizes • Hierarchical nature of meta-analytic data • Dependency of effect sizes within studies • Presence of publication bias

Essential steps of meta-analysis • Interpreting and presenting results • Generally, interpreting and presenting meta-analytic results poses few special problems • Keep in mind: most meta-analytic data are correlational • We never randomly assign studies to the categories analyzed • Quasi-experimental design features to rule out threats to validity are rarely used in meta-analysis • Meta-analysts record data observationally • Exception is for reviews of randomized experiments of the same intervention

Meta-analysis and the 5 principles of generalized causal inference • Surface similarity: assess the match between research operations and the prototypical features of the targets of generalization • Multiple studies typically represent many more constructs than any one study • Greater chances of finding studies with operations that reflect constructs that may be of particular policy or research interest • Reviews are superior to single studies in their potential to better represent prototypical attributes

Meta-analysis and the 5 principles of generalized causal inference • Ruling out irrelevancies: consider deviations from the prototype that might not change the causal conclusion • Must show that a given causal relationship holds over irrelevant features present in multiple studies • Some meta-analyses aim to decrease, not increase heterogeneity of irrelevancies • Others welcome heterogeneity to the extent that it resembles that found in the settings of desired application

Meta-analysis and the 5 principles of generalized causal inference • Making discriminations: demonstrate that an inference holds only for the construct as specified, not for some alternative or respecifiedconstruct • Aim is to make discriminations about the parts of the target persons, settings, measures, treatments, and time for which a cause and effect relationship will nothold • Meta-analysis helps to clarify discourse about important constructs that have and have not been studied, and to clarify the range and boundaries of causal relationships Relevance and irrelevance are always subject to revision over time

Meta-analysis and the 5 principles of generalized causal inference • Interpolation and extrapolation: specify the range of persons, settings, treatments, outcomes, and times over which the cause and effect relationship holds more or less strongly or not at all • Requires exploration of the existing range of instances to discover how effect sizes vary with position along the range • The extremes are likely to be further apart and the intermediate values more numerous than in a single study • Response surface modeling works better with meta-analysis than with single studies

Meta-analysis and the 5 principles of generalized causal inference • Causal explanation: developing and testing explanatory theories about the target of generalization • Easier to break down persons, settings, treatments, and outcomes into their component parts in order to identify casually relevant components • Multiple regression can be used to identify redundancy among variables that moderate outcomes • Full explanation requires analysis of the micromediating causal processes that take place after a cause has been varied and before and effect has occurred

Conclusion • “Our enthusiasm for the methodology must be tempered by realistic understandings of its limitations.” (Shadish et al, 2002; pg. 446) • Flaws of meta-analysis are less problematic than those of narrative reviews • Be no less or more critical of meta-analysis than of any other scientific method

Quasi-experimentation: Ruling out Threats • Relay heavily on researcher judgments especially on plausibility • Improve inferences by adding design elements

Quasi-experiments: Pattern matching • Description • Intended to rule out specific identified alternative causal pathways • Problems • Specificity of theory • Chance • More likely with more complex predictions • Statistical tests for fit over mean differences • not as well developed

Excuses Against RCT • Problem: Quasi designs undermine likelihood of conducting RCT • RCT: Must be convinced of inferential-benefits over quasi before incurring costs • Role of disciplinary culture • “Each study calls for the strongest possible design, not the design of least resistance.” (pg. 487)

RCT Objection: Successful Implementation • Comparison not between well don RCT and Quasi but between imperfect RCT and well done Quasi • Counter: • RCT’s may have better counterfactual even with treatment degradation • Even with attrition, RCT give better effect size estimates • ** “Careful study and judgment to decide” (pg. 489)

RCT Objection: Strong Theory and Standardized Treatment Implementation • Intervention-as-implemented • Loss of statistical power • Large samples • Reduce influence of extraneous variables • Examine implementation quality • Understanding of effectiveness in real world setttings

RCT Objection: Unfavorable Trade Offs? • Critiques • Conservative Standards • Descriptive over explanatory • Belated program evaluation • Counters • Based on artificial dichotomies, correctible problems, and oversimplifications

RCT Objection: Invalid Model of Research Utilization • Critique • Use of naïve rational choice model • Counters • Causal relationships explained with information from multiple sources • Contested rather than unanimous outcomes • Short-term instrumental use with minor variants on existing practices • Research for conceptual changes

RCT Objection: Experiment and Policy Conditions Differ • Experiment outcomes affected by: • Enthusiasm • Implementation differs form real-world setting • Random-assignment • Counters • Lack of generalizability is an empirical question • Representation of real world conditions is a tradeoff • Planning can improve similarities in populations • True of any methodology • Disruptions in implementation is true in locally invented novel programs • Major Problem: Effects of policy can only be seen by studding the entire event

RCT Objection: Imposing Treatment Vs. Local Solutions • Critique: • Locally generated solutions produce better results • Counters: • Experiments were needed to find out that imposed treatment did not work • Null effects could be due to methodological weaknesses • Distinguish between political-economic currency and intervention effects

Causal Generation Theory • Randomization allows for causation with few assumptions • Follow 5-principles of generalized causal inferences when randomization is not an option • Researchers rely to heavily on weak purposive alternatives • Causal generalization is more complicated than establishing causal relationships

Non-experimental Alternatives • Intensive causal studies • Theory-based Evaluations • Weaker quasi-experiments • Statistical controls

Review of Key Terms • How do we know what is true? • Correspondence theory • Coherence theory • Pragmatism

Review of Key Terms • Quine-Duhem thesis as a reaction to logical positivism • An epistemology is a philosophy of the justifications for knowledge claims. • Naïve realism • Epistemological realism • Epistemological relativism

Shadish, Cook & Campbell on “Facts” • Critical multiplism • “Stubbornly replicable” observations by independent critics • Building multiple theories into observations • Dialectic process • Example of psychotherapy • Use and misuse of causal knowledge

Reflection • What is the role of theory in the construction of knowledge? • How will you choose your research question? • What do you do when you end up with non-significant results? • If you have significant results, how can you be confident in your findings? • How will you approach issues around use and misuse of your findings?

C ausal inference from multiple studies, critical assessment of assumptions