Research Design: The Forgotten Stepchild of Quantitative Research Methods Stephen G. West

1. Research Design: The Forgotten Stepchild of Quantitative Research Methods Stephen G. West Arizona State University

2. Preamble: Motivating Quotes for An Overlooked Perspective “You can't fix by analysis what you bungled by design." Light, Singer & Willett (1990) “By Design” “When it comes to causal inference from quasi-experiments, design rules, not statistics” Shadish & Cook (1999)

3. Two Surveys of Quantitative Research MethodsAiken, West, Sechrest, & Reno (1990); Aiken, West, & Millsap (2008), American Psychologist Population: PhD programs in APA’ s “Graduate Training in Psychology” All Programs N = 234 (achieved n = 201, 86% response rate) Subsample: 25 Elite Programs according to NRC (achieved n = 23, 88%) Surveyed Three Broad Areas • Statistics and Statistical Modeling • Measurement/Assessment • Research Design/Causal Inference

4. Selected Results From 1990 to 2008 surveys, gains in exposure of PhD students to new techniques in statistics (e.g., SEM, MLM); also minor gains in judged competence in statistics Some gains in measurement (still not strong) NO improvement in research design Judged competence of graduates: high in laboratory experiments All other designs (field experiments, longitudinal, quasi) judged competence weak to nonexistent

5. Instruction: Research Methods 57% of departments had some form of graduate departmental research methods segment or course. About 30% had no course (either department or program-based) Elite and Non-elite programs did not differ Example: Statistical Power (36% of grad students) could use in own research—i.e., do a power analysis, not necessarily use design features to strengthen power Respondents: Free listing of topics to be added to curriculum or be covered in new faculty hires—research methods almost never listed or perceived to be needed

6. Some History: Psychology’s Current Perspective on Research Design Canon of Psychology’s Research Design Developed largely around Donald Campbell 1950s, 1960s, 1970s at Northwestern Group of scholars—Robert Boruch, Thomas Cook, Albert Erlebacher, Lee Sechrest, Benton Underwood; graduate students (e.g., David Kenny; Charles Reichardt) and postdocs (David Rindskopf, Will Shadish) Key Feature: Represented multiple substantive areas within psychology and education Developed the perspective: KEY SOURCES: Campbell (1957); Campbell & Stanley (1963); Cook & Campbell (1979); Shadish, Cook, & Campbell (2002)

7. Psychology’s Perspective:Campbell’s Perspective on Research Validity Four Types of Validity • Statistical Conclusion Validity – Validity of inferences about association • Internal Validity – Validity of inferences about whether association reflects a causal relationship • Construct Validity—validity of inferences about higher order constructs (independent, dependent variables) • External Validity– Generalization across variation (to) persons, settings, treatments, measurements Cook & Campbell (1979); Shadish, Cook, & Campbell (2002)

8. Campbell’s Perspective Campbell and colleagues developed exhaustive (!?) list of threats primarily from research in psychology and education. Published lists of threats for each type of validity and provided examples. Provided lists of potential remedies to address threat Strategy for the researcher: 1. Identify and minimize the threats that exist in the particular research setting. 2. Add design elements to address remaining threats. 3. Compare the pattern of results predicted from the threat(s) to validity and the pattern of obtained results (pattern matching)

9. Example List 1: Threats to Statistical Conclusion Validity • Low Statistical Power • Violated Assumptions • Fishing and the Error Rate Problem • Unreliability of Measures • Restriction of Range • Unreliability of Treatment Implementation • Extraneous Variance in the Experimental Setting • Heterogeneity of Units • Inaccurate Effect Size Estimation Source: Shadish, Cook, and Campbell (2002), p. 45

10. Methods to Increase Statistical Power • Use matching, stratifying, blocking • Measure and correct for covariates • Use larger sample sizes • Use equal cell sample sizes • Improve measurement • Increase the strength of treatment • Increase the variability of treatment • Use a within-participants design • Use homogeneous participants • Reduce random setting irrelevancies • Use powerful statistical tests; insure their assumptions are met. Source: Shadish, Cook, and Campbell (2002), p. 47

11. Comment Most reviews of journal articles in Psychology show: for moderate effect size, inadequate statistical power of .50 to .60 vs. desired .80 benchmark Rossi (2013) Health Psychology area one exception Maddock & Rossi (2001) In my opinion, serious consideration of the threats to statistical conclusion validity and methods of increasing statistical power would have avoided much of the replication crisis in psychology, particularly if considered in the context of publication bias from meta-analysis.

12. Example List 2: Threats to Internal Validity • Ambiguous temporal precedence • Selection • History • Maturation • Regression • Attrition • Testing • Instrumentation • Interactive Effects (e.g., Selection x History; Selection x Maturation) Source: Shadish, Cook, & Campbell (2002), p. 55. Parallel lists exist for construct, external validity

13. Adding Design Elements and Pattern Matching:An Illustration: Stores Matched on Prior Sales, Zip CodeReynolds and West (1987)

14. Pattern Matching Adding Design Element: Nonequivalent DVs Selection x History Threat

15. Pattern Matching Adding Design Element Short Time Series Selection x Maturation

16. New Perspective 1: Rubin’s Potential Outcomes Approach Developed primarily by statisticians and epidemiologists Original development in 1970s (Rubin, 1974; 1978) Used in field research in public health, medicine, education Recent texts by Hernan & Robins (in press), Hong (2015), Imbens & Rubin (2015), Morgan & Winship (2015) Some differences among variants, but much agreement Largely complements traditional Campbell perspective

17. Some New Emphases

18. Potential and Observed Outcomes Promotes focus on selection/ missing data mechanism

19. Some Gains Over Traditional Approaches Methods of Treating “Broken” (imperfect) Randomized Experiments Attrition from Measurement Treatment Noncompliance (binary and dose-related) Variation in Treatment Conditions Contamination of Treatments (Non-independence) Methods of Treating Observational Studies, e.g., propensity scores Methods of Addressing Mediation Methods of estimating effects of Sequential Treatments Sensitivity Analysis and Bounds on Effects to assess potential effects of violations of assumptions

20. Slow Diffusion of Potential Outcomes to Psychology and EducationIllustration: Propensity Scores (Rosenbaum & Rubin, 1983; 1984) Source: Thoemmes and Kim (2011)

21. New Perspective 2: Pearl Origins in Bayesian networks and control systems engineering Pearl asks researchers to write down explicit causal networks Represents networks using directed acyclic graphs—brings graph theory and powerful associated mathematics to bear Well developed for binary, linear models (can be extend to nonlinear) Some foci identifying key nodes in graph that can block (causal paths) identifying back door paths that can confound causal effects emphasis on local, not global fit do(X) operator—node set to value; prior links to node blocked Key sources: Pearl (2000, 2009)

22. Some gains Elucidates consequences of causal premises and the data Elucidates confounding Broadens understanding of missing data (e.g. MAR) Broadens understanding of mediation Provides understanding of conditions for generalizability (termed transportability)

23. Challenges in Reinvigorating Instruction in Research Methods 1 1. New methods best taught in the substantive context in which they will be applied (Lovett & Greenhouse, 2000) Relatively few application examples in many substantive areas No comprehensive texts written for psychology and education Only a couple of introductory chapters: Shadish & Sullivan (2012); West, Cham, & Liu (2014) 2. New methods require relatively high level of mathematical and statistical knowledge. Require more precision in statement of treatments, relationships, nature of question. Require understanding new conceptions, new ways of thinking.

24. Challenges in Reinvigorating Instruction in Research Methods 2 3. New Methods require careful specification and examination of assumptions. Also, probing those assumptions: bounds, sensitivity analyses. Helps disrupt confirmatory biases. 4. (Some) new methods only work well in large samples. 5. New emphases on computer-intensive descriptive/predictive methods for complex data sets (big data) which “find” relationships then implicitly treated as causal. 6. Several areas of psychology emphasize randomized laboratory experiments in contexts in which there are be less relevance of classic and new approaches to research methods.

25. A Challenge: Achieving Relevance Several areas (e.g., behavioral neuroscience, cognitive, social) primarily emphasize randomized laboratory experiments. What do we have to offer? We need to demonstrate relevance. Develop appropriate level articles/chapters with area specific examples to illustrate potential gains. Needs input from both substantive area specialists and methodologists. We have not done much of this. Can be challenging.

26. Example: Methods Chapter for Behavioral Neuroscience • Observations • Effect size = 1.0 not unreasonable • (brain area destruction; ovarectomy) • 2. N = 20 per group is large given $500 • per rat • 3. Design methods of increasing power • often cannot be implemented Source: Talboom, West, & Bimonte-Nelson (2015)

27. A Final Challenge Foreshadowing Leona Aiken’s talk, we need to interest the next generation in research methods and identify ways to transmit this knowledge to graduate students in psychology and education. According to the mathematics genealogy project, Donald Rubin has 51 PhD students and 186 descendants (many, not all work in research design). Majority in academia. None appear to have faculty appointments in psychology or education; a few consult in these areas (e.g., Jennifer Hill; Liz Stuart). The number of methods faculty trained during the heyday of the Northwestern group was small, and are nearing retirement. They trained few students of their own in research methods. Few descendants. Lack of fecundity.

28. Some Opportunities 1:Workshop in Quasi-Experimental Designs Sponsor: The Institute for Education Sciences; No Fees State-of-the-art quasi-experimental methods for evaluating education interventions. Dates: July 31 - August 11, 2017 (Two Weeks) Location: Northwestern University Scholarly homes of presenters—Policy, Education, with exception of one presenter Thomas D. Cook, Northwestern University , Institute for Policy Research Peter Steiner, University of Wisconsin, Madison, Educational Psychology Stephen G. West, Arizona State University, Psychology Coady Wing, Indiana University, Bloomington, Public and Environmental Affairs Vivian Wong, University of Virginia, Education Attendees in previous years —mainly outside Psychology Most from Education and Public Policy

29. Some Opportunities 2:Atlantic Causal Inference Conference Several Workshops on Design and Approaches to Causal Inference No additional fee E. G. 2016 Workshop Topics Principal Stratification Causal Inference Multilevel Observational Studies Causal Inference with Error Prone Covariates Design Based Inference for Experiments and Observational Studies Sensitivity Analysis Most Instructors, Particpants from Statistics, Biostatistics, Epidemiology

30. Our Hope: Young Faculty with Interests in Research Design and Behavioral Science Felix Thoemmes Cornell U. Human Dev. Heining Cham Psychology Fordham U. Peter Streiner U. Wisconsin Education Coady Wing Indiana U. Public Affairs Vivian Wong U. Virginia Education Only one has a primary faculty appointment in psychology

31. Thank You Arizona State University Psychology Building Freie Universität Berlin Silberlaube

33. We only think when we are confronted with problems.--attributed to John Dewey Researchers prefer to have standard methods (e.g., randomized experiment) and standard paradigms. To carefully think through a new methodological approach (i.e., to evaluate the tradeoffs) or to think through what evidence is required to disconfirm one’s hypothesis requires much effort.

34. Alternative Didactic PresentationIntroduce Structure into Lists Threats to Internal Validity A. Threats arising from participant’s growth and experience 1. History 2. Maturation B. Threats arising from measurement process 3. Testing 4. Instrumentation 5. Statistical Regression

35. Why the loss of interest in Campbell’s Approach?Some Conjectures • No longer central research group expanding knowledge • Perception—few new developments (nothing new to learn)--but see Cook, Shadish—e.g., four arm; single subject designs) • Few academic offspring (Campbell vs. Rubin) • Usual Didactic Presentation—Lists—are boring—need alternatives • Lack of relevance (real and perceived) to some areas of psychology

36. Alternative Didactic Presentation 2:Introduce Useful Principles (RD) (RE) (OS) Source: Reichardt, Psychological Methods (2006)