Herman Aguinis James C. Beaty Robert J. Boik Charles A. Pierce

1. Effect Size and Power in Assessing Moderating Effects of Categorical Variables Using Multiple Regression: A 30-Year Review Herman Aguinis James C. Beaty Robert J. Boik Charles A. Pierce

2. Introduction and Background • Moderated Multiple Regression (MMR) Equation for binary moderators: • Y = β0 + β1X + β2Z + β3XZ + εi • Null hypothesis for the MMR Equation: • β3 = 0 • Rejecting the null hypothesis indicates that Z moderates the relationship between X and Y

3. Information About MMR • The MMR models allows categorical moderators to take on any number of levels • MMR is the method of choice for testing hypotheses about moderating effects of categorical variables • MMR is used in a variety of research domains • i.e.: job performance and satisfaction, organizational citizenship behaviors, team effectiveness, etc. • Design, measurement, and statistical artifacts bias have been documented to cause observed downward moderating effects

4. Failure to Detect a Moderating Effect • Unavoidable artifacts in field settings may lead to erroneous sample-based conclusion there is no moderation when one does exist • This is an important problem because erroneous conclusions can have detrimental effects on theory development as well as on people’s careers

5. Goals of Study • Question 1: • What is the size of observed moderating effect of categorical variables in psychology and management published research? • Hypothesis 1: • There will be an increase in the magnitude of observed moderating effects over time • Question 2: • What would the size of moderating effect of categorical variables be if the studies were replicated under conditions where predictor X and criterion Y have perfect reliability?

6. Goals of Study • Question 3: • What is the a priori power of MMR to detect moderating effects of categorical variables in applied psychology and management published research? • Question 4: • Do MMR tests reported in applied psychology and management published research have sufficient statistical power to detect moderating effects conventionally defined as small, medium, and large?

7. Methods • Articles published in Journal of Applied Psychology (JAP), Personnel Psychology (PP), and Academy of Management Journal (AMJ) between 1969 and 1998 including: • a) at least one MMR analysis with • b) a continuous predictor and • c) a categorical moderator • Total of 106 articles and 636 MMR analyses

8. Computation of Effect Size • Sample size and predictor-criterion relationships across moderator=based subgroups are the two most influential factors on the observed effect size • ƒ2 is the ratio of systematic variance accounted for by the moderator relative to unexplained variance in the criterion • It is an effect size metric applicable in many diverse studies

9. Computation of Statistical Power • Violating MMR’s assumptions of homogeneity of error variance can possibly inflate Type II errors and, sometimes, Type I error rates

10. Q1: Observed Moderating Effects • Because of skewness, the distribution of effect sizes was normalized by using the Box-Cox power transformation • ANOVA was then conducted to examine differences across journals, showing a statistically significant result • Effect sizes in AMJ > PP > JAP (but difference between PP and JAP was not significant)

11. H1: Observed Effect Sizes Over Time • Pearson’s correlation coefficient between year of publication and effect size showed a statistically significant relationship • More recently published studies tend to report larger effect sizes compared to older studies

12. Q2: Size of Construct-Level Moderating Effects • Small increase of the overall effect size when error-free measures are used • Generally, measurement error does not substantially impact the magnitude of effect sizes

13. Q2: Size of Construct-Level Moderating Effects

14. Q3: Statistical Power • Effect sizes do not need to be large to be detected • Power in published research is sufficient to detect an effect size of .02

15. Q4: Power to Detect Small, Medium, and Large Effects • The mean power of the MMR test to detect what is conventionally defined as a: • Small effect (ƒ2 = .02) is .84 • Medium effect (ƒ2 = .15) ≈.98 • Large effect (ƒ2 = .35) is 1.0

16. Discussion • Observed moderating effects are small • Median effect size is .002 • None of the 95% CIs around mean effect size include zero • Observed moderating effects have increased in magnitude over time, although the trend is not strong • Statistical power in published literature has been sufficient • Approx 72% of tests had sufficient power to detect a small effect, approx. 85% to detect a medium effect and 100% to detect a large effect

17. Implications • Researchers need to be sensitive to methodological and statistical artifacts known to produce downward bias in observed effect sizes • Awareness of factors affecting MMR power needs to increase among researchers • In regards to Cohen’s definitions of effect size, the authors emphasize the choice of a targeted effect size in power analysis should be based on the specific research situation • Failing to recognize moderating relationships may lead to results with detrimental consequences

18. Class Questions • Is there another way to compute power accurately for MMR than by violating the MMR’s homogeneity of error variance assumption?  It seems that by doing this you decrease power in some situations and increase it in other situations that are not accurate.  Is there another method available that the authors could have used to reach the same results? • Why in Hypothesis 1 was Pearson correlation used as opposed to a t-test to analyze for significant difference over time?  What is the logic behind this approach?