Effect Sizes

1. Effect Sizes

2. Model Fit • Question regards variance accounted for, reduction of predictive error etc. • R-squared • Residual standard error • BIC etc. • Related p(H|Data) • Others in Path Analysis/SEM such as GoF, RMSE • Others in classification problems • Classification rate, Area under ROC, Sensitivity, Specificity

3. Question involves variable relationships that range from between two variables to two sets of variables Simple relationships Correlation r Population: ρ Nonparametric versions For example: Kendall’s tau, phi, point biserial etc. Model level correlation/variance accounted for Multiple R R-squared (aka ‘amount of variance accounted for) Eta squared η2 Adj R-squared Omega-squared ω2 Multivariate: Canonical R and R2 Variable/factor level unique effects Raw coefficient Product of the covariance Relative importance among predictors/factors Standardized Partial correlation Partial eta Semi-partial Eta for an individual factor Others Average squared semi-partial ‘LMG’ Loadings Principal Components, Factor analysis, Canonical correlation etc. R family

4. D family • Concerns group differences • i.e. If no grouping factor, not applicable • Standardized Mean Difference • Hedges g, Cohen’s d etc. • Case level effect sizes • Measures of overlap

5. Not an effect size • Test statistics are related to ES but confounded with sample size • Observed p-values • Observed power • prep • probability of replication, seen in Psychological Science and perhaps elsewhere

6. Effect Sizes are Variable • And greatly so • Except for huge sample sizes, CIs for ES will be large • Do not fool yourself into thinking that any one study is reporting the true effect any more than they are reporting the true mean or anything else • Single samples are not the population

7. Effect Size as Statistical Test • Effect sizes can be used as the basis for statistical inference • Does the R2 95% CI bottom out at zero? • Does the d or r CI contain zero? • Does the coefficient CI contain zero? • These would tell you exactly what the statistical test result would be for the effect of interest

8. Effect Size is Not Importance • Large effects may also be terribly obvious and uninteresting, and the ES also won’t tell you if it’s an indirect or spurious effect • Small effects may be extremely important or simply interesting theoretically • Effect sizes by themselves do not tell you ‘cause size’ • Effect sizes do not determine the importance of an overarching theory • ES are just like any other statistic, an aid to making the interpretational decisions, particularly regarding practical effects, but in the end, the importance of any research result is determined by the researcher alone