Practical Significance as the Economic Impact of Effect Sizes Werner W. Wittmann University of Mannheim, Germany and EGA

1. Practical Significance as the Economic Impact of Effect SizesWerner W. WittmannUniversity of Mannheim, Germanyand EGAD_International Session:Translating Research and Communicating Effectively with Stakeholders Chair: Stephanie M. Reich, Vanderbilt University Evaluation 2004 – Fundamental Issues Atlanta, Nov. 3-6, 2004

2. Output in $ Program input in $ Cost-benefitanalysis e.g. for productivity e.g. life-qualityor life-satisfaction Program input in $ Cost-effectivenessanalysis Output in meaningful test-scores Two approaches for summative program evaluation Cost-benefit- and cost-effectiveness- analysis are two important tools for summative program evaluation

3. Input = Resources to be invested Output = Outcome = Benefit Program or interventionin I/O,clinical psychology,medicine or education COST OUTCOME U = Utility in monetary terms Udir = direct utility Uind = indirect utility Uint = intangible utility E = Effectiveness(non- monetary outcomes from program evaluations or meta-analysis ) U = Outcome composed of different non-monetary utility measures (e.g.. QALYs= Quality of life in years) Cdir = direct costsCind = indirect costsCint = intangible costs Comparisons of alternatives:- Intervention 1- Intervention 2, etc. Cost-outcome distinctions

4. The cost-benefit equation • Effectsize for the intervention or program (d) • Time the effect holds on in years (T) • The number of subjects treated or trained (N) • Total costs per treated or trained subject (C) • The standard deviation of productivity (SDPROD) in $ • The proportional overlap of effect with productivity (a) • These informations can be used in the following cost-benefit equation, to estimate the total net benefit in $

5. Outcome distribution using the metaphor of Antoine d‘Exupery‘s „Little prince“- story

6. CONTROL INTERVENTION 3 sigma effects an unrealistic dream !

7. d Pre-intervention control + Iz - Iz + Iz - Iz A more realistic effect size Post-intervention

8. Question: Given what constraints, gross utility just balances the total costs? Answer: Find the break-even-point, where gross utility = total costs:N * T * SDPROD * d * a = N * C Question:What effect-size do we have to produce for it? Answer: d BREAK-EVEN = C/ (T* a* SDPROD)

9. Question: How large is SDPROD? Answer:Many thanks to Frank Schmidt and Jack Hunter! They have shown via systematic research, that SDPROD estimated in many diverse job areas, turned out to lay between 40 % und 70 % of the annual salary. Newer research, as well as my own German ones, resulted in estimates of 70 % and larger! Assuming that the yearly average salary for a job category is 40000$, then SDPROD is between 16000$ (40%) and 28000$ (70%) These two estimates can later be used in a sensitivity analysis to check the robustness of a cost-benefit analysis.

10. Effect size at the break-even-point an example: Assume that a training program total cost C=8000$ per trainee, The effect holds on for two years (T= 2). The outcome measure assessing communications skills to improve costumer oriented selling has an overlap of a=.60 with productivity. Yearly salary in that job category is 40000$ and we use the 70% estimate, thus SDPROD= 28000$. d = 8000$/(2*.60*28000$); d = .24 In Cohen‘s classification this is an effect close to a small one and meta-analysis results might additionally hint that programs similar to the one we use or want to evaluate have still larger effects.

11. THE RETURN ON INVESTMENT CONCEPT (ROI) Problem: Nobody is satisfied just getting back the money invested after a year. The amount of money returned after a year should be larger. Thus this quotient of money invested to money returned, which is labeled as ROI should be larger then one, i.e. : ROI > 1 But how much larger? To get a feeling what ROI-coefficients are considered as impressive we can look at other areas! Interest rates mirror this for example. As psychologists we have to compete against other stakeholders with respect to investment decisions. Should someone invest money in programs we have to offer or would that investment result into a waste of money or opportunity costs? How can we a-priori answer such a question?

12. Computing an effect size necessary to compete with others claiming to have a certain ROI • Other competitor about investments may claim having a ROI=2, which means after a year for each dollar the investor get two dollars back. Impressive indeed. Can we compete, what effect size do we have to demonstrate to get a similar ROI? Well • d necessary = ROI* dBreak-Even and using data from the example above: • d necessary = 2 * .24 • d necessary = .48! • This is a medium sized effect only. So shouldn‘t we be confident in competing?

13. Lipsey and Wilson (1993) effect size distribution

14. Summary and conclusion • Anxiety about cost-benefit ratios of psychological interventions is a major hurdle in communicating with stakeholders. • Break-even point effect sizes can be calculated a-priori and can help in reducing that anxiety. • Comparing that break-even point effect size with those we know from meta-analysis helps in estimating the return on investment (ROI) of psychological and other interventions. • Try it, and you may be surprised how competitive in terms of ROI‘s we are in many areas.