Differences in Using Common Metrics to Measure Business vs. Social Performance

1. Differences in Using Common Metrics to Measure Business vs. Social Performance Paul Bakker – Social Impact Squared SIAA – Canada Launch Sept. 13, 2013

2. Theories of Change Profit Bottom Line Only (Traditional Business) Social Bottom Line Logic Model Investment for Costof Production Inputs Income (Donations/Grants) Manufacturing, Marketing, Sales, etc Distribution of Needed Goods, Social Services Activities # of Goods and Services Sold (Income, Profit) Outputs # of people served Goal Positive Change in People’s Lives Goal Outcomes Assumed Positive

3. Measuring Outputs vs. Outcomes Outputs • Directly observable • Count how many people bought your products or services. Record how much they cost you to produce. Record how much people paid for them. Outcomes • Not directly observable • Hard to observe changes in ideas, attitudes, and beliefs • Often lose contact with clients • Observed changes might not be caused by you

4. How to Describe Alternative Reality (i.e. the counterfactual) • It’s all about making comparisons: • To similar groups that didn’t receive your goods/services (equivalent or nonequivalent comparison groups) • To the past (pre-post, longitudinal, time-series) • To groups that are different in known ways (cut-off score designs, regression point displacement) • To what statistics predict would have happened • To what experts think would have happened (including clients and other stakeholders)

5. The State of Social Impact Measurement There are different ways to attribute observed change to social programs, some of which are complex and expensive. The most appropriate methods depend on the context of the program. The most frequently used methods provide estimates that have a good chance of being off the mark.

6. Social Impact Common Measurement Systems • Some want to be able to assess performance of social investments like they assess the performance of financial investments. • To date, developed good activity & output metrics that are standardized and comparable across programs. • But, are challenged to incorporate outcome measurement. Examples of Social Impact Common Measurement Systems: Impact Reporting and Investment Standards Global Impact Investing Rating System Charity Navigator

7. Comparability of Outcome Metrics • Even if social outcome data was readily available, it wouldn’t be fully comparable. • Differences in attribution methods (counterfactual estimates). • Measurement tools like surveys have different degrees of error and biases can vary by cultural groups and contexts.

8. Where Do We Go From Here • Giving that outcome measures are estimates and never fully comparable, what should we do with systems design to compare social performance? • Abandon them? • Nothing, they are fine the way they are? • Improve them? How? • Let’s Discuss!

9. Recognizing Uncertainty Program 1 Program 2 SROI of 1:4 or GIIRS score of 95 Average Level of Evidence of Effective of Program Components: 2.75 • SROI of 1:16 or GIIRS score of 140 • Average Level of Evidence of Effective of Program Components: 6