How Policy Evaluations and Performance Management are Used

1. How Policy Evaluations and Performance Management are Used Maureen Pirog Rudy Professor of Public and Environmental Affairs, Indiana University Affiliated Professor, Daniel J. Evans School of Public Affairs, University of Washington June, 2010

2. “What Works?” from the US perspective • Policy or Program Evaluation • Impact Evaluation Insights into why or why not a program works are handled through process evaluation.

3. Program or Policy Evaluation The focus is on outcomes, not outputs. There is a strong emphasis in the US on setting up a reasonable counterfactual via random assignment OR reliance on econometric methods to make the comparison group as close as possible to the treatment group. (e.g., IV, propensity score matching, fixed effects, Heckman 2-step, RDD)

4. Social Experimentation vsProgram Evaluation • Social Experiments: Using random assignment to set up a counterfactual makes the statistics rather simple. • Nonexperimental Evaluations: Studies relying on other nonrandom selection mechanisms are statistically complex and less reliable in their findings.

5. “What Works?” is culturally dependent • Individual improvement versus an improvement in the distribution of income. • Increasing individual earnings versus a reduction in material deprivation or social isolation.

6. “What Works?” • In the US, the emphasis is on individual-level change. • Did you get a job ( that you would not otherwise have obtained)? • Did your pay increase (beyond what it would have been in the absence of the program)? • Are your children better off (compared to what would have happened in the absence of the program)?

7. “What Works?” • In many European countries, values are more focused on the societal level changes. Receiving a stipend to participate in a training program means that the distribution of income is more equal and there is less social exclusion and material deprivation. • What were program participation and completion rates? • Were the funds properly expended on participants.

8. “Among some set of alternatives, what should we choose?” • Benefit-Cost Analysis - Monetize the streams of benefits and costs for competing projects. But this depends on good estimates of program impacts! • Program Evaluation – What program has the largest impact?

9. How can government act more effectively? • Program Evaluation • Benefit-Cost Analysis • Accountability + Transparency Performance Management

10. Program Evaluation & BCA are not Performance Management • High performance bonuses do NOT necessarily go to the best performing programs! • Program evaluations answer different questions than information obtained from performance management systems.

11. Example: Jobs Program Performance Management----high participation, on time, funds distributed properly, individuals are employed at the end of the program….looks good. Program Evaluation --- maybe everyone would have had a job even in the absence of the program.

12. Performance Management can lead to perverse behaviors People respond to the incentives offered even if they do not make sense. e.g. US CSE program ($collected/$admin cost) Incentives may ultimately hurt the worst off. e.g., No Child Left Behind us Example 1: Old incentive payment system for child support enforcement. ($ collected/$ administrative cost) Example 2: No Child Left Behind

13. How Is PM Used? • PM is implemented throughout all levels of government . • Right or wrong, it is largely accepted that it is works – people and institutions usually respond to incentives. …so be very careful of the incentives that you establish.

14. How are PE Findings Generated? • Conducted on selected programs – often when the value of the program is uncertain or controversial. e.g. welfare reform waivers in the 1990s, negative income tax, abstinence-only sex education. • Conducted by governmental offices, think tanks, consulting companies, and university researchers.

15. How are PE findings used? • Results can be used by federal or state administrators (who often contract for the evaluations). • PE results can be used to build a body of knowledge about what works in a given area. • PE results may be ignored for political considerations. (e.g., Abstinence only sex education, Edutech)

16. СПАСИБО