V Konferencja Ewaluacyjna Warszawa, 23.10.2009

1. Jonathan D. Breul Evaluation as an instrument to improve the quality of public spending - a look at the global financial crisis V Konferencja EwaluacyjnaWarszawa, 23.10.2009

2. Evaluation as an instrument to improve the quality of public spending - a look at the global financial crisis Ministry of Regional Development & Polish Agency for Enterprise Development 5th Evaluation Conference Warsaw, Poland October 23, 2009

3. Role of evaluation with the economic crisis • Imperative to measure the impact of investments and help make adjustments. • While it may be too late to shape how governments respond to this economic crisis, we have a responsibility – at a minimum – to capture lessons that can inform responses in the future.

4. Defining success • Confusion whether the programs represent spending or investment • In US, the Recovery Act has multiple purposes: • Government spending • Tax cuts • State fiscal relief • Not only intended to quickly create jobs – but also to make longer term investments the will benefit the growth of the economy in the longer term.

5. Measurable criteria • Achieve economic stimulus: Jobs created or job losses avoided • Achieve long-term public benefits by investments in transportation, environmental protection or technology • Foster energy independence or improved educational quality • Deliver program results • Satisfy transparency and accountability requirements

6. Many challenges • Measure outputs or outcomes? • It might be impossible to measure the impact or results – since impact often times lags years behind the actual investment • Often other factors will influence any particular impact • Hugely difficult and probably impossible to make a causal connection without a counterfactual. • We don’t know what would have happened in the absence of the Recovery Act • Compound effect of Recovery Act, and other measures

7. Stimulus job creation • The models are based on historical experience • But, we are outside anything we have experienced historically. • We are completely in a world we do not understand and know. • High level of uncertainty. • White House: plan will create 3.3 million to 4.1 million jobs by late 2010 • Congressional Budget Office: estimates 1.2 million to 3.6 million additional jobs

8. Economists use a two-step process • First, they project how much the legislation will make the economy grow • Then they predict how many jobs that growth will create. • The second part is less complicated – because economists tend to rely on rules of thumb. • The White House assumed that each 1 percent increase in the economy’s size would produce 1 million jobs • The initial calculation is tougher. • To make it, economists rely on a mix of facts and assumptions. • Understandably, they come to different conclusions.

9. Council of Economic Advisors estimates • The White House method assumes that things were getting worse and that the stimulus is the sole factor responsible for stopping the bleeding. • Procedure is somewhat crude and does not take into account the obvious differences in wages and other costs across different types of projects and across different parts of the country • It does take into account key differences between tax changes or state fiscal relief, an direct government investment spending.

10. OMB reporting guidelines • Direct reporting requirements cover one-third ($271 billion) of stimulus funds. • Recipients and sub-recipients required to report how many jobs are created or saved. • Total hours of work created divided by the number of hours worked by a full-time worker, along with a brief description of the types of work involved. • This data will later be used to evaluate the quality of jobs created by stimulus funding.

11. Obama Administration program evaluation effort • October 7, 2009, the Office of Management and Budget has issued new guidance and funding to agencies that develop and perform high-priority evaluation activities • Step toward making program reviews an essential function of government • Most rigorous study designs appropriate for different programs, given size, stage of development, etc. • Focus on “impact evaluations” to determine causal effects http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf

12. Is stimulus stimulating jobs? • We may never know for sure. • But - the need to enhance public sector performance has become more urgent as governments face mounting demands on public expenditure and calls for higher quality services. • The benefits of being clearer inside and outside government about purposes and results are undeniable.

13. Contact information Jonathan D. Breul Executive Director IBM Center for The Business of Government 1301 K Street NW, 4th floor, West Tower Washington, DC 200045 (202) 515 -4382 jonathan.d.breul@us.ibm.com www.businessofgovernment.org