Cost-Benefit Analysis (CBA) Illustrated by the California GAIN Random Assignment Experiment

1. Cost-Benefit Analysis (CBA)Illustrated by the California GAIN Random Assignment Experiment

2. Plan of Discussion • What is Cost-Benefit Analysis? • How is it done for social programs? • How are the costs of social programs estimated? • (If time permits) what are some alternatives to CBA?

3. What Is Cost-Benefit Analysis (CBA)? • The key objective is to determine whether the benefits of a government policy or program or project exceed its costs • How does this differ from private sector investment analysis? • A broader perspective is used—society as a whole • It is often more difficult to put values on outcomes

4. Types of CBAs • Ex ante • Ex post • In Medias Res

5. Basic Steps in CBA • Decide whose benefits and costs count • Select the portfolio of alternatives • Catalogue (potential) impacts and select measurement indicators • Determine impact quantities over the life of the project

6. Basic Steps in CBA (Continued) 5. Monetize impacts 6. Chose the social discount rate 7. Add up the benefits and costs • Perform sensitivity analysis • Recommend the alternative with the largest net social benefits

7. An Illustration of the Steps • GAIN: An innovative welfare-to-work program for AFDC clients in California during the 1980s • Evaluated through random assignment in 6 counties by MDRC • The 6 counties varied considerably in the degree to which they emphasized a training first (human capital approach) or a work first approach • Alameda County was at one end of this spectrum and Riverside County at the other; but all 6 provided structured job search and training

8. (1) Decide Whose Benefits and Costs Count • This is called the “standing” issue • Who should have standing in the GAIN CBA—the AFDC beneficiaries, the government (‘taxpayers’), or both? • Analysis performed from 3 perspectives (see handout) • AFDC Recipients • Taxpayers (or Government) • Society (AFDC Recipients + Taxpayers)

9. (2) Select the Portfolio of Alternatives • A change in policy (‘the counterfactual’) must be compared with the existing policy (‘the status quo’) • The GAIN CBA compared one alternative to the status quo in each county by randomly assigning AFDC clients to 2 groups: a program and a control group • However, findings for the 6 counties could be (and were) compared non-experimentally

10. (3) Catalogue Potential (Physical) Impacts and Select Measurement Indicators • ‘Impacts’ refers to both inputs (costs) and outputs (benefits) • Important to list all non-trivial impacts, which isn’t done in Table 7.5 of handout, which is all in terms of effects on income • Beneficial impacts examined in the GAIN CBA included reductions in transfer payments and increases in tax payments, increases in earnings and fringe benefits • Cost impacts included staff time and materials • Measurement indicators and impacts specified simultaneously

11. (4) Determine Impact Quantities Over the Life of the Project • This step usually requires predictions of how long impacts will last and how they vary over time • This is often very difficult to do accurately • EXAMPLES from the GAIN CBA • earnings • transfer payments

12. (5)Attach Money Values to All Impacts • Very important step • Must be done to make all impacts “commensurable,” thereby allowing them to be summed • Example from GAIN evaluation • Staff time • Sometimes difficult to do • Value of time spent at work experience job • Value of scenery • Value of life

13. (6) Choose the Social Discount Rate • Needed because some benefits and costs occur in the future • Use of a discount rate makes benefits and costs that occur in different time periods commensurable • It is used to compute present values • The appropriate rate to use is controversial • MDRC used 5% for the GAIN CBA

14. Formula for Net Present Value (NPV) Where Bt = benefits over time Ct = costs over time r = the social discount rate t = a time period (GAIN was discounted over 5 years, not infinity)

15. (7) Sum the Benefits and Costs • Decision rule if all benefits and costs occur in the same time period: • If net benefits = B – C > 0, adopt new program • If net benefits = B – C < 0, stay with status quo • Decision rule if benefits and/or costs occur in different time periods, as in the GAIN evaluation: • If NPV = PV(B) – PV(C) > 0, adopt new program • If NPV = PV(B) – PV(C) < 0, stay with status quo

16. Bottom Line in GAIN CBA • In Butte, Riverside, San Diego, the programs passed the cost-benefit test from all 3 perspectives • In Los Angeles, the program failed the cost-benefit test from all 3 perspectives • In Alameda and Tulare, the gains to participants were more than offset by the losses to the government • But in Tulare, it was a fairly close call (What should one make of a close call?)

17. (8) Perform Sensitivity Tests • All CBAs require assumptions • Sensitivity tests involve attempts to see if the results are robust to alternative assumptions • This reduces the risks of a faulty assumption • For example, MDRC examined whether their results changed much when alternative assumptions were made about how earnings and transfer payments change over time (they didn’t)

18. (9) Recommendations • The counter-factual option with the largest net social benefits should be adopted • But only if net social benefits are positive • In GAIN, Riverside clearly had the largest net social benefits, and they were strongly positive • Counties throughout California adopted the Riverside model

19. Cost Analysis • Involves measuring expenditures on the resources required to operate a program • Uses • Essential for CBA • Needed for planning program roll-outs • Helpful in monitoring and modifying on-going programs • Some of the concepts are again illustrated with the GAIN evaluation • However, depending on circumstances, there are a variety of ways of determining costs

20. A Few Principles • The goal in a CBA is to estimate net costs, not just gross costs (This requires separate calculations of costs for program groups and control groups.) • It is usually more useful for CBAs of people-orientated programs to measure costs per case, rather than aggregate cost • Facilitates comparisons with other programs • Facilitates comparisons with benefits • The distinctions among ex ante, ex post, and in medias res remain important

21. Data Items Needed for Cost Analysis • Time spent by clients in each program component • Payments to clients • Vendor payments on behalf of clients • Salary and other costs of employing the staff operating the program and the counterfactual program • Special purchases made for program use • Office overhead rates or the cost of each item that comprises overhead

22. Steps in Estimating Net Costs • Enumerate program components (see Table 3.5) • Compute unit costs for each component for the program and control groups • Determine participation rates for each component for the program and control groups • Determine the average length of participation in each program component for the program and control groups • Compute the gross cost of each program component for the program and control group • Compute the net cost of each program component

23. (2) Compute Unit Costs • Compute the average cost of providing a unit of each program component (e.g., an hour, a week, a session) • In GAIN, total expenditures on many components were available from administrative data • These totals were divided by the total number of “participant months” • Note: ideally, total expenditures should include all costs (i.e., staff and overhead costs, costs engendered by both participants and “no shows”) • Note: both program group members and other may have participated in some of the same activities

24. (3) Determine Participation Rates • Participation rates are the fraction of clients that participate in each program component • It is essential to derive separate participation rates for the program group and the control group • In GAIN, participation rates were derived from a combination of data collected from hard-copy case files and survey data • Combining two sources of data can improve accuracy

25. (4) Determine Average Length of Participation • The average length of time clients who participated in a program component remained active (i.e., the number of units received by component participants) • If the average length of stay differs substantially between the program and control groups they must be measured separately • Again, in GAIN, participation rates were derived from a combination of data collected from hard-copy case files and survey data

26. (5) Compute Gross Costs • This step is completely mechanical • Gross costs per client = unit costs x participation rate x average length of participation • Note: Computing the cost of a program component in this way means that the costs are averaged over those who actually participate in the component and those who do not. If this was not done, the different cost components could not be summed to determine total costs.

27. (6) Compute Net Costs • This step is also mechanical • Net costs per program participant = gross cost for the program group minus gross cost for the control group

28. Alternatives to ‘Pure’ CBA • ‘Pure’ CBA is only possible • If all the benefits and costs can be measured • If all the benefits and costs can be monetized • If this isn’t possible, an alternative to pure CBA must be used • Two possible alternatives are: • Qualitative CBA • Cost-effectiveness analysis

29. Qualitative CBA • Probably much more frequently used than pure CBA • It works as follows: • The evaluator first obtains monetary measures for all benefits and costs for which this is possible • Then, the remaining benefits and costs are assessed as well as possible • For example, if feasible, they are quantified • GAIN has a qualitative aspect; non-monetary outcomes are discussed and sometimes measured (e.g., high school graduation, implications of increased employment, substitution effects)

30. Cost-Effectiveness Analysis (CEA) • CEA can be used if all benefits and costs are quantified, but some aren’t monetized • It is an attempt to get around putting a monetary value on all project impacts • Often used in the health and educations fields, where it is difficult to put a monetary value on some impacts

31. Example of CEA: Health and Safety Programs that Save Lives • In CBA, a monetary value would be put on each life saved B/C = Value of a life saved in dollars / Cost in dollars = Dollars of benefits per dollar of expenditure • In CEA, a monetary value is not placed on lives saved C/E = Cost in dollars / Number of lives saved = Number of dollars expended per life saved

32. Limitations of CEA • Mainly useful for comparing alternative policies or programs, but doesn’t indicate whether any of the alternatives should be adopted • Comparisons of alternatives are often subject to scale problems • Difficult to use when more than one benefit is not valued in monetary terms

33. Cost-Utility Analysis • To get around the last problem in the health area, techniques have been developed for combining two effects of health policies • Doing this is called cost-utility analysis • Specifically, the effects of health care on the length of life and on the quality of life are combined into something called QALYs—Quality-Adjusted Life-Years • Idea is to allow a policy that provides people with 10 years of enjoyable life to be ranked higher than a policy that keeps people alive for 10 years as vegetables, even if the first policy costs more