EVALUATING ACTIVE LABOR MARKET POLICY FOR THE DISADVANTAGED

1. EVALUATING ACTIVE LABOR MARKET POLICY FOR THE DISADVANTAGED A Review of US Experience Robert Moffitt Johns Hopkins University

2. Outline • Thumbnail review of recent US social assistance structure • Methods of evaluation used so far • Assessment of strengths and weaknesses of each • Methods that have not been often used

3. Current Social Assistance Structure • For details, see Means-Tested Transfer Programs in the U.S., ed. R. Moffitt, U of Chicago Press and NBER, 2003 • Temporary Assistance for NeedyFamilies (TANF): cash for (mostly) single mothers • Food Stamps: food coupons for all poor • Medicaid: medical care for TANF recipients and selected non-TANF recipients • Earned Income Tax Credit (EITC): earnings subsidy for all, though mostly families with children • Housing, Disabled and Elderly (SSI)

4. Real Expenditures, 1990-1996 TANF- FOOD MEDICAID* EITC HOUSING SSI AFDC STAMPS 1990 $24,758 $20,654 $84,658 $8,092 $16,922 $20,125 1996 23,677 27,344 159,357 24,088 19,877 32,065 Pct change from 1990 -4% 42% 88% 198% 17% 59% Share of -1% 7% 60% 13% 4% 10% Growth In Millions *Includes nursing home and elderly care FY 2000 expenditures: Child Care: $20,580 Job Training: $ 7,347 Child Support Enforcement: $3,255

5. Marginal Benefit Reduction Rates (=marginal tax rates) • TANF: differs by state, median = .50 • Food Stamps: .30 • Medicaid: either 0 or >1 • EITC: varies by family size; .34-.40 in phase-in range, 0 in middle range, .16-.21 in phaseout rane • Housing: .20-.30 • SSI: 0 or .50

6. Recent Reforms • TANF, 1992-1996: reduction in tax rates, work requirements, sanctions, time limits, diversion, family caps • Food Stamps: work requirements, sanctions • Medicaid: expansion of eligibility to non-TANF groups • EITC: expansion in generosity 1990+

7. Evaluation Methods Used to Date • Will focus mostly on TANF-AFDC • A little on EITC • A little on Medicaid • But not much done on Food Stamps, housing, or SSI • For more reading, see, Evaluating Welfare Reform in an Era of Transition, eds. R. Moffitt and S. Ver Ploeg, National Academy Press, 2001

8. Distinctions • When talking about evaluation, need to make several distinctions • First one is: what is the question being asked?

9. SOME QUESTIONS • Monitoring is of great evaluation and has an important role to play • But ultimately everyone wants causal analysis • Monitoring (=Descriptive, possibly Longitudinal) vs Evaluation (=causal analysis)

10. THREE EVALUATION QUESTIONS • (1) What has been the overall effect of welfare reform? I.e., of the whole “bundle” of components? • (2) What has been the effect of individual broad components of welfare reform (work requirements, time limits, etc) (i.e., what if everything else except each had been enacted) • (3) What are the effects of detailed strategies (e.g. work first vs hum cap) within components

11. DISTINCTION BETWEEN BACKWARD-LOOKING AND FORWARD-LOOKING QUESTIONS • Backward-looking: what was the effect of what has actually happened relative to the old program? • Forward-looking: what would happen if something different were changed from the current program, taken as baseline? • Both questions are of interest, but may require different evaluation methods

12. WHAT ARE THE BEST EVALUATION METHODS TO ANSWER THE QUESTIONS • Experimental (random assignment) • Nonexperimental: time series modeling, cross-area, cross-area fixed effects, eligibility-based differences in differences, matching, cohort comparisons • Answer: best method depends on the question

13. Alternative Evaluation Methodologies for Different Questions of Interest

14. OTHER EVALUATION ISSUES • Generalization • Need for microsimulation • MSM as a tool for integration and synthesis • Process analysis and qualitative analysis undervalued?

15. What questions were addressed and methods used in U.S.? • Monitoring questions that were addressed and answered: • What were the short-run outcomes of women who have left welfare post-reform? • What were the overall trends in income and poverty for single mothers over the 1980s and 1990s, pre-reform and post-reform?

16. Evaluation Questions Addressed • An evaluation question that was addressed and answered: what was the overall impact of early welfare reform on caseloads, employment, income, poverty, and other outcomes? • Pre-1996, when there was state variation: cross-state, stated fixed effects model; dummy variable in regression for reform vs no reform • Post-1996, when there was no state variation (in existence of any reform); eligibility-based DID • Compared nationwide trends in single mothers vs married women or single women; nationwide trends in low-educated single mothers vs high-educated single mothers • Problem with the latter: separating the effects of other things happening at the same time (other reforms, the economy, etc)

17. More Evaluation Questions Addressed • Impact of Detailed Strategies: • Methods: experiments, though not all directly relevant to what was actually enacted • What were the effects of mandating full time work? • What were the effects of Work First vs. Human Capital strategies? • What were the effects of significant work subsidies greater than those states have generally enacted?

18. Evaluation Questions Not Asked or Answered • Main one: effects of broad components • Do not know the effects of time limits, work requirements, sanctions, etc, either added to AFDC incrementally or subtracted from new program • Cross-state, state fixed effects: doesn’t work, too much cross-state variation, can’t isolate components • Experiments: could have been done, but weren’t • Other nonexperimental methods: have not been used; some data problems as well (states don’t collect the data)

19. Other Evaluation Questions not addressed: Detailed Strategies • What would have happened if the time limit had been 4 years instead of 5? 3 instead of 5? • What would have happened if there had been more exemptions to work requirements? Less? Stronger sanctions? Weaker ones?

20. Data • Data always an issue, but will not go over in detail; very US specific • Administrative vs Survey data • Despite its reputation for excellent data, there were (and still are) serious data limitations in evaluating welfare reform in US

21. Forward-looking questions • Have been mostly talking about backward-looking questions, with the exception of some detailed strategies • Forward-looking: some work done with experiments, detailed strategies • “What works for whom” is the goal of these analyses • Some new experiments of this type (ERA, etc)

22. Some Other Programs • EITC: evaluation mostly using eligibility-based DID • Example: difference in trends over time when EITC expanded between eligibles and ineligibles • Or between families with different numbers of children • See Hotz-Scholz (2003, Moffitt volume)

23. Medicaid • Have been expansions of coverage to new non-TANF groups • Evalulation method: cross-state, state fixed effects model

24. Other Programs (continued) • Job Training: no studies of new program (“WIA”) but both experimental and nonexperimental evaluations of old programs (JTPA, CETA, Job Corps, Supported Work, etc.) • Some matching methods have been used in the nonexperimental evaluations

25. Matching • Has not been used much in evaluation of welfare reform or its components • Much skepticism in the welfare reform community about its validity • Research results: mixed, especially in the experimental-matching comparisons • Have yet to establish a basis for determining when matching is working well and when it is not, esp in advance (i.e., at the time of evaluation design)

26. Conclusions • Much evaluation work has been conducted in the US • Much has been learned substantively • Much has been learned about strengths and weaknesses of alternative evaluation methods • But mistakes have been made and important questions have gone unanswered • Demonstrates the challenges

27. EVALUATING ACTIVE LABOR MARKET POLICY FOR THE DISADVANTAGED A Review of US Experience Robert Moffitt Johns Hopkins University