Carolyn J. Heinrich LaFollette School of Public Affairs, University of Wisconsin

1. New Estimates of Public Employment and Training Program Net Impacts: A Nonexperimental Evaluation of the Workforce Investment Act Program Carolyn J. HeinrichLaFollette School of Public Affairs, University of Wisconsin Peter R. MueserUniversity of Missouri, IMPAQ International, LLC, and IZA Kenneth R. TroskeUniversity of Kentucky and IZA Kyung-Seong JeonUniversity of Missouri Daver C. KahveciogluIMPAQ International, LLC November 2009

2. 1998 Workforce Investment Act (WIA) • Largest job-training program in the U.S. • Implemented in 2000 in most states—replaced Job Training Partnership Act (JTPA) • Current annual federal budget: $3 billion • Significantly lower relative to past public expenditures • Program implementation differs by state and local area • Work-first emphasis, service sequencing, referrals, access to training

3. WIA Adult Programs • Two primary adult programs serving: • Disadvantaged workers, both unemployed and those in low-paying and unstable jobs • Dislocatedworkers who have lost jobs or are slated to be laid off • Voluntary: participants recruited by local agency staffor referred by training providers– may have specific number of slots to fill • Locally-determined eligibility standards

4. WIA Service Sequencing Core services - outreach, job search, placement aid, and labor market information Intensive services - comprehensive assessments, individual employment plans, counseling and career planning Training services – mostly occupational/vocational training, some on-the-job training Most training provided with voucher Similar services available to anyone in Employment Service offices (“Wagner-Peyser” services) Training time varies from a few months up to 2 years 4

5. Impact Analysis • Impact estimates for • Adult program • Dislocated Worker program • Incremental impact of Training services vs. Intensive/Core services for these programs • Using state administrative data from 12 states:Connecticut, Indiana, Kentucky, Maryland, Mississippi, Missouri, Minnesota, Montana, New Mexico, Tennessee, Utah, Wisconsin

7. Comparison Groups • Comparison groups: UI claimants (9 states), Wagner-Peyser (3 states) • Programs have substantial overlap • WIA recipients receive the most meaningful services UI Claim WIA Wagner-Peyser Services Both are plausible comparison groups because they contain individuals with employment problems & those seeking assistance 7

8. Measures Outcomes: Earnings & employment 16 qtrs Control variables: • Calendar quarter of program entry • Demographics: gender (exact match), age, education, race/ethnicity • Disability, veteran • Local labor market • Employment and earnings over the two years prior to program entry • Industry of most recent job • Prior program participation (WIA, UI, ES) 8

9. Overview of Analysis 9

10. Estimates • Estimates for states are combined • Weighted by number of WIA participants • WIA participants entering in PY2003 and PY2004 • Estimate of average program impact across 12 states • Effect of the treatment (WIA) on the treated • Comparison group defined by program contact in quarter of entry (UI claim or benefit, Wagner-Peyser service receipt, WIA participants who did not receive training services for training impact estimate) 10

11. Matching Procedures • Matching is within state, within gender, and (usually) within quarter of participation • Logit specification predicts propensity score • Details of variable coding differ by state; approx. 100 individual characteristics, labor market experience and prior program participation, geographic area within state • Matching is many-to-one, using a caliper, with replacement (“radius matching”) • Standard errors use formula recommended by Imbens (2008) based on a conditional variance estimate

12. Basic Data: Demographics Summary Statistics for WIA Participants and Comparison Group: 12 States WIA WIA Dislocated Comparison Adult Worker Group Sample size Unique individuals 95,580 63,515 2,929,496 Units available for matching 97,552 64,089 6,161,510 Demographic Mean Mean Mean Male 0.420 0.482 0.585 Black 0.445 0.330 0.171 Hispanic 0.031 0.022 0.064 Age 32.70 40.24 39.59 Years of education 12.27 12.55 12.42 12

13. Basic Data: Adult Program Earnings Dip in earnings is large for comparison program participants Dip in earnings is small for Adult Program participants Comparison 13

14. Impact Estimates: Adult ProgramAll Services, Females Estimates in quarters 1-3 are probably upwardly biased All Services Earnings Females Other estimates are realistic Quarterly earnings increase by $600 in 16 quarters after program entry (Mean quarterly earnings are $2000-$4000) 14

15. Impact Estimates: Adult ProgramAll Services, Males Again, quarter 1-3 estimates upwardly biased All Services Earnings Males Other estimates are realistic Quarterly earnings increase by $400 for males 15

16. Impact Estimates: Adult ProgramTraining, Females Comparison group is WIA participants who don’t enter training Training Earnings Females 16

17. Impact Estimates: Adult ProgramTraining, Males Comparison group is WIA participants who don’t enter training Training Earnings Males 17

18. Impact Estimates: Adult ProgramAll Services, High Training States All Services Earnings Females Initial increase greater 7 High- Training States 18

19. Impact Estimates: Adult Program All Services, High Training States All Services Earnings Males No initial decline 7 High- Training States 19

20. Summary of Impact Estimates: Adult Program • At face value, results imply strong immediate impact • Aggressive initial counseling (plausible?) • Selection into program may cause positive initial impact estimates • Self-selection, counselor selection of those with good prospects • Training appears to be of some value, but there may be selection bias in results • Results for states with greater investments (“high training” states) differ • No initial decline, greater growth over time in impacts

21. Basic Data: Displaced Worker Program Earnings Dip in earnings is large for Dislocated Worker Program participants Comparison 21

22. Impact Estimates: Dislocated Worker ProgramAll Services, Females All Services Earnings Females Difference-in-difference estimates is much lower WIA entrants are advantaged relative to the comparison group: Causal impact is uncertain 22

23. Impact Estimates: Dislocated Worker ProgramAll Services, Males All Services Earnings Males Difference-in-difference estimates is much lower for males, too 23

24. Impact Estimates: Dislocated Worker ProgramTraining, Females Training Earnings Females 24

25. Impact Estimates: Dislocated Worker ProgramTraining, Males Training Earnings Males 25

26. Summary of Impact Estimates: Dislocated Worker Program • Pattern of results consistent with expectations that WIA services require time to produce impacts for workers facing serious difficulties in obtaining reemployment • DW participant earnings do not reach earnings of comparable nonparticipants until more than two years after participation • Estimates also imply little incremental impact of training for DW participants • Unlikely initial costs of training could be recouped • Results for 7 High-Training states show less evidence of bias (especially for women) but impacts on earnings even after training is complete appear minimal • There does appear to be some impact on employment 26

27. Conclusion • Adult Program • We observe long-term positive impacts of the WIA program • Training also appears to be valuable • Dislocated Worker Program • Selection on stable unobserved factors may induce positive bias in impact estimates • Program long-run impacts difficult to gauge • Long-term impacts appear minimal • Training appears to have little long-run effect 27

28. Conclusion (continued) • Analysis shows both the potential benefits and the limitations of nonexperimental estimates based on administrative data • Selection clearly affects results • Patterns of estimates provide an indication of where impacts may be greatest • Specification tests are important • Comparison across control types and programs is useful 28