1 / 19

Recent Advances in Performance Measurement of Federal Workforce Development Programs

Recent Advances in Performance Measurement of Federal Workforce Development Programs. Evaluation and Performance Management of Job Training Programs

arden-foley
Download Presentation

Recent Advances in Performance Measurement of Federal Workforce Development Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recent Advances in Performance Measurement of Federal Workforce Development Programs Evaluation and Performance Management of Job Training Programs Organized by European Commission Directorate-General for Employment, Social Affairs and Equal Opportunity and University of Maryland School of Public Policy Randall W. Eberts W.E. Upjohn Institute for Employment Research November 7, 2009

  2. Purpose • Describes a Department of Labor project conducted by the Upjohn Institute that addresses some of the problems with current workforce system performance targets outlined in previous presentations: • Cream skimming • Inability to account for differences in local labor market conditions • Accounts for factors outside the control of state and local administrators • By “outside the control,” we mean factors that affect performance outcomes but are not related to the services and other assistance provided by the programs • Local labor market conditions (unemployment rates) • Personal characteristics of participants (prior work history, educational attainment, barriers to employment) • Offers a systematic, objective and transparent framework for: • Setting targets • Focusing on the value-added of WIA services • Diagnosing WIA performance • Leveling the playing field

  3. Framework • Procedure follows the basic regression-adjusted approach used to adjust JTPA performance targets • Uses estimates of the effects of unemployment rates and personal characteristics on performance outcomes, based on all WIA exiters in all 50 states • Estimates are based on the experience of individual participants within their local labor markets • Procedure allows the aggregation of performance outcomes and factors from the individual to the WIB to the State to the Nation • By using the same weights for each level of jurisdiction, the differences add up • Thus the targets are consistent across jurisdictions • Focus on three WIA programs and three common measures

  4. Procedure • Three step process: one for each jurisdictional level: national, state, WIB • Step One: Adjust the national targets for assumed changes in unemployment rates • Step Two: Use the national adjusted targets as the departure for setting state performance targets • State and national performance outcomes differ because of differences in unemployment rates and participant characteristics • Step Three: Use each state’s adjusted targets as departure for setting targets of WIBs within the state • WIB and state performance outcomes differ because of differences in unemployment rates and participant characteristics

  5. Basic Equation • Use ordinary least squares regression to relate the performance measures to individual participant characteristics and local labor market conditions as measured by local unemployment rates Yisq = bo + b1*Xisq + b2Ds + b3Dq + b4Usq + error Yisq : performance measure Xisq personal characteristics and employment history Ds : state or WIB dummy Dq : quarter dummy Usq : quarterly unemployment rate by WIB or state • Unemployment rates are entered in three ways depending on performance measure • Personal characteristics and employment history are entered as categorical variables with one of the categories omitted from the equation as the reference group

  6. Variables included in estimation

  7. Sample

  8. Significant Differences in Unemployment Rates Across States and Counties Unemployment rates among counties with total employment of more than 100,000 ranged from 1.1 to 14.9 percent from 2000 through 2008.

  9. Estimates of the Effect of Unemployment Rates on Performance Measures (Percentage change of the performance measure associated with a one percentage point change in the unemployment rate; estimates are statistically significant at the 0.001 (***), 0.01 (**) and 0.05 (*) confidence levels)

  10. Step One: National Use estimates of the effect of unemployment rates on individual participants to adjust national performance targets based on President’s 2010 Budget

  11. The adjusted targets take into account changes in the assumed unemployment rates, whereas GPRA targets remain flat during increases in unemployment rates.

  12. State targets differ from national targets: Differences in unemployment rates Differences in personal characteristics Add adjustment to the departure national target rate Step Two: State Targets

  13. WIB targets differ from state targets: Differences in unemployment rates Differences in personal characteristics Add adjustment to departure state target rate Step Three: WIB Targets

  14. Adjustments add up from WIB to State to Nation Based on differences in characteristics Weights are the same at all levels Adjustments Add Up

  15. Examples of Performance Adjustments Adult Dislocated Youth State Adjustment ee ret earnings ee ret earnings place att lit The adjusted targets, and their components, are shown for six states. It should be noted that the direction of the effect of the unemployment rate may be different for retention than for the other two performance measures since retention is estimated as the change in the unemployment. Differences in the changes in the unemployment rate between the state and the nation may be different from the differences in the levels.

  16. Means of the Adjustment Components for WIA Adult and Dislocated Workers

  17. Significant Differences in Personal Characteristics Across States and WIBs State differences in personal characteristics contribute to a difference of as much as 12 percentage points in performance outcomes and the differences have increased in recent years. Maximum Mean Minimum Note: WIA Adult Entered Employment. Participant attributes are weighted by their estimated effect on performance outcomes.

  18. Summary • Target adjustment procedure provides a systematic, transparent, and objective way to set national, state, and WIB performance targets for WIA programs • Adjustment factors, since they are related to factors that are familiar to administrators, can be easily scrutinized to better understand and diagnose programs • Also familiar since state adjustment procedure is similar to the JTPA method • National performance targets have already been adopted by USDOL and GAO • Currently exploring adjustments at the state and WIB levels

  19. Contact Information Randall Eberts W.E. Upjohn Institute for Employment Research 300 South Westnedge Ave. Kalamazoo, MI 49007 269-343-5541 eberts@upjohn.org

More Related