1 / 51

Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training

Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training Richard West Social Policy Research Associates. Topics to Be Covered. WIA Performance Measures Definitions and calculations Challenges Working with wage records Supplemental data

lance-oconnor
Download Presentation

Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training Richard West Social Policy Research Associates

  2. Topics to Be Covered • WIA Performance Measures • Definitions and calculations • Challenges • Working with wage records • Supplemental data • When outcomes occur and are measured • Understanding performance reports • Implementing new data collection processes • What about those JTPA exiters included in WIA performance measures?

  3. 17 Performance Measures • Separate measures for funding streams and youth age groups • Adults • Dislocated workers • Youth under 19 at registration • Youth 19 to 21 (under 22) at registration. • Individual served with multiple funding streams included in all relevant measures. • (Nearly) Common measures for adults, dislocated workers, and older youth.

  4. 3x4=12 • For adults, dislocated workers, older youth: • Entered employment rate • Employment retention rate • Earnings change (Adults, Older Youth) orearnings replacement rate (DWs) • Credential attainment • For younger youth: • Goal attainment • Diploma attainment • Retention • Customer satisfaction • Participants • Employers 3 2 17

  5. Performance Measures Keyed to the Exit Quarter • Exit quarter is the quarter containing the exit date. • Individuals exit if: • Formally withdraw from or complete services OR • There is no service for 90 days (WIA or tracked partner-funded services) • Exit date is the date of withdrawal or completion or the date of the last service.

  6. WIA Exit End of 90-day period Last Service WIA Participation No services WIA Exit Date

  7. Time Periods Used for the Measures • Measures that use "real time" data. • Exiters who leave during the program year. • Applies to measures that do not use UI wage records: • Younger youth goal attainment • Younger youth diploma • Customer satisfaction • Measures that use UI wage records. • Exiters during the last 3 quarters of the previous PY and the first quarter of the current PY. • October 1 to September 30 • Applies to all other measures

  8. 1 2 3 4 1 2 3 4 Exit Periods for Performance Previous PY Current PY Real Time Measures Wage Record Measures EE 2Q Lag

  9. Challenge:Working with Wage Records

  10. Sources of Employment Data • Wage records the primary source for entered employment and retention; • Wage records do not include: • Individuals in uncovered employment, such as self-employment • Federal government employees and others outside the state UI system • May use out-of-state wage records • Bilateral agreements with nearby states • Wage Record Interchange System (WRIS)

  11. When Are Wage Records Good Enough? • Measures sensitive to incomplete wage records • Example: • Wage records 90% complete • “True” employment is 70% • Then measured employment is: 0.9 x 70 = 63% A seven percentage point bias • Really want records close to 99% complete • Typically a one or two quarter lag before wage records are good enough

  12. WRIS • Enables states to obtain out-of-state wage records efficiently and cost-effectively. • WRIS is currently fully operational with 20 pilot states • By July 1, 2001 all states will be able to join. • Expected to be self supporting • Cost of participation has not been determined. • An initial pricing study is available on • WorkforceATM.org, as is other more information on WRIS

  13. How Does WRIS Work? • State submits a list of SSNs for matching • WRIS matches those SSNs against an index to identify the states with records for those SSNs. • Wage request is sent by WRIS to only those states with records for an SSN. • Receiving states send wage records back to WRIS. • Requesting states can download the information from WRIS.

  14. Challenge:Developing a Realistic Process for Collecting Supplemental Data

  15. Sources of Employment Data 2 • Supplemental data sources may be used if not person not found in wage records. • Case management, follow-up services, participant surveys. • Records sharing or automated record matching with other administrative databases. • Note: records that provide information on quarterly earnings will be considered as wage records, • Must be collected within 30 days after individual determined to not be in wage records

  16. Supplemental Data “By the Book” • State gets wage record data • Local area is told who is not in wage records • Local area collects supplemental data within 30 days • Supplemental data is reported to state

  17. SD 30 days to get supplemental SD Data Collection Timing for Employment Quarters After Exit 1 2 3 4 5 6 7 8 9 Get Wage Records Exit EE 2Q Lag RTN 2Q Lag Follow-up Services Can get supplemental early during follow-up services

  18. “Real World” Supplemental Data • Local areas collect supplemental data during follow-up services • For everyone • For those likely to not be in wage records: • Self-employed, other noncovered jobs • Out of state employment • State gets wage record data and tells local areas who is not in wage records • Use supplemental for those not in wage records • Supplemental data reported to state

  19. Calculating Performance Measures

  20. Exclusions from all performance measures: Adults & dislocated workers who received only self-service and informational core services Institutionalized at exit (Item 622 = 1) Health/medical exits (Item 622 = 2) Deceased (Item 622 = 3) National Emergency Grants only Rapid Response only(except additional assistance) Some statewide programs only Global Exclusions

  21. Very specialized statewide programs with their own specific goals, for example: Incumbent worker training authorized under Section 134(a)(3)(A)(iv)(I), or A project for chemically dependent TANF recipients. These statewide programs are also excluded from WIASRD reporting (if they are the only program) What Statewide Programs Can Be Excluded?

  22. In What Performance Groups do Statewide Participants Count? Youth Youth Statewide? Item 307 = 1 YES Adult or Youth Displaced Homemaker Item 308 = 1 YES Excluded Incumbent Worker Item 309 = 1 only YES Adult or Dislocated Worker Other Statewide Item 310 = 1 YES Excluded Rapid Response only Item 311 = 1 YES Dislocated Worker Rapid Response Additional Assistance Item 312 = 1 YES

  23. Ambiguous Cases--Suggested Approach NO Other Statewide Item 310 = 1 Displaced Homemaker Item 308 = 1 • If already in a performance group, ignore these statewide programs YES YES NO Age < 18? Age >= 22? Has Dislocation Date? Item 125or also in NEG grant Item 313a NO YES YES YES NO Youth Services? YES NO Dislocated Worker Adult Youth

  24. Number Employed in Quarter after Exit divided by Number of Exiters Entered Employment Rate • Wage records the primary data source • Supplemental data may be used for persons not found in wage records • Exclude adults, older youth employed at registration • Exclude older youth in advanced training/ postsecondary education (but not employed) in quarter after exit.

  25. NO Entered Employment RateAdults and Dislocated Workers Dislocated Worker Measure? NO YES Excluded Employed at Registration? Item 115 = 1 YES NO Numerator Employed in Quarter After Exit ?Item 601 = 1 YES YES Denominator

  26. YES Entered Employment Rate Older Youth Employed at Registration? Item 115 = 1 Excluded YES NO Numerator Employed in Quarter After Exit Item 601 = 1 YES YES NO In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 Denominator NO Excluded

  27. Number Employed in 3rd Quarter after Exit divided by Number Employed in Quarter after Exit Employment Retention Of those employed in the quarter after exit: • Based on those employed in quarter after exit • Wage records the primary data source • Can use supplemental data if not in wage records. • Excludes older youth in advanced training/ postsecondary education (but not employed) in 3rd quarter after exit.

  28. NO Employment Retention RateAdults and Dislocated Workers Employed in Quarter After Exit Item 601 = 1 Excluded NO YES Numerator Employed in 3rd Quarter After Exit Item 608 = 1 YES YES Denominator

  29. YES Employment Retention RateOlder Youth Employed Quarter After Exit? Item 601 = 1 Excluded NO YES Numerator Employed 3rd Quarter After Exit Item 608 =1 YES YES NO In Advanced Training or Postsecondary Education in 3rd Quarter After Exit Item 624 = 1 Denominator NO Excluded

  30. Earnings ChangeAdults and Older Youth Of those employed in the quarter after exit: Earnings in 2nd and 3rd Quarters after Exit Earnings in 2nd and 3rd Quarters before Registration • Wage records only • Exclude if supplemental data used to determine employment: • in either the 1st or 3rd quarter after exit. • Exclude if preprogram quarters too far back to get data • Exclude older youth in advanced training/ postsec. education (but not employed) in 3rd quarter after exit. minus divided by Number employed in quarter after exit

  31. Earnings in 2nd and 3rd Quarters after Exit divided by Earnings in 2nd and 3rd Quarters before Dislocation Earnings ReplacementDislocated Workers Of those employed in the quarter after exit: • Wage records only • Exclude if supplemental data used to determine employment: • in either the 1st or 3rd quarter after exit. • Exclude if preprogram quarters too far back to get data • Use 2nd and 3rd quarters before registration if no dislocation date or dislocation date after registration date • Add earnings over all included exiters before dividing

  32. YES YES YES Earnings Change--AdultsEarnings Replacement--DWs Employed in Quarter After Exit Item 601 = 1 Excluded NO YES Supplemental Data Used Q1 Item 602 = 1 or 2 NO Supplemental Data Used Q3 Item 609 = 1 or 2 Numerator and Denominator NO

  33. YES YES Earnings Change Older Youth Excluded Employed Quarter After Exit? Item 601 = 1 NO YES YES Supplemental Data Used Q1 Item 602 =1 or 2 YES NO Supplemental Data Used Q3 Item 609 =1 or 2 Employed 3rd Quarter After Exit Item 608 =1 Yes NO NO Included In Advanced Training or Postsecondary Education in 3rd Quarter After Exit Item 624 = 1 NO

  34. Credential Rate • Flexible definition of credential • Standard credentials • Certificates that recognize the successful completion of training services …. • Credential attainment by end of 3rd qtr after exit • Positive outcome is attain credential and: • Enter employment (adults, dislocated workers) • Enter employment or in advanced training/postsec. educ. in quarter after exit (older youth) • Includes: • Adults/DWs who received training. • All older youth

  35. Number who enter employment and attain credential divided by Number of exiters Number who enter employment and or are in advanced training or postsecondary education in the quarter after exit divided by Number of exiters Credential--Adults and DWs Of those who receive training: Credential--Older Youth

  36. YES Employment and Credential RateAdults and Dislocated Workers Received Training? Item 333 is a valid date Excluded NO YES Employed in Quarter After Exit Item 601 =1 NO Denominator YES NO Attained a credentialItem 621 = 1, 2 ,3, 4, 5, or 6 YES Numerator

  37. YES YES NO NO Credential Rate--Older Youth Employed in Quarter After Exit Item 601 =1 Attained a credential Item 621 = 1, 2 ,3, 4, 5, or 6 NO In Advanced Training or Postsecondary Education in Quarter After Exit Item 623 = 1 YES Denominator YES Numerator

  38. Younger Youth Goal Attainment • Outcome attained during participation • 3 goal areas: • Basic skills • A basic skills goal must be set if the youth is deficient in basic skills • Work readiness skills • Occupational skills • Flexibility in defining goals • Youth has a year to attain the goal • Maximum of 3 goals set per year (for performance) • First year begins on registration date and ends a year later. • Goals used for performance must be identified up front

  39. Number of goals attaineddivided byNumber of goals attained or reached anniversary/exit date without attaining Younger Youth Goal Attainment-2 • Add numerator and denominator over all youth before dividing. • Include all in-school youth and out-of-school youth assessed as needing basic skills, work readiness skills, or occupational skills

  40. Number of exiters who attained diplomadivided by Number of exiters not still in school at exit HS Diploma/GED Attainment Of youth who were not HS grads at registration: • Diploma can be attained during participation or by end of quarter after exit • Exclude youth still in secondary school at exit

  41. YES Younger Youth Diploma Rate HS grad at registration? Item 123 > 11; Item 130 = 2 or 4 Excluded YES NO Numerator Attained Diploma? Item 673=1 or 2 YES YES NO In Secondary School at Exit? Item 673 = 3 Denominator NO Excluded

  42. Younger Youth Retention • In one or more qualifying outcomes in 3rd quarter after exit • Postsecondary education (Item 676 = 1) • Advanced training (Item 676 = 2) • Military service (Item 676 = 3) • Qualified apprenticeship (Item 675 = 4) • Unsubsidized employment (Item 676 = 5) • Exclude youth in secondary school at exit • Not related to outcome in quarter after exit

  43. NO Younger Youth Retention Rate In Secondary School at Exit? Item 673 = 3 Excluded YES NO Retained in 3rd Quarter After Exit Item 676=1,2,3,4, or 5 Numerator YES YES Denominator

  44. ChallengeUnderstanding When Outcomes Occur and When they Are Measured

  45. When Outcomes Occur PY 2000 PY 2001 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Exit  Entered Employ. EE   Retention RTN 12 Mo. Retention 12 Mo  Credential  When outcome occurs Youth Diploma  =When Measured

  46. ChallengeUnderstanding Performance Reports What you get when you ask for “the most recent data available”

  47. 1 2 3 4 1 2 3 4 Previous PY Current PY Real Time Measures Exiters Cust Sat Yth Diploma Yth Goals Wage Record Measures EE Look at Data Available May 15 RTN Earnings Credential

  48. Challenge:Implementing New Data Collection Processes

  49. Outcome Data Collection Activities • Wage records • Collect quarterly (both in and out of state) • Supplemental employment data • Collect during follow-up services • Collect quarterly after not found in wage records if necessary • Credential • Collect during participation • Update after the 3rd quarter after exit • Diploma • Collect during participation • Update after the quarter after exit

  50. Challenge:What about those JTPA Exiters Included in WIA Performance Measures?

More Related