Changing Perspectives on Workforce System Performance

1. Changing Perspectives on Workforce System Performance Data Validation Workforce Innovations San Antonio July, 2004

2. Purpose of Today’s Session • Update states and grantees on what is new in data validation • Provide information on the findings to date from the first round of validation • Allow states and grantees to provide feedback to ETA on how to improve the DV process

3. Role of Data Validation in ETA Performance Assessment • Data validation is a key component in overall performance strategy • Program funding is being directly tied to reliable performance outcomes (performance budget integration) • Data validation required by OIG and now being reviewed by GAO • Data validation is integrated into reporting • Validation tools are evolving to meet state needs

4. Programs Included in the Data Validation Effort • Unemployment Insurance Benefits and Tax (UI) • Workforce Investment Act (WIA) • Trade Adjustment Assistance (TAA and NAFTA-TAA) • Labor Exchange • National Farmworker Jobs Program (NFJP) • Indian and Native American Programs (INA) • Senior Community Service Employment (SCSEP) • Office of Apprenticeship, Training, Employment, and Labor Services (OATELS)

5. How Does Validation Work? • Two separate processes are required to ensure that performance data is reliable • Report Validation • Data Element Validation • ETA provides software to states and grantees that analyzes participant records

6. Report Validation • Ensures that performance calculations are accurate • DV software creates an audit trail for the numerator and denominator for each performance measure • Classifying participant records into performance outcome groups enables non-technical staff to validate and analyze program outcomes

7. Data Element Validation • Report will not be accurate if the data being used by the software are wrong • Requires checking data elements against source documentation to verify compliance with federal definitions • Handbooks contain instructions and examples of acceptable source documents for each data element validated • States identify state-specific source documentation to reflect the variability of state MIS systems and state/local documentation standards

8. Reporting of Validation Results • Data validation software produces • Report validation summary • Data element validation summary and analytical reports • WIA and LX software creates files with the annual report validation values for upload to ETA

9. Validation Efforts to Date • Many states have shared their validation results • First round of validation was a valuable learning experience for all • ETA has not set standards for acceptable data quality • Standards will be set for PY 2004 data validation

10. What is New for PY 2003 Validation • New schedule for reporting and validation • New software • New policies for collection and retention of source documentation

11. Schedule for Reporting of Validation Results • WIA RV will be due October 1, 2004 when the annual report is due • WIA and TAA data element validation will be due February 1, 2005 • LX report validation will be due November 15, 2004 with the report.

12. Data Validation for National Programs • NFJP validation to begin in February • Reports due in June • Pilot of process was conducted in spring • Software will be tested further in fall • Training session scheduled for November • Data validation to be added to SCSEP in late 2005 • Indian and Native American validation will be incorporated into existing reporting software

13. Revised Software • WIA Version 3.0 to be released in mid-August • New versions TAA (version 1.3) and LX (version 1.8) validation software • All will include automated upload of DV reports to ETA

14. WIA Software Changes • Calculate performance for the new reporting periods • Calculate Table O • More complete edit checks • Ability to filter source table and performance outcome groups to provide greater analytical flexibility • Accept records for participants served only by NEGs

15. Software Upgrades for WIA Data Element Validation • Revised Data Element Validation Worksheets to reflect reduction in elements • Improved ability to identify records that have not been validated • Ability to identify sampled records that have are missing, invalid, wrong SSN, or whose location is unknown. • Ability to trace exported samples

16. States Experiences with Data Validation • States had to determine staff to be responsible for data validation • Communication of expectations and requirements to local areas • Mode of data element validation – onsite, centralized or both

17. State and Local Roles and Responsibilities • States had varying experiences in identifying validation assignments • Some states had no problem • Some states took time to sort through roles of different units • Some states still have not clarified assignments (particularly for TAA) • Organization of case files at local areas was often not standardized or adequate

18. Improving the Clarity of Source Documentation Requirements • ETA has not had clear and specific policies for collection and retention of source documentation • States need to provide clear guidance to local areas • ETA will clarify requirements in change 1 to TEGL 3-03 • Currently in clearance • To be issued in August

19. Streamlined Data Element Validation Requirements • As a result of state feedback, ETA reviewed and reduced the number of elements to be validated • All elements directly related to performance or eligibility

20. Detail for reduction in elements

21. Various Methods for Data Element Validation • Onsite validation is essential to preserve the integrity of the process • Ideal for state staff to perform validation onsite • Promotes communication and mutual understanding • In some cases, onsite validation is impractical • Distances are too great • Small number of records • States can therefore pursue a combination of onsite and remote validation if necessary

22. Findings for WIA Report Validation • States had problems in two areas • Problems with extract file imported into software • Problems with calculations

23. WIA Report Validation -- File Problems • Extract file imported into the software is incorrect • Data in extract file does not match data in state’s data system • Inconsistent data • File is different from the file used to calculate the report submitted to ETA • Missing records • Changed/Updated Data

24. WIA Report Validation – Calculation Problems • States excluded older Youth in advanced training/post-secondary school from performance, even if the youth is employed. • Failure to distinguish pre-dislocation earnings from pre-registration earnings for dislocated workers • Exclusion of records for earnings calculations due to 99,999.99.

25. Data Element Validation Findings • Significant number of errors – error rates exceeded 20% for some elements • Many errors can be explained by lack of clarity in expectations for local source documentation • Problems with changing wage records and WRIS data

26. WIA Data Element Validation Results for Adults

27. WIA Data Element Validation – Results for Dislocated Workers

28. WIA Data Element Validation – Results for Older Youth

29. WIA Data Element Validation Results for Younger Youth

30. TAA Data Element Validation – Results by the Numbers

31. Continuing Challenges • State wage record files are always changing • One solution is to “freeze” the file to avoid changes • States should track changes in order to validate wages • Confidentiality of WRIS data • WRIS has rules restricting access to information • Software allows states to suppress display of wage values

32. Future of Data Validation • Standards for acceptable error rates to be established • ETA is moving toward a consolidated reporting system • Data Validation will be integrated into the new reporting system

33. For More Information • Contact Information Traci Di Martini • 202-693-3698 • Dimartini.traci@dol.gov MPR Technical Assistance • William Borden – 609-275-2131 • Jonathan Ladinsky – 609-275-2250 • WIATA@mathematica-mpr.com • TAATA@mathematica-mpr.com • ESTA@mathematica-mpr.com • http://www.doleta.gov/Performance/reporting/tools_datavalidation.cfm

34. We Need Your Feedback • Tell us about your experiences with Data Validation • What did you learn that may help others • What improvements can be made