Collecting and Reporting Follow-up Data Or What’s Behind Table 5?

Collecting and Reporting Follow-up Data Or What’s Behind Table 5? American Institutes for Research February 2005

Table 5: Outcomes • Earned a High School Diploma or GED • Entered post-secondary School • Entered Employment • Retained Employment

Table 5: Outcomes • States submit six (6) data points for each of the four outcomes: 1 2 3 4 5 6

Table 5: Column (B) All participants who had the outcome as a goal and who exited during the program year.

Table 5: Column (C) The number of participants who were surveyed to assess whether they had achieved the outcome. Note: C should be equal to B unless sampling was used, in which case C would be less than B. For data matching C = [BLANK]

Table 5: Column (D) Of those surveyed or data matched, the number who responded or were used for data match. Note: D should be equal to or less than B and C each.

Table 5: Column (E) Percent of respondents or participants successfully matched; column D divided by column B. E = D/B * 100 Should be > 50%, but likely not 100%

Table 5: Column (F) Number of participants achieving outcome. Note: F must be equal to or less than D.

Table 5: Column (G) Percent achieving outcome. G= F/D * 100

Table 5: Example

Real Data – PY 2002–03 Examples What type of follow-up was used?

Real Data – PY 2002–03 Examples Why is this number less than column B?

Real Data – PY 2002–03 Examples What data should be reported here?

Real Data – PY 2002–03 Examples Is this percent computed correctly?

Exercise Questions For states 2–6: • What method of follow-up did each state use? • Which states, if any, filled in the table correctly? Which did not? • Do you see any errors in the table? • Do you have confidence in these data? Why or why not?

Exercise Example

Top Errors for Table 5 • Missing Data • Put “0” for # surveyed in Column (C) • Put “0” for # responded or matched in Column (D) • Unreasonably high response rates • Many 100% Unreasonably high data matching • Many 100% (DOL goal for matching = 90%) Unreasonable matching of # who had goal to # who responded • Column (D) = Column (B)

Top Errors for Table 5 (cont’d.) • Incorrectly computed percentages: • Response rates calculated as D/C instead of D/B (B and C should be the same unless using random sampling, however) • Number achieving outcome computed as F/B instead of F/D

2002–03 Data Collection Methods

Data Matching Methodology • Produces better quality data: • More student coverage • Better data validity • Better response rate • Matches Title I—better for OMB common measures • 28 states use it for employment measures—we need more states! • What are the reasons for not data matching? • How can we promote use of this methodology?

What’s Behind the Data? • State responsibility to ensure accuracy and quality of follow-up measures • Requires monitoring of local program procedures • Key steps: • Identify students—goal setting and tracking • Uniform data collection • Training • Reporting

Percent of Students with Goals * 38% Unemployed (Table 6) ** 18% Enrolled in ASE

Quality Control: Surveys • Identify students • Set student goals appropriately • By exit quarter for employment measures • Data Collection • Quarterly collection for employment • State survey instrument • Sufficient resources (staff, time, funds) • Sampling method (if applicable)

Quality Control: Surveys (cont’d.) • Training • Trained staff on survey procedures • Improving response rate • Reporting • Quarterly reporting • Database to link outcomes to programs and students

Quality Control: Data Matching • Procedure for collecting and validating Social Security numbers • Report all with goal, prior to eliminating invalid or missing numbers (Table 5, Col. D) • Data have exit quarter and employment records matched after correct exit quarter • Only students with goal matched and/or reported—should not have 100% with outcome

Discussion: Issues and Strategies • Improving reporting and collecting • Resources • Training on Monitoring

Collecting and Reporting Follow-up Data Or What’s Behind Table 5?