An Administrator’s Guide to Student Follow-Up and Data Quality

1. An Administrator’s Guide to Student Follow-Up and Data Quality Presented by: Dean Smith Lennox McLendon Kathi Polis

2. Training Objectives • To review NRS guidelines for determining what participant populations need to participate in follow-up surveys • To outline the seven-step process for conducting NRS-approved participant surveys • To examine strategies for training telephone interviewers McLendon and Polis

3. Training Objectives • To develop a local plan for conducting NRS-compliant follow up • To review and provide input into the development of data quality standards for local programs • To discuss and clarify MAERS-related data collection and reporting questions and issues McLendon and Polis

4. Introduction to Student Follow-Up • Requirements established by National Reporting System • Defines: • Outcomes you must collect • Methodologies for collecting them • State reporting requirements McLendon and Polis

5. Core Measures • Educational Gains – completion of EFLs • Follow-Up Measures • Entered employment • Retained/improved employment • Earned a secondary school diploma or GED • Entered postsecondary education or job training McLendon and Polis

6. Two Follow-Up Methodologies • Data Match (recommended) • Most cost and labor effective method • Student Survey • Michigan • 2005 - 2006 = student survey • 2006 – 2007 = exploring data match for employment and GED measures McLendon and Polis

7. What Participants to Survey • Only participants with follow-up goals • Enter, retain, improve employment • Enroll in postsecondary • Earn a GED/diploma • High school diploma completion can be verified through program records. • GED can be verified through local GED Examiner (with participant’s permission granted on Follow-Up Notice) McLendon and Polis

8. When to Survey for Program Year 2005 – 2006 • Postsecondary, HS Diploma, GED • Anytime prior to October 25, 2006 (for 2005- 2006 year) • Tips • Postsecondary/job training – know the college’s enrollment schedule to determine most appropriate time (e.g., mid-September for fall semester college enrollments) • GED – allow as much time as possible for participants to complete test and receive their test scores (e.g., mid-September for previous program year) McLendon and Polis

9. When to Survey – The Plot Thickens! • Participants with entered employment follow-up goals • Timing is the challenge! • Must determine TWO outcomes • If they got a job by the 1st quarter after exit quarter AND • If they were still employed by the 3rd quarter after exit McLendon and Polis

10. When to Survey for 2005 - 2006 McLendon and Polis

11. Activity 1 – Match Game • Let’s see how well you do at matching the right information together. • Refer to Activity 1 Handout. McLendon and Polis

12. The Survey Process Getting Ready: • Select and train telephone interviewers. • Determine your telephone survey schedule. Participant Intake: • Inform participants of the survey. • Ensure accurate contact information. Data Collection: • Identify survey respondents. • Conduct the telephone survey. • Record the results. McLendon and Polis

13. Getting Ready: Step 1: Select and train telephone interviewers. • Selecting telephone interviewers • May be you, your teachers, volunteers, other staff • You need folks who: • Speak clearly • Understand the program and reasons for follow-up • Understand the adult learner population • Are polite and courteous • Are patient yet persistent AND • Who WANT to be telephone interviewers! McLendon and Polis

14. Getting Ready • Training telephone interviewers • Training should include • How to use the script, including what to say to participants to introduce the survey and get their cooperation • Ways to avoid refusals • How to ask the survey questions • How to record responses • How to answer participant questions about the survey. McLendon and Polis

15. Getting Ready • Training telephone interviewers • Go over every question to ensure interviewers understand: • the purpose, • what is being asked, and • what responses are desired. • Include mock interviews and other practice. • Manual – Appendix C: Training Strategies for Telephone Interviewers McLendon and Polis

16. Program Survey • Appendices D1 – D6 • Standardized script for each follow-up measure/goal • Must follow script precisely • Three sections • 1 – includes general participant information • 2 – relates to follow-up goal • 3 – requests participant feedback and recommendations McLendon and Polis

17. Activity 2: Can We Talk? • Divide into pairs. One person will be the student; the other will play the role of interviewer. • For your designated follow-up goal, conduct the telephone interview. • What was the result? • Did you see any problems you might encounter? • If so, what were they and how might you handle them? McLendon and Polis

18. Step 2: Determine your telephone schedule. • Employment goals – quarterly • Don’t forget • Participants with “enter employment” goals will need to be contacted twice (first for employment, second for job retention) • Postsecondary and GED goals • Depends on college schedules, GED test dates, etc. – You be the judge. • All follow-up must be entered into MAERS no later than October 25! McLendon and Polis

19. Step 3: Inform participants of the survey. • At intake • Follow-Up Notice – Appendix A • Explains follow-up process • Requests their participation • Provides the participant’s permission to be contacted for follow-up • Provides permission to verify GED completion with state or local GED testing examiner • Given to all participants with follow-up goals (or potential to convert to follow-up goals) McLendon and Polis

20. Step 4: Ensure accurate contact information. • Important to collect alternative contact information • Addresses and phone numbers of friends, relatives, etc. • Use the Follow-Up Contact Information Form – Appendix B • Can input into MAERS’ Update Alternative Contact screen (Appendix E) • Encourage participants to keep their contact information up-to-date McLendon and Polis

21. Step 5: Identify survey respondents. • CRITICAL – Input participant data into MAERS at least quarterly! • Enables MAERS to generate a Follow-Up Contact List • Lists all participants that need to be contacted and when. McLendon and Polis

22. Activity 3: Beginning Your Follow-Up Plan • Refer to the Student Follow Up Plan • Review the questions for Steps 1 – 5. • Complete the “Response Plan” column for any items that you can answer right now. McLendon and Polis

23. Step 6: Conduct the telephone survey. • Ask for the individual by first name or nickname. • If teacher is not making the call, have follow-up staff state they are calling on behalf of the participant’s teacher. • Must use the Contact Log (Appendix F). • Must attempt minimum of four contacts. • No answer after four attempts – mark as a non-respondent McLendon and Polis

24. Step 7: Record the results. • Record follow-up results in MAER’s Follow-Up Information screen (at least quarterly). • File program survey in participant’s paper file. • Non-respondents – file blank survey • Adapt the Participant Follow-Up Checklist (Appendix G) for your use. McLendon and Polis

25. Activity 4: Completing Your Follow Up Plan • Refer to the Student Follow Up Plan. • Complete the “Response Plan” column for Steps 6 and 7. McLendon and Polis

26. Data Quality Standards • Improving the accuracy of our data • U.S. Department of Education developed data quality standards that states must address annually • A guide to the policies, processes, and materials that need to be in place for quality data • Data foundation and structure • Data collection and verification • Data analysis and reporting • Staff development • Levels of performance range from acceptable to exemplary McLendon and Polis

27. Data Quality Standards • Michigan’s Draft Data Quality Standards • Adapted the national data quality standards to fit local programs • Draft form only – want your input • Recommendations will be forwarded to the taskforces for further review and submission to State Office McLendon and Polis

28. Data Quality Standards • Michigan’s Draft Data Quality Standards • “Acceptable” level standards will go into effect July 1, 2006. • “Superior” level – July 1, 2007 • “Exemplary” level – July 1, 2008 • Training and technical assistance will be made available. McLendon and Polis

29. Data Quality Standards • Format • Three columns • 1st column – states standard or process • 2nd column – provides examples of sample verification • 3rd column – requires “yes” or “no” response • A few questions require brief narrative response. McLendon and Polis

30. Data Quality Standards • Process • Program director submits completed checklist and signed certification page to State Office by September 30, 2007 (for previous program year). • Written improvement plan for unmet standards • Describe new policies or procedures you will put in place to meet the standards, • Identify barriers to moving to a higher quality level, and • Describe the technical assistance needed to implement the plan. • DLEG will provide customized technical assistance to help you. McLendon and Polis

31. Activity 5: Getting Your Input • Group 1 tables will review A and C. • Group 2 tables will review B and D. • Make a list of any concerns, questions, or recommendations for the standards or verification samples. McLendon and Polis

32. Dean’s Corner • Let the games begin!!!! McLendon and Polis

33. Always willing to help… • Dean Smith • SmithD20@michigan.gov • Lennox McLendon • lmclendon@naepdc.org • Kathi Polis • polis123@adelphia.net McLendon and Polis

34. This project was developed by National Human Resources Development, Inc. (NHRD) and the National Adult Education Professional Development Consortium in cooperation with the Michigan Department of Labor and Economic Growth and funded through a grant under Section 222(a)(2) State Leadership Activities of the Adult Education and Family Literacy Act, Title II of the Workforce Investment Act of 1998, amended. For more information visit: http:www.maepd.org McLendon and Polis