Becoming an NRS Data Detective: Enhancing Data Collection and Analysis

1. Learning to be an NRSDataDetective: The Five Sides of the NRS American Institutes for Research June-July 2006 6/7/2014 L. Condelli/M.Corley

2. Objectives (Refer to H-01) By the end of this training, participants will be able to • Identify the characteristics of good data collection procedures and database systems; • Describe NRS requirements for assessment, goal setting, and follow-up procedures; • Identify ways to motivate state and local staff to take an interest in data and become dataliterate;

3. Objectives (Cont.) (Refer to H-01) • Use data reports to highlight data quality problems and promote program improvement; • Create a suite of data reports for program quality and improvement; and • Develop a dissemination plan.

4. Broad Concepts of this Training • Effective processes and procedures for collecting data, both for the NRS and for state and local purposes • The role of understanding and motivation: the elements • How to use data for program improvement • How to become a datadetective: look for clues and identify potential problems with data human

5. Agenda • Day 1 • Welcome, introductions, objectives, agenda • Warm-up activity • Overview: The Five Sides of the NRS • Tools every data detective needs • Day 2 • Developing your suite of reports • Day 3 • Sharing your reports • Disseminating your reports • Developing your action plan for program improvement (Refer to H-02)

6. Introductions • Each member of state team introduceself (name, title, role re: NRS) • One team member name one thing state has doneto improve data qualityand thebiggestchallengeyou face re: data quality • Another team member name one policy decision state has made asa result of reviewing dataand/or one thing state plans to dofor continuous program improvement (Refer to H-03; Take 5 minutes to prepare responses)

7. Warm-up Activity:List as Many Statements as You Can… • One statement per blue Post-It Note • What is the value of using data in adult education programs?(5 minutes) • One statement perpurplePost-It Note • How can we influence/create a state and local program culture in which adult educators use data continuously, collaboratively, and effectively? (5 minutes) • Following whole group discussion, post notes to appropriate wall charts (Refer to H-04)

8. Overview of Guide Learning to be a Data Detective: The Five Sides of the NRS On the Top 10 Non-fiction Best-Seller List for 2006

9. What are the 5 Sides of the NRS? • Foundational Elements • Solid Database for Recording and Retrieving Data • Sound Data Collection Procedures and Policies • Policies and Procedures for Collecting Core Outcome Data • Assessment • Goal Setting • Follow-up Measures 5 Easy Pieces 

10. Side 1 SolidDatabase for Recording and Retrieving Data

11. Characteristics of anEffective Data System • Tracks a relevant and complete set of data based on needs you anticipate; • Provides tools for detecting missing data and for identifying potential data quality problems; • Provides data that is up-to-date and accurate.

12. Data Reports, Elements, Functions (Refer to H-05) Review the NRS-required data reports, data elements, and system functions listed on H-05. Then consider the following questions: What data system are you using? • Please complete the Data System Inventory chart on the wall for your state (by the end of the day). Does your data system… • Enable you to produce each of these reports? • Contain all the necessary data elements? • Perform all the required functions?

13. NRS Data System Reports

14. Data Elements

15. NRS Data System Functions

16. Developing an NRS Data System Refer to H-06a and b • In the process ofdesigning or developing a database? Then you may wish to use this checklist as a guide to help in writing the requirements document. • Already have a database that meets your needs? Then you may wish to use this checklist to consider potential adjustments to your database or to congratulate yourselves that your system is solid and contains all required features.

17. Side 2 Data Collection Procedures andPolicies

18. Good Data Collection… • A series of regimented procedures and policies that people must perform routinely and with little error. • So what’s the problem here?

19. The Data Equation Data = Procedures + People with many opportunities for error

20. Simplified View of Data Flow Federal Level State Level Program Database Clerical staff Teachers Students

22. 4 Keys to the Success of a Good Data Collection System • Many people working together as a team • Each person has specific role and ongoingtraining • Different levels of staff review data, look for clues, and decipher them to identify problems

23. 4 Keys to the Success of a Good Data Collection System • Standardization of definitions, forms, and coding categories tied to the database to ensure that all members of the team operate from a common understanding • There are various checkpoints and feedbackloops for correcting errors and providing missing information • Constantmonitoring and adjustment

24. Do You have Each of These Essential Elements in Place? (Refer to H-07) • Staff knowledge and training • Standard forms and definitions • Error checking • Data entry If not, what’s missing?

25. Questions for Consideration (Refer to H-08a and H-08b) • How good is your data collection system? • Do you have total confidence in the quality of your data? Why/Why not? • Where are the points along your data flow process at which error can be introduced? At the state level? At the local level? • Who reviews data along each step of the data collection and reporting process? At the state level? At the local level? • How can you improve your data collection processes and system? At the state level? At the local level?

26. Side 3 Assessment Policies and Procedures

27. Policies and Procedures for Collecting Core Outcome Measures Assessment • Select tests that • Are standardized; • Have different but equivalent pre- and posttest forms; • That provide formative and summative information • Evaluate overall performance at various levels (e.g., class, program, state). • Determine students’ educational gain and level advancement. • Administer tests within the appropriate timeframe • Between program entrance and pre-test ; • Between pre- and posttest.

28. Side 4 Goal Setting Policies and Procedures

29. Policies and Guidelines for LearnerGoal Setting • Four outcome (follow-up) measures are goal-dependent: • Receive a secondary credential • Enter postsecondary education • Enter employment • Retain employment • Have clear, documented procedures for helping learners to set realistic goals, both short-term and long-term SMART goals = Specific, Measurable, Attainable, Reasonable, Time-limited • Help learners revisit and revise goals, as needed

30. Side 5 Follow-up Policies and Procedures

31. Policies and Procedures for Collecting Outcome (Follow-up) Measures • Database must have ability to identify students who exited program and had one of the following goals: • Obtaining a job; • Retaining current job; • Obtaining a secondary diploma or passing the GED Tests; • Entering postsecondary education or training. • Collect data either through data matching or by conducting student survey. • Identify Students forfollow-up • Process for identifying & contacting students from database • Policy for sampling procedures for survey, if appropriate

32. Procedures for Follow-up Survey and Data Matching • Collect data—Survey • Survey conducted at proper time • Uniform survey instrument used statewide • Staff trained to conduct the survey • Resources available to conduct survey • Procedures to improve response rates • If sampling is used, use randomization procedure to draw the sample. • Collect data—DataMatching • Data matching requires 3 pieces of student info: • SSN,student goal, and exit quarterfor employment outcomes • Data in proper format for matching to external database • Manage and report follow-up data • State database and procedures for reporting results. • Data archived for multi-year reporting.

33. What is Data Literacy? The ability to • Examine multiple measures and multiple layers of data, • Draw sound inferences, • Engage in reflective dialogue, and • Design program improvement and evaluation strategies

34. What are Some Reasons for Staff Resistance to Data? • Lack of Proper Training • Lack of Time • Feast or Famine • Fear of Evaluation • Fear of Exposure • Confusing a Technical Problem (Lack of Know-how) with a Cultural Problem (Lack of Data-use Culture!) Source: Holcomb, E. (1999). Getting Excited about Data. Thousand Oaks, CA: Corwin Press.

35. The Best Data Collection Procedures… …by themselves are not enough. Your approach to data collection may also empower and motivate program administrators and teachers.

36. Data Use in the Classroom For example, teachers may use the data to… • Check their implementation • Learn more about their students • Learn more about their teaching • Use that learning to be a better teacher!

37. Six Psychological Motivators* *Pane, N. (2004).  The Data Whisperer: Strategies for Motivating Raw Data Providers.  In A. R. Roberts., & K. R. Yeager (Eds), Evidence-based Practice manual. Oxford University Press.

38. Motivator Examples • Compete: How do I compare? • Rank • Anonymous comparisons • Reward: Can I make it to the top? • Reward top 1-5% • Learn: What am I doing well and what might I do better? • Benchmarks linked to resources

40. A B C YOU H D E F G Even anonymous comparisons can make a point…

41. Motivator Examples • Compete: How do I compare? • Rank • Anonymous comparisons • Reward: Can I make it to the top? • Reward top 1-5% • Learn: What am I doing well and what might I do better? • Benchmarks linked to resources

42. Teachers of the Year! • Teachers who had the most GEDs! • Teachers who had the largest student gains! • Teachers who had the best retention!

43. Motivator Examples • Compete: How do I compare? • Rank • Anonymous comparisons • Reward: Can I make it to the top? • Reward top 1-5% • Learn: What am I doing well and what might I do better? • Benchmarks linked to resources

45. How Can You Use These Motivators? (Refer to H-09) How can you give teachers avoiceand a lens for looking at data? Are your state and local staff membersData Literate? It is only with teachers as change agents that we will begin to see real improvement…

46. Motivating Staff and Teachers: Building Data Literacy (Refer to H-09) • In your state team,brainstorm strategiesyou might employ tomotivatelocal program staff to become data literate and to use their data. • Take 10 minutes. List your ideas on H-09. • Select one team member torecordand one to be prepared toreportyour ideas to the whole group.

47. Put Your Data Fears on the Table Whatconcernsyou most about using data to make policy decisions? • Afraid you don’tunderstandthe data? • Afraid your questions willsound silly? • Afraid thetruthabout your data will make your program look bad? • Afraid people will take dataout of contextfor their own agendas? • Afraid your data might not bevalid and reliable? • Other?

48. Don’t Get Stuck in a Data Swamp… DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA DATA

49. Data are Merely Numbers To turn data into information, we must first: • Organizethe data • Describethe data • Interpretthe data

50. Businesses don’t keep data that’s useless, that doesn’t inform them of anything; yet, in education, we have data that justrunsall over us. We have totargetit and organize it in such a way that itinformsus so that we can makebetter decisions. -David Benson, Superintendent Blue Valley (KS) School District