How are Researchers Using Data from State Longitudinal Systems?

1. How are Researchers Using Data from State Longitudinal Systems? Sean W. Mulvenon, Ph.D. Professor of Educational Statistics Billingsley Chair for Educational Research and Policy Studies University of Arkansas

2. Background • Ph.D. Arizona State (1993) • Power estimation in repeated measures designs … growth models • Professor of Educational Statistics, University of Arkansas • Spent 31 months as Senior Advisor • Office of the Deputy Secretary • U.S. Department of Education – Growth Models • Internal report on growth models

3. “We need to use value-added analysis!” (Teacher, 2005) • Why? • What are value-added analyses? • What type of data do you have? • Does everyone agree? • What are you trying to do?

4. Review of Literature using Longitudinal Data Systems • Interesting, but problematic in most cases • Great ideas! • Problematic due to lack of understanding of the actual longitudinal data structure • What can you do with the data? • Incongruence in reporting and models • Analysis and models correct, but too complicated for extension to professional development for teachers [Note: 50% of values from studies that are recomputed are shown to be incorrect in multiple regression class]

5. Use of Longitudinal Data Systems for Research • What are you trying to do? • Identify research questions and objectives • Develop appropriate data sets • Select the appropriate analyses

6. Goals of Presentation • What are Longitudinal Data Systems (LDS) • Implications for LDS with School Improvement and Policy associated with NCLB • Evaluate Use of Growth Models with LDS • Strengths • Weaknesses • Limitations • Challenges • Expand research capacity to use growth models in school

7. Longitudinal Data Systems • Issues that must be addressed: • Matching • Merging • Functionality of data systems • Data quality

8. Merging Data Sets • What data are you merging? For what purpose? • What do you expect to happen? • Traditionally, data are merged on one variable • All matches considered successful matches • Different models • Probabilistic neural net (probabilities) • “Bashing” (Just merge) • Multiple merging variables • SQL joins

9. Data Merging • What to expect? (Fantasyland Model) • A state system has 1,300,000 students K – 12 for two consecutive years and approximately 100,000 students per grade. • Growth Model for Grades 3 – 8 • A total of six grades in growth model or 600,000 possible students? No! • Grade 3 new in 2nd year • Grade 8 exited from previous year • Only 500,000 students expected in growth model! • Can create confusion in system, i.e., 99.1% match rate, but only 495,500 students in model from 1.3 million

10. Data Merging for Growth Models • Data Merging should go beyond match rate to consider horizontal and vertical functionality of merged data sets! • Horizontally functional data sets • Vertically functional data sets • What the ….?

11. Horizontally Functional Data Sets • Example Data Set School gender sr307 sm307 Diff A M 37 51 14 A M 42 53 11 A F 41 52 11 A F 38 54 16 Note: sr307 is Scale Score Reading Grade 3 in 2007 Average Difference sr307 versus sm307 is 13 points. You can subtract values horizontally in the system or perform any appropriate function/operation horizontally to create variables of interest.

12. Vertically Functional Data Sets • Example Data Set School gender sr307 sm307 A M 37 51 A M 42 53 A F 41 52 A F 38 54 I can sum the columns to produce average performance for grade 3 Reading and Math of 39.5 and 52.5, respectively. You can sum or operate on the columns vertically, i.e. a vertically functional data set!

13. Seem Obvious? • MYSQL version of same data set Student School Grade Gender Subject Score 1 A 3 M Reading 37 2 A 3 M Reading 42 3 A 3 F Reading 41 4 A 3 F Reading 38 1 A 3 M Math 51 2 A 3 M Math 53 3 A 3 F Math 52 4 A 3 F Math 54 Even with data management features, not readily horizontally or vertically functional.

14. Assessing Data Quality • You must cross-validate data with other data sources • 30,000 3rd and 4th Grade Students Merged • 2006 100% of students had an assigned FRLP status • 2007 100% of students had an assigned FRLP status • Cross-Tabulation revealed 12% of these students changed their FRLP status • This is simply too volatile for FRLP • Expected half that volatility • Typically data quality would be reported as high • However, clearly there is reporting problem with this FRLP data

15. Implications of Data Quality Example Which status do you assign in growth model? • Students’ 2006 FRLP status • Students’ 2007 FRLP status What about the unmatched students? What patterns are evident that impacted data quality?

16. Clarity of Research Means Sample Data Set Subject sr305 sr306 sr307 sr308 1 25 31 38 42 2 23 32 39 41 3 26 33 37 45 4 . 34 37 42 5 29 36 41 44 6 . 33 38 . 7 . 32 . 45 8 28 35 42 44

17. Model 1: Reported Mean1 = 26.20 Mean2 = 33.25 Mean3 = 38.86 Mean4 = 43.29 Model 2: Employed Mean1 = 26.20 Mean2 = 33.40 Mean3 = 39.40 Mean4 = 43.20 Most researchers will run repeated measures models. The results are predicated on model 2, not model 1 means:

18. Understanding Meaningfulness in Improvement in Education Not Meaningful 80 75 Meaningful 75 80

19. Data Quality • It is not just a list of variables • It is not just matching rates • Growth Models are much more complicated because you are involving multiple years of data … Most have difficulty with current year data • It is really a global process of validating, cross-validating, understanding, and studying your data sets.

20. Growth Modeling is a Field in Statistics • Difference Scores • Trend analyses • Randomized Block Designs • Covariance models • Univariate models • Multivariate models • Hierarchical Linear Models • Value-Added Models • Latent Growth Curve Models • Structural Equation Models • Regression/Projection Models All potentially appropriate

21. NCLB Methods for Growth Models • Equipercentile Models • Growth Trajectory Models

22. 70th Percentile Scale Score 670 Actual Gain = 22 Points (PGI = 1.1) Value-Added Gain! 70th Percentile Scale Score 650 Expected Gain = 20 Points Performance Growth Index (PGI) 30th Percentile Scale Score 640 Actual Gain = 36 Points (PGI = .90 Actual Growth Expected Growth PGI = Expected Gain = 40 Points 30th Percentile Scale Score 600 22 20 1.1 = Year 2 Year 1

23. Representing “Value-Added” Increases in Student Performance Actual Value Added Predicted Student B Actual Value Added Predicted Student A *Red Lines represent predicted student improvement *Blue Lines represent actual student improvement *Value-Added is the increases over what was predicted for student performance

24. Growth Models Research: Develop Goals • Identify student improvement • District? • School? • Classroom*? • Predict Performance • Student? • Identify curriculum areas in need of improvement • Grade? • Class? • Professional Development • Target areas to provide instructional support *Note: Classroom is “Code” for teacher level!

25. Growth Models Research: Evaluate test data • Can we actually measure student achievement or change in student achievement? • Student level • Linking data … accuracy • Are the tests valid? • Vertically equated? • Vertically articulated? • Multi- versus Uni-Dimensional • Correlation versus redundancy? • Issue of content strands

26. Growth Models Research: Summative Measures • Accountability • Secretary Spellings Pilot Growth Model Program (PGMP) • Prospective versus Retrospective? • Two Components • Growth Model • Scoring Model • 13 states participating • Limited impact • Why?

27. Growth Models Research:Formative Measures • District/School/Classroom Based • Standardized or individualized assessments … both? • Tests equated/linked to curriculum? • Link of state and local assessments? • Local assessment aligned with state curriculum? • Prospective versus Retrospective? • Individual student information for teachers and parents

28. Growth Models Research: Methodology • What is Appropriate? • Student matching? • Across all groups • Change in status (FRLP) • Covariance models? • Use of demographics in models • Imputation procedures? • Missing data • Confidence intervals • How and where to apply? • What are the decision rules? • What constitutes adequate growth? • Use of results? • Ability of educational stakeholders to understand the results

29. Growth Models Research: Outcomes • Professional development • Using results in constructive professional development • Reporting results • Personnel reports (Private) • Parent reports (Private) • School, district or state level reports (Public)

30. Example of Research QuestionExpected Scale Score Growth for Students at the Proficiency Cut Score -- Arkansas

31. Arkansas Scale Scores Grades 3 - 8: Non-Linear and Autoregressive

32. Summary Report for School Level Questions • What percentage of students met the expected gains for the year for each group? • Did any group differ sizably from the combined population in % meeting growth? • Which group(s)? …. What subject(s)?

35. Researchers are Summarizing Growth Information • Evaluating why students did not make expected progress • Evaluating why students did make expected progress • Evaluating the differences for these two groups of students • Identifying if any systematic changes to instruction, materials, pacing, order of presentation, etc. impact growth of students • As a whole or for certain sub groups • Identifying if any individual characteristics or situations negatively impacted growth • Investigating possible curriculum modifications that may help specific students achieve as expected?

36. Key Research Questions Being Investigated: Classroom and Student Level • Which students did not meet expected growth? • Is there a pattern among the students who did not meet growth? • Which students did not meet the proficiency threshold (lost ground this year)? • Is there a pattern? • What do you know about the students’ performance in the subject that may inform further instructional action or intervention? • What additional information do you need to guide your instructional decisions? • What resources do you have to gather the additional information?

37. An Example … District Growth Model Analysis • Outcomes • Professional Development • Teacher programs and reviews • Computerized data for teachers and principals • Student assessments of performance • Early intervention strategies for students • Public reporting of school performance • Making “real” educational improvements • Outpacing national trends

38. Mean ITBS Literacy Equipercentile Method by Grade Level District grade level PGIs indicate students had greater than expected growth at all grade levels in Reading, and all grade levels in Language, except first grade, when compared with national expected growth.

39. Mean ITBS Math Equipercentile Model by Grade

40. Teacher Classroom Performance: Identifying areas for professional development

41. Analysis Although one school outperforms the other according to national percentile ranking, the lower performing school is making greater gains with their students. The higher performing school should look closely at their 7th graders to determine why they are not making expected growth across all subject areas ITBS PGI ITBS PGI

42. Closing Comments:Impact of Growth Models • Represent the best method to comprehensively evaluate student achievement • Link to curriculum effectiveness • Link to professional development • Limited Expertise in Education • About 50 Ph.D.’s annually in this field • Demonstrated need to expand this field • Understanding the quality and limitations of your data set is paramount!

43. Closing Comments:Growth Models Work in Education • Growth models definitely work in education!!!! • No such thing as “the” growth model • Integrate with specific needs and goals • Incorporate with professional development programs • Develop internal capacity/critical mass to help with your growth models