Measuring Student Growth

1. Measuring Student Growth

2. Feedback Loop

3. Key Concept All models are wrong, but some are useful. George Box

4. Two Conflicting Models Ptolemaic Copernican

5. Key Problems Theta Lack of Randomness Neither teachers nor students are randomly assigned

6. Topics Assessment and scoring What are the main types of growth models and their application What are the major problems presented by the use of growth models What does the actual application of growth models look like

7. Item Analysis Difficulty Index p=(number correct)/(number of responses) Discrimination Index d=(((number correct upper group)-(number correct lower group))/(number in each group))

8. Raw Score v. Scale Score Raw Score Total point value of correct responses to valid items Scale Score Equating adjustment to ensure that any given assessment is comparable to previous assessments Vertical Scale Equating adjustment that requires a higher score in higher grades

9. CRT v. NRT CRT Measure of performance relative to a delimited domain of learning tasks NRT Measure of performance relative to an individual’s standing in a group In Practice The primary difference is in the way the scores are interpreted

10. Three Measures Center Mean, Median, and Mode Spread Variance and Standard Deviation Distribution What does the data look like?

11. Distribution

12. The Mean Follows the Tail

13. Grade 6 Math 2010

14. Grade 6 Math 2010

15. Main Types of Growth Models Trajectory Value/Transition Table Projection/Linear

16. Trajectory (Growth to Proficiency) Begins with the current score and the needed score for proficiency in the future, dividing the required student gains needed to reach this score into annual targets Usually dependent on assessments with a vertical scale

17. Trajectory Model

18. Arkansas Example

19. Value Table/Transition Model Creates subdivisions of performance and awards credit for moving students to higher levels. A categorical approximation of a trajectory model, not dependent on a vertical scale

20. Value Table/Transition Model

21. Arkansas Example

22. Projection (Linear) Model Uses current and past scores to predict performance in the future Such models can be quite complex and difficult for stakeholders to understand

23. Projection Model

24. EVAAS Example

25. SGP Example

26. GMPP Evaluation

27. GMPP Participation

28. Results by State

29. Schools Meeting AYP Overall, the increase of schools making AYP when growth was included was 16% Biggest rates were in Ohio, 50%, Arkansas, 13%, and Tennessee, 10% If Ohio’s results are excluded, the overall rate is only 4%

30. Growth Model Comparisons

31. Two Viewpoints

32. Classification Errors False negative: an effective teacher is classified as a less-effective teacher False positive: a less-effective teacher is classified as an effective teacher

33. The Widget Effect The vast majority of school districts presently employ teacher evaluation systems that result in all teachers receiving the same (top) rating…. In districts that used binary ratings more than 99 percent of teachers were rated satisfactory. In districts using a broader range of ratings, 94 percent received one of the top two ratings and less than 1 percent received an unsatisfactory rating.

34. Teaching is Complex …student test scores alone are not sufficiently reliable and valid indicators of teacher effectiveness to be used in high-stakes personnel decisions, even when the most sophisticated statistical applications such as value-added modeling are employed.

35. Teaching is Complex The use of imprecise measures to make high stakes decisions that place societal or institutional interests above those of individuals is wide spread and accepted in fields outside of teaching…. nearly all selective colleges use SAT or ACT scores as a heavily weighted component of their admission decisions even though that produces substantial false negative rates (students who could have succeeded but are denied entry).

36. Perverse Incentives …research and experience indicate that approaches to teacher evaluation that rely heavily on test scores can lead to narrowing and over-simplifying the curriculum, and to misidentifying both successful and unsuccessful teachers. These and other problems can undermine teacher morale, as well as provide disincentives for teachers to take on the neediest students. When attached to individual merit pay plans, such approaches may also create disincentives for teacher collaboration.

37. Perverse Incentives At least in terms of SGP, the “neediest” students represent a teacher’s best chance to demonstrate superior growth. Much of the fear concerning growth is about “use.” Take that fear away, and what is left is something that is very useful and which teachers are interested in knowing as well.

38. Data Requirements All growth models are dependent upon the ability to track a large percentage of student over time This is especially difficult at the teacher level, where a host of issues will probably never fully be resolved

39. Terminology Is the term “value-add” loaded? Is the term “growth” more palatable and perhaps more descriptive?

40. Other Populations

41. Who Had More Growth? G5 2008 Math G6 2009 Math Increase Noah 702 77573 Ben 425 527 102

42. The Normal Distribution Percentiles 2 16 50 84 98

43. The Normal Distribution

44. Student Growth Percentiles

45. Quantile Regression Ben Noah

46. Who Had More Growth? G5 2008 Math G6 2009 Math 2009 Math SGP Noah 702 775 54 Ben 425 527 44

47. Math Scale and Growth

48. Aggregated Growth Percentiles

50. Goodness of Fit