1 / 31

Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder

Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder. Study Methods Looked at MEAP growth/change over two consecutive years (grade 4 MEAP to grade 5 MEAP): Both Fall 07 to Fall 08 and Fall 08 to Fall 09. Counts. -- 20 of the 21 school districts In Genesee ISD.

dobry
Download Presentation

Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using MEAP To Evaluate Teachers MERA Fall 2011 Conference Dave Treder

  2. Study Methods • Looked at MEAP growth/change over two consecutive years • (grade 4 MEAP to grade 5 MEAP): • Both Fall 07 to Fall 08 and Fall 08 to Fall 09 Counts -- 20 of the 21 school districts In Genesee ISD

  3. Description of Study Data • 4th Grade Teachers • 20 of the 21 Districts In Genesee ISD • Based on the Districts’ SIS system • Business rules: • - If schedules weren’t used in the district, the homeroom teacher was coded as the math and reading teacher – in the districts that used schedules in the elementary grades, math and reading classes were used to designate the math and reading teacher • - Teachers were included if they were the teacher of record at the beginning of the school year (1st semester grades) and the end of the school year (last semester grades) • - Students included if they had the same teacher at the beginning of the school year (1st semester grades) and the end of the school year (2nd semester grades)

  4. Number of 4th Grade Teachers, by District: F07-08, F08-09, and in the District Both Years 74%

  5. What was the size of the classrooms, for the teachers in this study? (2007-08 School Year) F07-08 Class Size Frequency Class Size

  6. Study Methods Method 1 1) Compute by-student residuals (non-linear regression, student predicted - actual scale score) 2) Compute Mean Teacher residual Method 2 1) Compute by-student Performance Level (PL) Change Index 2) Compute Mean Teacher PL change index • For each method, compute indices for F07-F08 and F08-F09. • Look at the relationship between the two years. Also Tried: 1) Converting Scales Scores to Z Scores 2) Compute by-student ‘’change”: Year2 Z_Score – Year1 Z_Score 3) Compute mean teacher change

  7. Computing By-Student Residuals (Predicted – Actual) Math F07-F08 Scale Score, Grade 5 Scale Score, Grade 4

  8. Method 1 – General Results How Much of the Variance can be Explained at the Teacher Level? By Teacher, Mean Residual, Fall07-08 Math: 15% Reading: 1.5% What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09? Correlation: Math: .32 Reading: .19

  9. How do Teachers’ Ranks Compare, F07-08 to F08-F09? Math –Teacher Ranks (Mean Residual), F07-08 X F08-F09 Rank – F08 – F09 Rank – F07 – F08

  10. How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09? MATHWithin District Relationship - Quintile Rank – SS residual

  11. How do Teachers’ Ranks Compare, F07-08 to F08-F09? Reading –Teacher Ranks (Mean Residual), F07-08 X F08-F09 Rank – F08 – F09 Rank – F07 – F08

  12. How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09? READINGWithin District Relationship - Quintile Rank – SS residual

  13. Method 2 1) Compute by-student Performance Level (PL) Change Index 2) Compute Mean Teacher PL change index -- Computed with the New Cut Score Sub-Level Cut Scores -- 2011 Top-to-Bottom PL Change Business Rules State Top-to-Bottom Ranking Performance Level Change Index FOR EACH THE SCHOOL, THE PERFORMANCE LEVEL CHANGE SCORES ARE SUMMED ACROSS STUDENTS AND AN AVERAGE IS TAKEN TO CREATE THE TWO-YEAR AVERAGE PERFORMANCE LEVEL CHANGE INDEX. THE INDEX FOR EACH SCHOOL IS THEN COMPARED TO THE REST OF THE SCHOOLS IN THE STATE TO CREATE A SCHOOL-LEVEL PERFORMANCE LEVEL CHANGE Z-SCORE. THAT Z-SCORE IS THEN MULTIPLIED BY 1/4 TO CONTRIBUTE TO THE OVERALL SCHOOL INDEX IN THE CONTENT AREA.

  14. Top-to-Bottom List*- School-Level Correlation - Reading, Grds 3-8 Relationship between: PL Change/Growth F07-to-F08 & PL Change/Growth F08-to-F09 N = 2500 r = .032 p = .11 F07 to F08 F08 to F09 *Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched

  15. ELEM/MS PL Change & Achievement Status

  16. Method 2 – General Results How much of the Variance can be Explained at the Teacher Level? By Teacher, Mean Residual, Fall07-08 Math: 15% Reading: 4% What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09? Correlation Math: .42 Reading: .15

  17. Relationship Between Teacher PL Change Index, F07 to F08 X F08 to F09 Math –By Teacher PL Change Index, F07-08 X F08-F09 R = .42 PL Change Index – F08 – F09 PL Change Index – F07 – F08

  18. What’s the Relationship Between a Teacher’s Classroom Starting Point (Mean Scale Score) and Their PL Change Index? MATH –By Teacher, Mean MEAP Scl Score Z– F07 X PL Change IndexF07-08 R = -.46 PL Change Index – F07 – F08 Mean MEAP Scl Score Z– F07

  19. 2011 State Top-to-bottom Math – Pl Change X Achievement Status (Old Cut Scores, New Rules) R = - .81 PL Change Z score Achievement Status z score

  20. How do Teachers’ Ranks Compare, F07-08 to F08-F09? MATH –Teacher Ranks (PL Change Index), F07-08 X F08-F09 Rank – F08 – F09 Rank – F07 – F08

  21. How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09? MATH –Within District Relationship quintile rank - PL Change Index

  22. How do Teachers’ Ranks Compare, F07-08 to F08-F09? READING –Teacher Ranks (PL Change Index), F07-08 X F08-F09 Rank – F08 – F09 Rank – F07 – F08

  23. How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09? READING –Within District Relationship quintile rank - PL Change Index

  24. Relationship Between Teacher Mean Residual & PL Change Index READING –By Teacher, Mean Residual X PL Change Index (F07to 08) R = .88 PL Change Index – F07 – F08 Mean Residual

  25. Relationship Between Teacher Mean Residual & PL Change Index MATH –By Teacher, Mean Residual X PL Change Index (F07to 08) R = .77 PL Change Index – F07 – F08 Mean Residual – F07 – F08

  26. …Error rates for teacher-level analyses will be about 26 percent if three years of data are use for estimation. This means that in a typical performance measurement system, more than 1 in 4 teachers who are truly average in performance will be erroneously identified for special treatment, and more than 1 in 4 teachers who [exceed] average performance by will be overlooked.

  27. Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School)? MEAP PL Change Index, Math & Reading, F06 -- F10 (GISD Students) Math PL Change Index Reading Gr3-Gr4 Gr4-Gr5 Gr5-Gr6 Gr6-Gr7Gr7-Gr8 Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched

  28. Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School? (OLD Cut Scores, NEW PL Index Rules) MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students) Math PL Change Index Reading

  29. Old & New Cut Scores MATH READING

  30. Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School? (NEW Cut Scores, NEW PL Index Rules) MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students) Math Reading PL Change Index Reading School-Level Std Dev- Math: .21 (08-09) .27 (09-10) Reading: .16 (08-09) .18 (09-10)

  31. Conclusion Developing a Growth/Progress Measure, Based on a Single Summative Assessment, that: 1) is stable across years; 2) doesn’t punish high performing students/ classes/schools; 3) incorporates the need to expect low performing students/classes/schools to “grow” at a greater rate than low performing students/classes/schools. Ain’t easy

More Related