400 likes | 584 Views
Fast Track Program Evaluation Using Assessments Diagnostically. Philip A. Streifer, Ph.D. Bristol Superintendent of Schools; UCONN Executive Leadership Program Yvel Crevecoeur Doctoral Student, Department of Educational Psychology, program in Special Education. Primary Issues.
E N D
Fast Track Program Evaluation Using Assessments Diagnostically Philip A. Streifer, Ph.D. Bristol Superintendent of Schools; UCONN Executive Leadership Program Yvel Crevecoeur Doctoral Student, Department of Educational Psychology, program in Special Education
Primary Issues • Bristol an urban leader in reform • Next phase of development: program evaluation and focus on instructional excellence • Primary reading intervention program: Fundamentals of Literacy Development (FOLD) • Rising CT AYP Standard for 2008: Need to track student gains over time vs. NCLB/AYP year by year analysis
Pressures • 181 First Days of School • Growing Poverty Rate • Manage Expectations in City • Article for upcoming ‘Chamber Views’ • Ongoing speaking and publications on challenges • Focusing work of 1000 people • Rising AYP Standards and City confidence in schools: can we ‘get over the bar’?
Challenges • Migration • Economically Disadvantaged • Rising State NCLB Standards for 2008 • Reading – 68 to 79% proficiency • Math – 74 to 82% proficiency • Special Education Placements and Costs
181 First Days of School One School’s Experience Transfers In Transfers Out July-Aug- Sep 61 October 8 November 14 December 3 January 8 February 12 March 10 April 2 May 3 June 1 TOTAL 122 • July-Aug- Sep 53 • October 16 • November 11 • December 4 • January 24 • February 12 • March 11 • April 1 • May 0 • June 0 • TOTAL 132 GRAND TOTAL for ONE YEAR: 254 Children in and out/School Population +/-350
Data Driven Reform and Improvement Past, Present and Future
DISTRICT DATA TEAM POWER STANDARDS SCHOOL DATA TEAM UNWRAPPED STANDARDS IMPROVED STUDENT ACHIEVEMENT BIG IDEAS ESSENTIAL QUESTIONS GRADE/DEPT DATA TEAM CURRICULUM & PACING GUIDES ASSESSMENT OF STUDENT LEARNING EFFECTIVE TEACHING STRATEGIES The Bristol Accountability Initiative Data-Driven Decision Making MAKING STANDARDS WORK
CT Mastery Test and CAPT • CMT – new version for 2005-06 • CAPT – new version for 2006-07 • Both are tougher tests • Each year NCLB/AYP standard rises • NCLB/AYP standard for 2007-08 (spring 2008) is rising dramatically
District CAPT 2007 Results Areas highlighted in yellow show where Bristol exceeded State averages.
Gains Analysis: Matched Cohorts • Grade 3 to 4 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Expected Gain • Grade 4 to 5 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Loss Below Expected • Writing: Expected Gain
Gains Analysis: Matched Cohorts • Grade 5 to 6 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Expected Gain • Grade 6 to 7 (2006 to 2007) • Math: Expected Gain • Reading: Expected Gain • Writing: Expected Gain
Gains Analysis: Matched Cohorts • Grade 7 to 8 (2006 to 2007) • Math: Expected Gain • Reading: Moderate Gain Beyond Expected • Writing: Moderate Bain Beyond Expected • Grade 8 to 10 (2004 to 2007) • Math: Moderate Gain Beyond Expected • Reading: Expected Gain • Writing: Expected Gain
Role of Program Evaluation • FOLD: • working; needs to be expanded • Read-180 vs. ReadAbout: • Gains Analysis • ReadAbout Deployed in More Classes 2007-08 with Grant Funds
Return on Investment • Bristol Per Pupil Expenditure is 125th out of 169 (upper end of lower third) • CMT 2007: at or just above state averages • CAPT 2007: above state average • City of Bristol getting a good return on its educational investment
New Pressing Issue • CT AYP standard rises dramatically in 2008 in reading and math • Projection of 2008 AYP indicates many schools will be cited under NCLB • Current district performance at state average on CMT; above state average on CAPT • Poor AYP performance in 2008 will likely cause erosion of public confidence and City support • Deeper analysis ongoing to determine: • Areas of focus for each school • Safe Harbor status and why it is important • Gains analysis to show progress of students over time who remain in the district; to shift focus and opinion and show staff that they are doing a good job
Key Strategies Going Forward • AYP and Safe Harbor – achieving improvement for all children • Program Evaluation – FOLD (first pilot) • Gains Analysis – Cohort Improvement Over Time and Across Disparate Tests • Instructional Excellence
FOLD Evaluation • Why were Hubbell Elementary School’s reading scores so high? • Development of the Foundations of Literacy Development (FOLD) evaluation process • Identifying Growth Model of Improvement for each elementary school • “SWAT Team” and focused plans on other intervnetions • Long term objective: Maximizing what really works for students participating in FOLD.
FOLD Evaluation Process • Semi-structured interview of principal who originally implemented FOLD: • Key design features • Developed pilot survey questions • Reviewed and refined survey questions • Administered survey to literacy teachers and principals
FOLD Evaluation Process • Preliminary method of analysis: • Identified each school’s performance on indicators of DRA2, CMT, and history of achievement • Ranked each school’s performance to identify: • Differences across high and low performing schools • Differences between respondents (i.e., principals and literacy teachers) to identify features of implementation that need revision
Gains Analysis Logic • AERA paper presentation 2007 • Data must be matched pairs and equal interval scales • Rescale data if using different tests • Must know the absolute possible range of each dataset • Apply Modified Effect Size analysis • Interpret with reconceptualization of Cohen’s d
Gains Analysis – Different Scales • Rescale • Reset to zero scale • Ratio of one scale to the other • Analysis • Cohen’s denominator for pooled SD for populations (for unequal variances
Interpretation: Less Overlap = More Difference in Scores +/- We would expect little overlap and a positive Cohen’s d
Transformation Logic – Step 1 • Two Scales: • 0 to 10: range = 10 • 20 to 40: range = 20 • Step 1: Put scales on same starting point: • 0 to 10: range = 10 • 0 to 20: range = 20 • [Scores are reset to start at a value of zero. That is, tests with possible scoring ranges of A→B are rescaled such that scores range from 0→(B-A).]
R1 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
R1 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ratio of R1 to R2 = .5
R1 | | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | | | | | | 10| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Step 2: Multiply R2 scores by ratio of the scales (.5) to reset to R1 scale
R1 | | | | | 5 | | | | || | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0 10 20 30 40 50 R2 | | | | | 5 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Rescale • Range1/Range2 = Resets one score to the other • Example: • R1/R2 = 5/10 = 0.5(or…R2/R1 = 10/5 = 2.0)
Gains Analysis Findings…So Far All Schools Improving the Same Over Time: Poorer Performing Schools Not Catching Up – Need Faster Growth – May be Unreasonable to Achieve
Using Data Mining to go Deeper - Variables • School • Grade 3 Teacher • Grade 4 Teacher • Grade 5 Teacher • Gender • Ethnicity • SPED status • F-R Eligible • ELL status • GR5 CMT Math Scale • GR5 CMT Read Scale • GR5 CMT Writing Scale • GR4 CMT Math Scale • GR4 CMT Read Scale • GR4 CMT Write Scale • GR3 DRA Reading Level
Results So Far: • Poverty not a significant factor in achievement; all schools performing: one year gain (+/-) for one year instruction • Teacher is associated with achievement – but not clear as to how • Strong teachers are assigned in many cases to lower performing students which probably accounts for gains analysis • Need to focus interventions on specific groups of students; identified through data mining and ‘teacher/administrator knowledge’
Summary • Past work by district on DDDM and Cultural acceptance critical • Next Steps: • Program evaluation and advanced data analysis • Focus on instructional excellence • Marzano: Effective Teaching Strategies • Saphier & Gower: The Skillful Teacher • Maintain City confidence – get to ‘safe harbor’ for 2008 • Change the Conversation from NCLB/AYP to Growth Over Time