Power Considerations for Educational Studies with Restricted Samples that Use State Tests as Pretest and Outcome Measure

1. Power Considerations for Educational Studies with Restricted Samples that Use State Tests as Pretest and Outcome Measures June 2010 Presentation at the Institute for Education Sciences Research Conference Russell Cole ● Josh Haimson ● Irma Perez-Johnson ● Henry May

2. The research reported here was supported by the National Center for Education Evaluation and Regional Assistance, U.S. Department of Education, through contract ED-04-CO-0112 to Mathematica Policy Research.

3. Measuring impact of education intervention • Randomized controlled trial (RCT) • Unbiased estimate of program impact • Increasingly prevalent in education research • Probability of detecting a true program impact is based on n, , effect size (ES) • Use of pretest can increase power (1- b) • Pretest-Posttest correlation shrinks minimum detectable effect size (MDES)

4. MDES Increases as Pretest-Posttest Correlation Decreases

5. State Tests Prevalent, But Appropriate? • State assessments as outcomes • Used to define proficiency for AYP • Universal in grades 3–8 (Math and ELA) • Minimizes burden • Low(er) cost and scale scores readily available • State tests tend to have lower CSEM at middle of ability distribution • Largest CSEM at tails • Variance (2) can be partitioned into explainable and unexplainable (measurement error) components • Given increased CSEM at tails, samples of students selected at tails will have higher proportions of unexplainable variance

6. General Methodology • If there is greater measurement error for low-performing students, does this mean that pretest-posttest correlations will be attenuated? • To capture variability in correlation coefficients associated to measurement error, select samples with different average achievement levels and calculate r • Compare pretest-posttest correlations across different achievement levels (and across states) to inform power calculations

7. Research Questions • What is the average pretest-posttest correlation coefficient for samples of students selected at different pretest achievement levels? • Do correlation coefficients differ by state?

8. Population Data • 4 complete states + 2 large districts from 2 additional states • 3 years of population data • 2 sets of pre-post correlations • (Year1,Year2), (Year2,Year3) • English/Language Arts & Mathematics • Grades 3–8

9. Analysis Decisions • Sample pretest achievement level determined • Lowest performers • Proficiency threshold • Average performers • Grade grouping (pretest year) • Early elementary (grades 3 and 4) • Late elementary (grade 5) • Middle school (grades 6 and 7)

10. Analysis Procedure For each state, year, subject, and grade-group: • Pretest standardization • Selection of study samples (n = 500) • Calculation of pretest-posttest correlation • 6 states, 2 years pre-post data, 2 subjects, 3 grade groups for each achievement level • Cross-cutting aggregation (ANOVA)

11. Pretest-Posttest Correlations Attenuated for Lowest-Performing Samples

12. Large Variation in Pretest-Posttest Correlation Across States

13. r = .89 r = .60 r = .37 Observed for Power Analysis

14. r = .60 r = .65 Implications for MDES Might Be Modest

15. Discussion/Summary • Pretest-posttest correlations • Large attenuation when homogeneous sample selected • Might be lower than anticipated for low performers on state assessments • Similar for ELA/Mathematics and across grade levels • Affected by other factors (ceiling/floor effects) • Use available administrative records to gauge

16. Thank yourcole@mathematica-mpr.comMay, Henry, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason (2009). “Using State Tests in Education Experiments: A Discussion of the Issues.” (NCEE 2009-013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.http://ies.ed.gov/ncee/pdf/2009013.pdf