The Power of Comparison Groups

1. The Power of Comparison Groups Rita O’Sullivan Evaluation, Assessment, & Policy Connections School of Education, University of North Carolina at Chapel Hill (ritao@unc.edu) NCCEP Capacity Building Workshop, Las Vegas, NV- February 2013

2. Important GEAR UP Evaluation Questions: • Which GEAR UP activities contribute to improved achievement? • Do students who participate in GEAR UP do better academically? • Does GEAR UP participation influence college enrollment and retention?

3. Possible Evaluation Design Strategies: A. Which GEAR UP activities contribute to improved achievement? • Increase in Percentage/Number of students participating in GEAR UP Activities • Improved achievement from last year • Growth in Pre-test/Post-results

4. Possible Evaluation Design Strategies: B. Do students who participate in GEAR UP do better academically? • Improved achievement from last year • Growth in Pre-test/Post-results • Comparison of GEAR UP students with non-GEAR UP students in a school

5. Possible Evaluation Design Strategies: C. Does GEAR UP participation influence college enrollment and retention? • Comparison of GEAR UP students with previous cohort in a school • Comparison of GEAR UP students with school district average • Comparison of GEAR UP schools with non-GEAR UP schools within a district

6. Possible Alternative Explanations for Evaluation Findings(i.e., Threats to Internal Validity) Evaluation findings vary in how convincing they are due to design strategies usedwith accompanying possible alternative explanations for results:

7. Key Threats for the Designs: • Which GEAR UP activities contribute to improved achievement? • Do students who participate in GEAR UP do better academically? • Does GEAR UP participation influence college enrollment and retention?

8. EQUIVALENT Comparison Designs: • EQUIVALENT Groups directly address Sampling threats and indirectly: • History (i.e. services comparison group students received) can still be a problem

9. Three Credible Comparison Group Design Options: • Cohort Analysis • Multiple Variable Matching • Propensity Score Matching

10. Cohort Analysis • Definition: Comparing one GEAR UP cohort with a previous cohort within the same school that did not experience GEAR UP • Strengths: • Controls for school differences • Cautions: • One cohort is not necessarily the same as another • One year is not necessarily the same as the next

11. Multiple Variable Matching • Definition: Creating a “comparable” comparison group based on a variety of relevant variables (e.g., previous achievement, ethnicity, SES, etc.) by setting selection criteria • Strengths: • Matching on multiple variables will provide relatively more equivalent groups for comparison • Challenges: • Need matching data on comparison group • Don’t necessarily know what services selected students have received.

12. Propensity Score Matching • Definition: Statistically creating a “comparable” comparison group by generating a propensity using a variety of relevant variables (e.g., previous achievement, ethnicity, SES, etc.) • Strengths: • Large number of matching variables make the identification of an “equivalent” group more possible • Cautions: • Need matching data on comparison group • Don’t necessarily know what services selected students have received.

13. Thank you!Look for a Propensity Score Workshop at NCCEP at July annual meeting in San Francisco