1 / 19

Third-Party Evaluation Studies as a Basis for Determining Program Effectiveness and Improvement Needs

Third-Party Evaluation Studies as a Basis for Determining Program Effectiveness and Improvement Needs. Center for Research and Reform in Education Johns Hopkins University Steven M. Ross, Ph.D. Professor and Evaluation Director. The Johns Hopkins and EIA Partnership.

kateb
Download Presentation

Third-Party Evaluation Studies as a Basis for Determining Program Effectiveness and Improvement Needs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Third-Party Evaluation Studies as a Basis for Determining Program Effectiveness and Improvement Needs Center for Research and Reform in Education Johns Hopkins University Steven M. Ross, Ph.D. Professor and Evaluation Director

  2. The Johns Hopkins and EIA Partnership • JHU has an MOU with EIA to market our evaluation services to its members • Dr. David Andrews, Dean of the School of Education (SOE), is on the EIA board of Directors

  3. The Johns Hopkins and EIA Partnership • The SOE believe strongly that the future success of 21st Century education must include “nontraditional forces,” such as social entrepreneurs (TFA, New School Venture Fund, others) and the $4 billion education industry as key components in American (and international) education

  4. The Johns Hopkins and EIA Partnership • SOE is also building online courses on educational leadership, entrepreneurism, education organizations, educational policy for rising executives in member companies • SOE and EIA are working to endow an institute on private sector education at SOE

  5. The Johns Hopkins and EIA Partnership We want SOE to be the place where education companies go for research and development and instruction

  6. The Center for Research and Reform in Education: Evaluation Services • Independent studies of program implementation, products, and outcomes • Literature reviews and research papers on selected topics • Best Evidence Encyclopedia (BEE)

  7. Recent and Ongoing CRRE Evaluations • Parent Engagement and Partnership Program (EIA) • Middlebury Interactive Program (EIA) • JUMP Math in NYC • Middle School Matters • National Institute of School Leaders • Three principal preparation programs • Women’s Initiative Fellowship Program • High school reform in Minnesota • Social-emotional learning in Northern Ireland • The Leader in Me Program in two schools • Pre-K and K Early Literacy • English Language Learners in Texas

  8. Types of Evaluation Studies • Simplest and Least Costly • Case Study Example: Examining a middle school’s use of a new computer program for supplementing math instruction

  9. Types of Evaluation Studies • Simplest and Least Costly • Survey/Interview Study Example: How 325 principals who participated in online leadership training react to the program and their application of the skills taught

  10. Types of Evaluation Studies • Simplest and Least Costly • Achievement Profile Study Example: Descriptive analysis of posted state assessment scores for 25 schools before and after using a new after-school program in E/LA

  11. Types of Evaluation Studies • Medium Rigor and Cost • Mixed-Methods Control Group Study Example: Program Schools A and B are compared on district science assessments to Control Schools C and D

  12. Types of Evaluation Studies • Medium Rigor and Cost • Quantitative Control Group Study Example: Using statistical controls, comparisons are made on school-level AP scores in chemistry between 26 program schools and 50 control schools

  13. Types of Evaluation Studies • Medium Rigor and Cost • Qualitative Control Group Study Example: Through observations, interviews, and surveys, teaching methods and student engagement are compared at two schools receiving professional development in project- based learning and two control schools

  14. Types of Evaluation Studies • Most Rigorous and Costly (Often funded by federal grants) • Mixed-Methods Randomized Comparison Study Example: 10 schools randomly selected to use a new program are compared on student-level test scores and qualitative measures to 10 schools randomly selected to serve as control sites

  15. Types of Evaluation Studies • Most Rigorous and Costly (Often funded by federal grants) • Mixed-Methods Matched Comparison Study Example: 10 schools that elected to use a new program are compared on student-level test scores and qualitative measures to 10 matched schools serving as control sites

  16. What Determines Rigor? • Multiple measures (triangulation) • Standardized (unbiased/objective) measures • Treatment-control group comparisons • Equivalent comparison groups

  17. What Determines Cost? • Accessibility of data • Cooperativeness of participants • Travel

  18. The Best Study for You: Major Considerations • What questions do you want to answer? • How quickly do you need the answers? • What resources are available to fund the study? • How accessible are participants and data?

  19. Steven M. Ross Evaluation Director, CRRE sross19@jhu.edu

More Related