Investing in Innovation (i3) Fund Evidence & Evaluation Webinar

1. Investing in Innovation (i3) Fund Evidence & Evaluation Webinar June 30, 2011 Note: These slides are intended as guidance only. Pleaserefer to the official documents published in the Federal Register.

2. Agenda • Evidence Standards—eligibility requirement • Requirements • Review process • Independent Evaluation—program requirement • Guidance on qualities of high-quality evaluation plans • Questions & answers • Live—submission via the Webinar chat function • Post-webinar: E-mail to i3@ed.gov

3. Evidence Standards— Eligibility Requirement

4. Evidence Eligibility Requirements Are Specific to the Type of Grant • An application must meetthe applicable evidence requirements before an award is made • Any application failing to meet the applicable evidence requirements is noteligible for a grant award, regardless of its score on the Selection Criteria • Any application failing to meet the applicable evidence requirements will notbe considered for a different type of i3 grant award

5. Scale-up Grants: Require “Strong Evidence” of Effectiveness • High internal validity of the evidence • Studies designed and implemented in ways that support causal inference • High external validity of the evidence • Studies based on a sufficient representation of participants and settings that the findings support scaling • Minimum size of evidence base • More than one well-designed and well-implemented experimental or quasi-experimental study OR • One large, well-designed and well-implemented multi-site randomized controlled trial

6. (Mis-) Interpretive Statement: Based on the evidence, early care & education programs should reduce pupil teacher ratios in order to lower the expulsion rate. Problem: There are competing explanations for why expulsions increase with increases in pupil-teacher ratios Caution: Not All Associations Support Causal Inferences

7. Validation Grants: Require “Moderate Evidence” of Effectiveness • Validity of the evidence • High internal validity and moderate external validity or • High external validity and moderate internal validity • Minimum size of evidence base • At least one well-designed experimental or quasi-experimental study May have small sample sizes OR may fail to demonstrate equivalence between the intervention & comparison groups, but has no other major flaws or • A correlational study with strong statistical controls for selection bias and for discerning the influence of other potential confounds

8. Development Grants: Require Evidence to Support the Proposed Intervention • Theoretical support for the proposed practice, strategy, or program that is based on research findings or reasonable hypotheses, including related research or theories in education and other sectors and • Some empirical evidence of the promise of the proposed practice, strategy or program based on prior implementation and evaluation of something similar, albeit potentially on a limited scale or in a limited setting

9. Evidence Standards—Eligibility Review Process

10. Responsibility for the Evidence Eligibility Reviews • The Institute of Education Sciences (IES) conducts the reviews and reports their findings to OII • IES uses What Works Clearinghouse (WWC) evidence standards to judge the causal validity of the evidence • IES uses experienced WWC consultants • Reviews are limited to evidence identified in the application as that supporting the eligibility requirement

11. Evidence Reviews: Scale-up Applications Two focal questions: 1. Does the evidence include a sufficient number and quality of studies? (1) More than one well-designed and well-implemented experimental study or well-designed and well-implemented quasi-experimental study? or (2) One large, well-designed and well-implemented multi-site randomized controlled trial? 2. Does evidence include a reasonable representation of the kinds of participants and settings proposed to receive the practice, strategy, or program under the Scale-up grant to expect the results would generalize?

12. Evidence Reviews: Validation Applications Question 1 of 3: What is the internal validity of the evidence of effectiveness? High = At least one well-designed and well-implemented experimental or quasi-experimental study supporting the effectiveness of the practice, strategy, or program Moderate = (a) At least one well-designed experimental or quasi-experimental study, with no major flaws except possibly failure to demonstrate equivalence between the intervention & comparison groups or (b) At least one correlational study with strong statistical controls for possible selection bias Low = No study with high or moderate internal validity

13. Evidence Reviews: Validation (Cont’d) Question 2 of 3: What is the external validity of the evidence? High = Evidence pertains to the kinds of participants and settings that are the focus of the Validation Grant application Moderate = Evidence pertains to participants and settings that overlap with but may be more limited than those that are the focus of the application Low = No study with high or moderate external validity

14. Evidence Reviews: Validation (Cont’d) Question 3 of 3: Does the evidence of effectiveness provided meet one of the following conditions: 1. High internal validity and at least moderate external validity (as defined above)? or 2. High external validity and at least moderate internal validity (as defined above?

15. Evidence Review: Development Applications Two focal questions: 1. Is there a well-reasoned, theoretically supported rationale for the proposed practice, strategy, or program based on research findings or reasonable hypotheses? 2. Is there empirical evidence of the promise that the proposed practice, strategy, or program (or one similar to it) will improve student outcomes (albeit, possibly, based on implementation on a limited scale or in a limited setting compared with the participants and settings proposed to receive the treatment under the Development grant)?

16. What Works Clearinghouse Evidence Standardshttp://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=19&tocid=1Used to Judge Internal Validity of Evidence

17. How Does the WWC Assess Research Evidence? • Type of design: does the study design allow us to draw causal conclusions? • Strength of data: does the study focus on relevant outcomes and measure them appropriately? • Adequacy of statistical procedures: are the data analyzed properly?

18. Ineligible Designs Anecdotes and testimonials Case studies Descriptive Correlational WWC Standards Apply to Causal Designs Eligible Designs • Randomized controlled trials (RCTs) • Quasi-experimental designs (QEDs) Potentially Eligible Designs • Regression discontinuity (RDD) • Single case (SCD)

19. Key Elements of the WWC RCT/QED Standards Randomization Yes No Attrition Equivalence Low High Meets Evidence Standards with Reservations Does Not Meet Evidence Standards Meets Evidence Standards Yes No

20. Caution 1: RCTs with high sample attrition must demonstrate baseline equivalence Randomization? Yes No Attrition? High Low Baseline Equivalence? Yes No Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Standards

21. Standards Account for Overall and Differential Attrition Rates Sample maximum attrition thresholds for meeting WWC evidence standards

22. Caution 2: All QEDs must demonstrate baseline between the treatment and control groups Randomization? No Yes Attrition? Low Baseline Equivalence? Yes No Meets Evidence Standards Meets Evidence Standards with Reservations Does Not Meet Evidence Standards

23. Baseline Equivalence Standard Equivalence must be demonstrated on the analytic sample

24. Other Criteria • Studies must measure impacts on relevant outcomes • Outcome measures must be reliable • Outcomes must not be over-aligned with the intervention • There is not a confound between the treatment condition and one or more other factors that also could be the cause of differences in outcomes between the treatment and comparison groups

25. Questions & AnswersPlease submit your questions on evidence eligibility requirements via the Webinar chat function now

26. Independent Evaluation--Program Requirement and Guidance

27. Evaluation Requirements All i3 Grantees MUST MUST MUST Conduct an independent project evaluation Cooperate with technical assistance provided by the Department or its contractors Share the results of any evaluation broadly Share the data sets for Validation and Scale-up evaluations MUST * Note: The quality of an applicant’s project evaluation is also a selection criterion.

28. Selection Criterion C: Quality of Project Evaluation “The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes.” Scale-up, Validation, and Development “The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively.” Scale-up, Validation, and Development Understanding of Implementation and Intermediate Outcomes Sufficient Funding to Carry Out Evaluation

29. Selection Criterion C: Quality of Project Evaluation (Cont’d) “The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings.” Scale-up and Validation “The extent to which the proposed evaluation will provide sufficient information about the key elements and approach of the project to facilitate further development, replication, or testing in other settings.” Development Generates Information to Support Follow-on Scaling or Other Activities

30. Notes on Selection Criterion C. Quality of Project Evaluation “The extent to which the methods of evaluation will include a well-designed experimental study, or if a well-designed experimental study of the project is not possible, the extent to which the methods of evaluation will include a well-designed quasi-experimental design.” Scale-up “The extent to which the methods of evaluation will include a well-designed experimental study or a well-designed quasi-experimental study.” Validation Design & Methodology for the Evaluation

31. Guidance on Qualities of High-quality Evaluation Plans

32. Evaluation Goals • Aligned with i3 performance measures • Increase strength of evidence on the impact or promise of i3-supported interventions • i3 performance measures set the expectation for all i3 independent evaluations to provide high-quality implementation data and performance feedback • Other expectations vary by grant type

33. Evaluation Goals: Scale-up & Validation Projects • Estimate the impacts of the i3-supported intervention as implemented at scale • Expectation they will be “well-designed and well-implemented,” meaning designed, implemented and reported in a manner to meet WWC evidence standards • Provide information on key elements of the intervention for replication or testing in other settings • Reflect changes to the intervention or delivery model • Reflect the nature of sites served, implementation settings, and participants • Document costs/resource use

34. Evaluation Goals: Development Projects • Provide evidence of the promise of the practice, strategy, or program for improving student outcomes: • Using a research design that provides a comparison for the outcomes of the intervention group (e.g., a randomized controlled trial or strong matched comparison group design) • Using valid outcome measures (e.g., following the WWC standards) • Provide information on key elements of the intervention for further development, replication, or testing in other settings

35. Pointers Based on the FY 2010 Competition (1 of 6) • Provide a fully developed plan in the application • Maximizes the ability to adequately address key questions • Minimizes the time spent at the beginning grant specifying details and negotiating revisions with ED • Key components of a high-quality evaluation plan • Logic model (What is the intervention? Who is it intended to serve? What outcomes is it expected to produce? How?) • Research questions (What do you want to learn? Who will the results pertain to?) • Proposed methods (What is the sampling frame? How will treatment and comparison groups be formed? What is the minimum detectable effect size? What data collection measures and methods will be used? How will the data be analyzed and reported?) • Coherent links among the above components

36. Pointers Based on the FY 2010 Competition (2 of 6) • Provide a well-developed logic model to guide the evaluation plan: • Clear description of the i3-supported intervention/scaling mechanism • Key elements of the intervention in sufficient detail to guide replication • Target population for the innovation • Expected pathways through which the innovation is expected to affect intermediate and ultimate outcomes of interest • Expected mechanisms of change—e.g., shows how factors such as content, organization, duration, training, and other procedures are expected to lead to the intended outcomes

37. Pointers Based on the FY 2010 Competition (3 of 6) • Clarity on key research questions in the evaluation plan • Creates shared expectations with partners and ED about what will be learned • Clarity on aspects of the logic model the evaluation will examine • Design (methods, sample, data collection, analysis) appropriate to address the key questions

38. Pointers Based on the FY 2010 Competition (4 of 6) • Specify proposed methods • For questions about program effectiveness (i.e., causal inference questions): • Rely on experimental methods, when possible NOTE: Challenging to meet WWC evidence standards with quasi-experimental methods (as noted above) • Describe how treatment and control groups will be formed and monitored

39. Pointers Based on the FY 2010 Competition (5 of 6) • Summarize data collection and analysis procedures • Describe data sources, samples, data collection procedures and timeline, and analysis methods for each research question • Ensure critical components of the logic model are represented in the data collection plan • NOTE: ensure measurement of implementation fidelity • NOTE: comparative studies should document the experiences of comparison participants

40. Pointers Based on the FY 2010 Competition (6 of 6) • Ensure good match among logic model, research questions, proposed methods, and i3 goals: • Sample design should yield a sample representative of the relevant population group for the i3 project & support strong impact analyses (relevant to the intervention as implemented under i3) • Analysis plan should be appropriate for addressing the study questions and well-matched to the sample design • Plan for reporting the findings should be consistent with the evaluation goals and design

41. Questions & AnswersPlease submit your questions on evaluation via the Webinar chat function now

42. Other Important Resources • Investing in Innovation Fund Website: • (http://www2.ed.gov/programs/innovation/index.html) • Notices of Final Revisions to Priorities, Requirements, and Selection Criteria • Application Packages for each competition (includes the respective Notice Inviting Applications) • Frequently Asked Questions • What Works Clearinghouse Website: • (http://ies.ed.gov/ncee/wwc) • Reference Resources, Procedures and Standards Handbook • Quick Review Protocol All questions about i3 may be sent to i3@ed.gov Note: These slides are intended as guidance only. Please refer to the official Notices in the Federal Register.