Research Grants Focused on Systematic Replication

1. Research Grants Focused on Systematic Replication CHRISTINA CHHIN, Ph.D. National Center for Education Research KATIE TAYLOR, Ph.D. National Center for Special Education Research Note: Additional information is available in the slide notes of some slides.

2. Agenda • Introduction to the Institute of Education Sciences (IES) • Research Grants Focused on Systematic Replication • Purpose • General Requirements • Project Narrative • Appendices • Application Submission and Review Process

3. Introduction to IES

4. IES, U.S. Department of Education ED's mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and ensuring equal access.

5. Organizational Structure of IES

6. Objectives of IES Grant Programs • Develop or identify education interventions (practices, programs, policies, and approaches) that enhance education outcomes and can be widely deployed • Identify what does not work and thereby encourage innovation and further research • Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

7. Research Grants Focused on Systematic Replication (Continued) 84.305R: Research Grants Focused on Systematic Replication 84.324R: Research Grants Focused on Systematic Replication in Special Education

8. Purpose • Systematically replicate IES-identified interventions in reading and math that have shown beneficial effects on education outcomes in a prior IES-funded impact study • Systematic Replication ≈ Conceptual Replication • Investigate factors that may lead to and sustain successful intervention implementation • Better understand the conditions under which interventions will likely work and for whom!

9. Replication Types • Effectiveness: Replication studies that involve the independent evaluation of an intervention when implemented under routine conditions • Efficacy: Replications that provide more support than is typically provided under routine conditions and may or may not include an independent evaluator

10. Standards for Excellence in Education Research (SEER) Systematic Replications are expected to follow the SEER principles, as applicable. SEER encourages researchers to: • Pre-register studies • Make findings, methods, and data open • Identify interventions’ core components • Document treatment implementation and contrast • Analyze interventions’ costs • Focus on meaningful outcomes • Facilitate generalization of study findings • Support scaling of promising results

11. Who Can Apply? • Eligible applicants include non-profit and for-profit organizations and public and private agencies and institutions, such as colleges and universities. • Applications from minority-serving institutions are strongly encouraged. • The applicant institution is responsible for identifying the Principal Investigator (PI) for the project. All awards will be made as Cooperative Agreements

12. General Requirements • Student population • Education outcomes • Education settings • Interventions

13. Student Population • 84.305R: Learners without disabilities • 84.324R: Children and/or youth with or at risk for disabilities • A child with a disability is defined in IDEA • Risk for a disability is identified on an individual basis

14. Education Outcomes • Research must address reading or mathematics outcomes for learners • Other outcomes may be measured as appropriate

15. Education Settings • Education settings include child care centers, preschools, public and private K-12 schools, and formal programs under the control of education agencies that take place outside of school. • Settings vary slightly between 84.305R and 84.324R: • For 84.324R they also include homes and natural settings for early childhood special education services • For 84.305R they also include colleges/universities

16. Interventions • Research must replicate an IES-identified intervention. Interventions were selected based on several criteria: • Focused on reading or math • IES-funded causal-impact study completed, results published, and study meets WWC standards with or without reservations • Beneficial and meaningful impacts found for education outcomes targeted by the intervention • Ready to be implemented • No recent scale-up or expansion studies • Has not been replicated multiple times with IES funding

17. NCER-Funded Interventions Reading • Targeted Reading Intervention (TRI) • Kindergarten Peer-Assisted Learning Strategies (PALS) • Read Well Kindergarten • Intelligent Tutoring for Structure Strategy (ITSS) • Strategic Adolescent Reading Intervention (STARI) • Promoting Acceleration of Comprehension and Content Through Text (PACT)

18. NCSER-Funded Interventions Reading • Early Reading Intervention • SRA Early Interventions in Reading • Enhanced Core Reading Instruction (ECRI) Math • Moving Up! Mathematics (originally ROOTS) • Enhanced Anchored Instruction • Fraction intervention • Pre-K Mathematics Tutorial (PKMT)

19. Interventions (84.305R and 84.324R) • 84.305R: Must replicate a NCER or NCSER intervention with learners without disabilities • 84.324R: Must replicate a NCSER or NCER intervention with learners with or at risk for disabilities

20. Required Sections of the Project Narrative • Significance • Research Plan • Personnel • Resources

21. Significance Describe the study and why it’s needed, including: • Type of replication (Effectiveness or Efficacy) • Intervention to be evaluated • Evidence from prior impact studies of the intervention • Components to be systematically varied and why those variations are important • How the results will increase the intervention’s scalability and improve commercial opportunities

22. Research Plan Describe the evaluation, including: • Sample and setting • Research design • Power analysis • Outcome measures • Fidelity • Implementation study • Data analysis • Cost and cost-effectiveness analysis

23. Research Design and Outcome Measures • For the Research Design, describe: • Its similarity to prior impact studies of the intervention • How the study will meet WWC standards • The counterfactual and how it compares to previous studies • A plan for looking at longer-term outcomes in the current study or in a future follow-up study • Outcome Measures should align with those used in the prior impact study or studies. If you propose additional or different outcome measures, justify their inclusion and discuss how they are linked to the intervention.

24. Implementation Study • Document the implementation process • Identify adaptations in response to the local context • Examine factors that affect fidelity of initial and/or sustained implementation • Results could be used to improve the efficacy, efficiency, and scalability of the intervention; and the intervention’s theory of change and future revisions.  Implementation studies are different from studies examining fidelity of implementation.

25. Cost and Cost-Effectiveness Analysis • Cost Analysis Plans specify how intervention costs will be identified and how the following will be computed: • Annual cost and cost across the lifespan of the program • Overall cost and cost at each level of the intervention • Cost per component • Start-up vs. maintenance costs • Plans for Cost-Effectiveness Analyses specify the method that will be used to determine the costs to achieve an impact on the primary outcome measure(s). • Must include a plan for both!

26. Personnel Describe the project team and demonstrate they have the expertise to conduct the research and disseminate the results, including: • Roles, qualifications, past successes, contributions of pervious work, and time commitments of all key personnel • Personnel responsible for the cost analysis and cost-effectiveness analysis and their qualifications • Additional personnel, including consultants and other personnel who will assist in carrying out the research and data management plan

27. Objectivity • If an independent evaluation is proposed: • Show that key personnel responsible for the evaluation design and analyses do not participate in the development or distribution, or have a financial interest in the intervention • The developer or distributor cannot serve as PI, but can be involved • If an independent evaluation is not proposed: • Include a plan to ensure the objectivity of the research

28. Resources Describe the resources to conduct the project and disseminate the results, including: • The primary institution’s capacity to manage the grant • Access to resources at the primary and any subaward institutions • Plan for acquiring any additional resources • Access to the settings in which the research will take place • Any offices or organizations expected to take part in dissemination • Resources to support dissemination through electronic means such as a website or social media account(s)

29. Appendices

30. Appendix A: Dissemination Plan • Your dissemination plan should: • Identify the audiences that may benefit from your research, such as policymakers, educators, parents, and/or researchers • Discuss the ways you intend to reach these audiences • Reflect the purpose of a Systematic Replication project

31. Appendix B • You must include Appendix B if... • Your application is a resubmission of a previous application to IES replicate the same intervention • Include your responses to prior reviewer comments. • You submitted a somewhat similar application in the past but are submitting the current application as a new application • Provide a rationale explaining why the current application should be considered a “new” application rather than a “resubmitted” application.

32. Appendix F: Data Management Plan • Plan for pre-registering the study • Type of data to be shared • Procedures for maintaining the confidentiality of Personally Identifiable Information (PII) • Roles and responsibilities of staff • Expected schedule for data access • Format of the final dataset • Dataset documentation to be provided • Method of data access • Whether or not users will need to sign a data use agreement • Circumstances that may prevent some data from being shared

33. Award Parameters

34. Compliance and Responsiveness Checklist Compliance • Have you included a Project Narrative? • Have you included Appendix A: Dissemination Plan? • If you are resubmitting an application, have you included Appendix B: Response to Reviewers? • Have you included Appendix F: Data Management Plan? Responsiveness • Have you met all the General Requirements for an application? • Does your Project Narrative include the four required sections? Did you describe the elements required for each section? (See below.) Required Project Narrative Elements • Required Project Narrative Elements • The intervention to be evaluated, selected from the tables under Part III in RFA • Evidence from prior impact studies of the intervention • Components of the prior study or studies that will be systematically varied • Whether the proposed study is an Effectiveness or Efficacy Replication • Research Plan • Sample • Setting • Research design • Power analysis • Data analysis procedures • Plan for an implementation study • Cost analysis plan • Cost-effectiveness analysis plan • Personnel • Project team • Resources • Resources to conduct the project

35. Application Submission and Review

36. Important Dates Applications are accepted once a year. Letter of Intent – due July 11, 2019 • https://iesreview.ed.gov/ Application due date – August 29, 2019 , no later than 11:59:59pm Eastern Time • https://www.grants.gov/ • We do NOT accept late applications.

37. Application Process • Review the Request for Applications (84.305R/84.324R) • Review the IES Application Submission Guide for information about preparing and submitting your application using the required application package • FY 2020 Application Packages are available on Grants.gov

38. Peer Review Process Applications are reviewed for compliance and responsiveness to the RFA. Applications that are compliant and responsive are assigned to a review panel. 2-3 panel members conduct a primary review of each application. Applications are rank ordered according to average overall score. The most competitive applications are reviewed and scored by the full panel.

39. Peer Review Process Information The Standards and Review Office https://ies.ed.gov/director/sro/peer_review/index.asp

40. Notification All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS): • Status of award • Reviewer summary statement

41. In Summary Read the Request for Applications and the IES Application Submission Guide carefully View IES on-demand webinars Review Resources for Researchers Email the program officer • 84.305R: Christina Chhin (Christina.Chhin@ed.gov) • 84.324R: Katie Taylor (Katherine.Taylor@ed.gov)