Low-Cost Evaluation Projects for Education Improvement

1. 84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions(FY 2016 Grant Competition) Robert Ochsendorf, Ph.D. National Center for Special Education Research Phill Gagne, Ph.D. Allen Ruby, Ph.D. National Center for Education Research

2. Overview • Overview of IES and its mission • General Requirements • Purpose and Products • Project Narrative • Significance • Partnership • Research Plan • Personnel • Resources • Other important sections of the application • Preparing an application

3. Legislative Mission of IES Describe the condition and progress of education in the United States Identify education practices that improve academic achievement and access to education opportunities Evaluate the effectiveness of Federal and other education programs

4. Organizational Structure of IES Office of the Director National Board for Education Sciences Standards & Review Office National Center for Education Evaluation National Center for Education Research National Center for Education Statistics National Center for Special Education Research

5. IES Grant Programs: Research Objectives • Develop or identify education interventions (i.e., practices, programs, policies, and approaches) • that enhance academic achievement • that can be widely deployed • Identify what does not work and thereby encourage innovation and further research • Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

6. Partnerships & IES Priorities IES seeks to... Encourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies (See http://ies.ed.gov/director/board/priorities.asp)

7. Short Description of Low-Cost Evaluation Projects • Carried out by Partnerships • New or established • Minimum: research institution and a state or local education agency • Purpose • Carry out rigorous evaluations of education interventions implemented by state or local education agencies • High importance to the education agency • Use secondary data (e.g., administrative data) • Low-cost: maximum grant of $250,000 • Short-duration: 2 years

8. Impetus for Low-Cost Grant Program Take advantage of opportunities to use administrative data to do evaluations Provide useful information to education agencies in a more timely manner than traditional evaluations Create additional opportunities for research institutions and education agencies to work together Identify the strengths, weaknesses, and applicability of this type of evaluation

9. General Requirements • Focus on student education outcomes • 84:305L: For students from prekindergarten through postsecondary and adult education • 84.324L: For students from prekindergarten through grade 12 with or at-risk for disability • Research occurs in an authentic education setting • Evaluate education interventions using secondary data • Partnership between research institutions and state and local education agencies • Disseminate findings in ways useful to agency decision-making

10. Student Population • 84:305L: Students from prekindergarten through postsecondary and adult education • 84.324L: Students from prekindergarten through grade 12 with or at-risk for disability • A student with a disability is defined in Public Law 108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA) • Additional requirements for identifying students at risk for developing a disability • see http://ies.ed.gov/ncser/definition.asp

11. Focus on Student Education Outcomes • Research must address education outcomes of students. For both 305L and 324L these include • Academic outcomes • Social and behavioral competencies that support student success in school • For 324L, these also include • Functional and transitional outcomes for students with or at-risk for disability

12. Student Outcomes

13. Additional 305L Student Outcomes

14. Education Interventions The wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the student, classroom, school, district, state, or federal level to improve student education outcomes High importance to state or local education agency Expected to produce meaningful improvement in student education outcomes within a short period (e.g., a semester or year) Administrative data (or other source) focused on intervention and outcomes (primary data collection not supported by grant).

15. Applications Must be from a Partnership • Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from a U.S. state or local education agency • PI from research institution:Must have the ability and capacity to conduct scientifically valid research and expertise in the education issue to be addressed • PI from State or local education agency:Must have decision-making authority for the intervention within his or her agency

16. Partnership • Partnership may be new or existing • Research institution has a broad definition • Ability and capacity to conduct scientifically valid research • Not restricted to types of institutions

17. Partnership: SEAs • State education agencies • Examples: education agencies, departments, boards, commissions • Oversee early learning, elementary, secondary, and/or postsecondary and adult education • Also includes education agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas.

18. Partnership: LEAs • Local education agencies are primarily public school districts • Community college districts • Tribal education agencies • State and city postsecondary systems • If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner • If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner • A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project

19. Additional Partners Partnerships may include more than one State or local education agency if they share similarities and interests Non-education state and local agencies may be partners as long as an education agency is a partner Partnerships may include more than one research institution if they have shared interests and will make unique contributions Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups, non-public organizations that oversee or manage schools) that will contribute to the partnership and its work

20. Dissemination/Products • Projects are to aid education agencies in decision-making • Required dissemination • Oral briefing on results to education agency • Written brief, written for non-technical audience, made available free to public • Recommended dissemination • Partner presentations to academic and practitioner audiences • Partner publications in academic and practitioner journals • Toolkit or guide for other education agencies on how to conduct a similar study

21. Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant Program • If you are not looking at student outcomes, then IES is not the appropriate funding agency • If you need time and effort to build a partnership and prepare for an evaluation, consider: • Researcher-Practitioner Partnerships under 84.305H • If the intervention you want to evaluate • Is not implemented by a state or local education agency, • Cannot be evaluated using secondary data, or • Is not expected to improve student outcomes within a short period. • Then consider the: • Education Research Grants Program (84.305A) or • Special Education Research Grants Program (84.324A)

22. Low-Cost Evaluation: Purpose • Promote joint evaluation research by research institutions and state and local education agencies • On an education intervention identified as having great importance by the education agency • That includes practitioner input into the research • That will provide timely rigorous evidence for the agency’s decision-making regarding the intervention • And the results of which will be broadly disseminated in ways easily accessible to researchers, practitioners, and the public

23. What should the partnerships do during the grant? • Identify an education intervention • Implemented by an SEA or LEA • Of high priority to that agency • Intended to improve student education outcomes • Carry out an evaluation of that intervention • Using a RCT or an RDD design • Using secondary data • Obtain overall impacts • Obtain subgroup impacts for subgroups identified in the secondary data and of interest to the education agency and/or literature • Examine other moderators and mediators of interest, fidelity of implementation, and comparison group practice if such data are available

24. Expected Products of the Grant • Causal evidence of the impact of a clearly specified intervention implemented by an SEA or LEA • Overall impacts • Impacts for available subgroups of interest • Advice for the SEA or LEA • Continuing and/or expanding the use of the intervention • Further research needs, e.g., • Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication • Implementation

25. The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources

26. Significance • The education intervention to be evaluated • The education problem/issue the intervention is to address within the SEA/LEA • Relevance to other SEAs or LEAs (secondary importance) • Components of the intervention • Rationale for why the intervention can improve student outcomes within a short period (e.g., 1 semester, 1 year) • May include theory of change • Difference from status quo • Related findings from previous studies and how this study will improve upon past work

27. Significance • The implementation of the intervention • Who will implement it and how will it be implemented • That education agency will implement or will oversee implementation • Adequate funding available for implementation • Implementation during Year 1 of the project at a level expected to impact student outcomes • Sources of secondary data to be used in the evaluation • How these data are collected • How these data will be obtained by researchers by 1st quarter of Year 2 of project

28. The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources

29. Partnership • Describe the partners • The research institution and the education agency • Offices or divisions within agency whose cooperation is necessary • Any other members of the partnership • Common interest in and benefit from this evaluation • The process through which the partners determined the specific intervention to evaluate • Data sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1st quarter of the second year

30. The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources

31. Research Plan • State research questions/hypotheses • Describe sample and setting • Define population and how your sample and sampling procedures will allow inferences to the population • Exclusion and inclusion rules and their justification • How sample size set to address expected attrition • Strategies used to increase participation and reduce attrition • Describe the setting and its implications for the generalizability of your study

32. Research Plan: Design • Discuss how design will support causal inferences and identify potential threats to internal validity • Discuss how degree of equivalence at baseline will be determined • Discuss possibility of bias from overall and differential attrition • Required use of RCT or RDD • Meet WWC evidence standards without reservations

33. Research Plan: Design Randomized Controlled Trial (RCT) • Note unit of randomization and justify choice • Describe process for random assignment and maintaining its integrity • Different Approaches to RCTs - Potential Issues • Entire population: Treatment fidelity • Volunteers: Comparison group status • Lotteries: Attrition of non-accepted parties • Staggered roll out: Little time for true comparison • Variations of program/policy: Issue of overall significance

34. Research Plan: Design Regression Discontinuity Design (RDD) Appropriateness of assignment variable Show true discontinuity Discuss possibility of manipulation of design variable and analyses to determine such manipulation Sensitivity analyses to assess influence of key procedural or analytic decisions on results

35. Research Plan: Statistical Power • Detailed description of power analysis • Justify method used to calculate power • Justify parameters used and assumptions made • Provide power for main analyses and important subgroup analyses • Along with identifying minimum detectable effect for your analysis, justify its • Reasonableness • Practical meaning • Reviewers should be able to check power calculations

36. Research Plan: Outcome Measures • Student education outcome measures relevant to states, districts, and schools • Found in administrative data or other secondary data • Discuss reliability, validity, and appropriateness • Must be collected during Year 1 of project • Additional data from previous years of intervention’s implementation may also be used if appropriate to the evaluation design • Clearly link measures to rationale for the intervention

37. Research Plan: Optional Measures • If available in secondary data, describe measures of • Moderators (subgroups expected) • Mediators (intermediate outcomes) • Fidelity of implementation • Comparison group practice

38. Research Plan: Analysis • Detail impact analyses • Make clear how analyses directly answer your research questions • Show that analyses are based on the design • Address clustering of students in classrooms in schools • Address missing data • If multiple datasets are to be linked, detail how this will be done • Describe any other analyses to be done (e.g., subgroups, other moderators, mediators, and fidelity of implementation)

39. The Project Narrative Page Limit: 15 Significance Partnership Research Plan Personnel Resources

40. Personnel • Identify all key personnel on the project team • The PI from the research institution who has previous experience carrying out the proposed evaluation design (RCT or RDD) • The PI from the education agency who makes program decisions • Other key personnel • Roles and responsibilities on the project • Each individual’s roles and responsibilities on the project • Their qualifications (i.e., expertise and experience) for their role • Their % FTE on the project • Past success at working in similar partnerships • PI qualifications for managing a grant of this type • Ensure objectivity of evaluation

41. Resources: To Conduct the Project • Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research • Institutional capacity to manage the grant • Resources available at the partner institutions to support the project • Plans to acquire any major resources not yet in hand (e.g., secondary data) • Joint Letter of Agreement by partners (Appendix D) • Letter of Agreement to provide administrative data (App. D)

42. Resources: Dissemination of Results • Results expected to be useful to the SEA/LEA partner and, perhaps, other SEA/LEAs • Findings of both beneficial impacts or no impacts • Describe your capacity and resources to disseminate findings • Required dissemination through an oral briefing for the agency and a written brief freely available to the public • Dissemination to other audiences (e.g., researchers, policymakers, practitioners, students and their families, public)

43. Other Important Sections of the Application Appendix A Appendix B Appendix C Appendix D Budget & Budget Narrative

44. Appendix A (Required for Resubmissions) Page Limit: 3 If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments

45. Appendix B (Optional) Page Limit: 5 Figures, charts, or tables that supplement the project narrative Timelines for the project (very useful) Examples of instruments used in the collection of the administrative or other secondary sources of data Do NOTinclude narrative text

46. Appendix C (Optional) Page Limit: 5 • Examples of materials used in the intervention: • curriculum materials • computer screen shots • training documents • assessment items • other materials • Do NOTinclude narrative text

47. Appendix D (Required) No Page Limit • Required Letters of Agreement • Joint Letter from the research institution and the SEA/LEA • Document participation and cooperation in the partnership • Set out each’s roles and responsibilities under the project • Letter from the office in charge of the agency’s data • Project will have access to data required in time to do analysis • Optional Letters of Agreement • Separate Letters from other organizations taking part • Letters from any consultants and schools taking part

48. Budget & Budget Narrative • Maximum project length is 2 years • Maximum award is $250,000 • Funds must be used for evaluation only (e.g., cannot be used for implementation of the intervention or primary data collection) • Award size depends on project scope • Include a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested

49. Preparing Your Application • Important dates • Information sources • Read the RFA • Talk with a program officer • Review process

50. Important Dates & Deadlines