June 4, 2013 Elizabeth Albro, Ph.D. Associate Commissioner, National Center for Education Research

1. IES Grant Writing Workshop June 4, 2013 Elizabeth Albro, Ph.D. Associate Commissioner, National Center for Education Research Joan McLaughlin, Ph.D. Deputy Commissioner, National Center for Special Education Research

2. Purpose of the Workshop This workshop will provide instruction and advice on writing a successful application to the Institute of Education Sciences’research grant programs, specifically to the: Education Research Grants Program (84.305A) Special Education Research Grants Program (84.324A)

3. Agenda • Introduction to IES • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative • Significance • Research Plan • Personnel • Resources

4. What is IES? Research arm of the U.S. Department of Education, non-partisan by law. Charged with providing rigorous and relevant evidence on which to ground education practice and policy and share this information broadly. By identifying what works, what doesn't, and why, we aim to improve educational outcomes for all students, particularly those at risk of failure.

5. IES Organizational Structure Office of the Director National Board for Education Sciences Standards & Review Office National Center for Education Research National Center for Education Statistics National Center for Education Evaluation National Center for Special Education Research

6. Missions of the Research Centers • NCER • Supports rigorous research that addresses the nation’s most pressing education needs, from early childhood to adult education. • NCSER • Sponsors a rigorous and comprehensive program of special education research designed to expand the knowledge and understanding of infants, toddlers, and students with or at risk for disabilities from birth through high school.

7. Overall Research Objectives Develop or identify education interventions (i.e., practices, programs, policies, and approaches) that enhance academic achievement and that can be widely deployed Identify what does not work and thereby encourage innovation and further research Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

8. Primary Research Grant Programs Education Research Grants (84.305A) Special Education Research Grants (84.324A)* These grant programs are organized by research topic and research goal. *84.324A is not being competed for FY 2014

9. Special Education Research NCSER will not hold research or research training competitions for FY 2014 If funds for research are available in FY 2014, NCSER will use these funds to make additional awards from the FY 2013 grant slates NCSER anticipates being able to hold a grant competition for FY 2015

10. Opportunities for the Study of Individuals with Disabilities in NCER Partnerships & Collaborations Focused on Problems of Practice or Policy grants program (84.305H) Postsecondary & Adult Education topic of the Education Research Grants competition (84.305A)

11. NCER Ultimate Outcomes of Interest: Student Outcomes

12. NCER Ultimate Outcomes of Interest: Student Outcomes

13. Agenda • Introduction to IES • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative • Significance • Research Plan • Personnel • Resources

14. Grant Topics • All applications to the primary research grant programs must be directed to a specific topic • Note on SF 424 Form, Item 4b (Agency Identifier Number) • Note at top of Abstract and Project Narrative

15. Education Research Topics (84.305A) Cognition & Student Learning Early Learning Programs & Policies Education Technology Effective Teachers & Effective Teaching English Learners Improving Education Systems: Policies, Organization, Management, & Leadership Mathematics & Science Education Postsecondary & Adult Education Reading & Writing Social & Behavioral Context for Academic Learning

16. Issues about Topics All require student outcomes Grade range may vary by topic Topics can overlap

17. Choosing among Overlapping Topics • What literature are you citing? • To which topic is your area of expertise best aligned? • If your focus is on a specific population of students/teachers, go to that program/topic: • Is your focus on a specific type of student/teacher (e.g., English Learners), or are you studying them as a subgroup of your sample?

18. Issues about Topics • Pre-service programs • Only exploratory research can be done on teacher pre-service programs – no development of pre-service programs, evaluation of them, or measures-development for them • Can develop or evaluate pre-service components with in-service teachers • Support for leadership pre-service programs if the programs last 24 months or less

19. Grants Primarily Focused on Professional Development for K -12 Teachers • Many topics that study K-12 education now require PD grants to be submitted under the Effective Teachers & Effective Teaching topic • Early Learning Programs & Policies • Effective Teachers & Effective Teaching • Cognition & Student Learning • Education Technology • English Learners • Improving Education Systems: Policies, Organization, Management, and Leadership • Mathematics & Science Education • Reading & Writing • Postsecondary & Adult Education • Social & Behavioral Context for Academic Learning

20. Agenda • Introduction to IES • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative • Significance • Research Plan • Personnel • Resources

21. Grant Research Goals • All applications to 84.305A must be directed to a specific goal • Note on SF 424 Form, Item 4b • Note at top of Abstract and Research Narrative • The goal describes the type of research to be done • Every application is directed to a specific topic/goal combination

22. What Topic X Goal Fits Your Project?

23. FY 2014 Research Goals Exploration Development & Innovation Efficacy & Replication Effectiveness Measurement

24. Purpose of Exploration Projects To identify malleable factors associated with student outcomes AND/OR To identify factors and conditions that may mediate or moderate relations between malleable factors and student outcomes

25. Malleable Factors • Malleable factors must be under the control of the education system • Something that can be changed by the system • Examples • Student characteristics: behavior, skills • Teacher characteristics: practices, credentials • School characteristics: size, climate, organization • Education interventions: practices, curricula, instructional approaches, programs, and policies

26. Possible Methodological Approaches for Exploration • Analyze secondary data • Collect primary data • Complete a meta-analysis

27. Awards for Exploration • Secondary data analysis or meta-analysis: • Maximum of $700,000 total cost (direct + indirect) • Maximum of 2 years • Primary data collection and analysis (with or without secondary analysis): • Maximum of $1,600,000 total cost (direct + indirect) • Maximum of 4 years • Applications proposing more than a maximum will not be accepted for review

28. Purpose of Development & Innovation Projects Development process must be iterative! • AND collect data on feasibility and usability in actual education settings • ANDcollect pilot data on student outcomes. Develop an innovative intervention (e.g., curriculum, instructional approach, program, or policy) OR improve existing education interventions

29. Range of Options for Pilot Study • Efficacy study (randomized controlled trial) • Underpowered efficacy study (randomized controlled trial with small sample size that provides unbiased effect size estimates) • Single-case study that meets the design standards of WWC • Quasi-experimental study based on the use of comparison groups with adjustments to address potential differences between groups (i.e., use of pretests, control variables, matching procedures)

30. Awards for Development Maximum of $1,500,000 total cost (direct + indirect) Maximum of 4 years In budget narrative, note budgeted cost of pilot study to ensure it does not exceed 35% of total funds Applications proposing more than a maximum will not be accepted for review

31. Efficacy & Replication Design must meet What Works Clearinghouse evidence standards! • 3 types of Efficacy and Replication projects • With or without reservations • Randomized controlled trial (RCT) favored • Strong quasi-experiment

32. Purpose #1 • Evaluate whether or not a fully developed intervention is efficacious under limited or ideal conditions • Widely-used intervention • Intervention not widely used • Possible to do so through a retrospective analysis of secondary data collected in the past OR

33. Purpose #2 • Replicate an efficacious intervention varying the original conditions • Different populations of students (e.g., English language learners) • Education personnel (e.g., reading specialists versus classroom teachers) • Setting (e.g., urban versus rural) OR

34. Purpose #3 • Gather follow-up data examining the longer term effects of an intervention with demonstrated efficacy • Students • Education personnel carrying out intervention

35. Key Features of Efficacy & Replication Goal Ask what might be needed to implement intervention under routine practice Consider role of developer to avoid conflict of interest for developer-evaluators Do not require confirmatory mediator analyses but recommend exploratory ones

36. Awards for Efficacy & Replication Efficacy • Maximum of $3,500,000 total cost (direct + indirect) • Maximum of 4 years Efficacy Follow-Up • Maximum of $1,200,000 total cost (direct + indirect) • Maximum of 3 years • Applications proposing more than a maximum will not be accepted for review

37. Purpose of Effectiveness Projects Evaluate whether a fully developed intervention that has evidence of efficacy is effective when implemented under routine conditions through an independent evaluation OR Gather follow-up data examining the longer term impacts of an intervention implemented under routine conditions on students

38. Effectiveness Goal • IES expects researchers to • Implement intervention under routine practice • Include evaluators independent of development/distribution • Describe strong efficacy evidence for intervention (from at least 2 previous studies) • Does not expect wide generalizability from a single study • Expects multiple Effectiveness projects to this end • Sample size is not a key distinction from Efficacy • Does not require confirmatory mediator analyses but encourages exploratory ones • Cost of implementation is limited to 25% of budget

39. Awards for Effectiveness Effectiveness • Maximum of $5,000,000 total cost (direct + indirect) • Maximum of 5 years Effectiveness Follow-Up • Maximum of $1,500,000 total cost (direct + indirect) • Maximum of 3 years • Applications proposing more than a maximum will not be accepted for review

40. Purpose of Measurement Grants • Develop new assessments • Refine existing assessments (or their delivery) • To increase efficiency • To improve measurement • To improve accessibility • To provide accommodations • Validate existing assessments • For specific purposes, contexts, and populations

41. Focus of Measurement Grants • Assessments may also be developed in other goals, but not as the primary focus • Primary product of measurement grant is the design, refinement, and/or validation of an assessment

42. Awards • Maximum of $1,600,000 total cost (direct + indirect) • Maximum of 4 years • Applications proposing more than a maximum will not be accepted for review

43. Attend to Changes from Previous 84.305A See Page 11 for highlights of changes in the FY 2014 RFA. Carefully read the full RFA. Applicants to all goals must describe plans for dissemination as appropriate to the proposed work.

44. Expected Products Expected Products for each goal can help you identify the right goal for your project At the end of a funded project, IES expects you to provide…

45. Expected Products for Exploration • Clear description of malleable factors (and mediators/moderators) and empirical evidence of link between malleable factors (and mediators/moderators) and student outcomes • Clear conceptual framework - theory • Determination about next steps (do results suggest future Goal 2, Goal 3, or Goal 5 project?)

46. Expected Products for Development & Innovation • Fully developed version of the intervention • including supporting materials • a theory of change • evidence that intended end users understand and can use the intervention • Data that demonstrate end users can feasibly implement the intervention • Pilot data regarding promise for generating the intended beneficial student outcomes • including fidelity measures • evidence of implementation fidelity

47. Expected Products for Efficacy & Replication Evidence of intervention impact on relevant student outcomes relative to a comparison condition using a research design that meets (with or without reservation) WWC standards Conclusions on and revisions to relevant conceptual framework (i.e., the theory of change) If beneficial impact - identification of organizational supports, tools, and procedures needed for sufficient implementation in future Replication or Effectiveness study If no beneficial impact - determination of whether a future Goal 2 is needed to revise intervention/implementation

48. Expected Products for Effectiveness Evidence of intervention impact under routine implementation conditions on relevant student outcomes relative to a comparison condition using a research design that meets (with or without reservation) WWC standards Conclusions on and revisions to relevant conceptual framework (i.e., the theory of change) If beneficial impact - identification of organizational supports, tools, and procedures needed for sufficient implementation under routine conditions If no beneficial impact – examination of why findings differed from previous efficacy studies and a determination of whether a future Goal 2 is needed to revise intervention/implementation

49. Expected Products for Measurement • Projects to develop/refine and validate an assessment: • Description of the assessment and its intended use • Description of the iterative development processes used to develop/refine the assessment, including field testing procedures and processes for item revision • Conceptual framework for the assessment and its validation activities • Description of the validation activities • Reliability and validity evidence for specific purpose(s), populations, and contexts • Projects to validate an existing assessment: • Conceptual framework for the assessment and its validation activities • Description of the validation activities • Reliability and validity evidence for specific purpose(s), populations, and contexts

50. Maximum Award Amounts (84.305A)