Investing in Innovation (i3) Reviewer Orientation

1. Investing in Innovation (i3) Reviewer Orientation June 2010 Scale-Up Grant Competition Note: These slides are intended as guidance only. Pleaserefer to the official documents published in the Federal Register.

2. Schedule for Today

3. i3 Peer Review Process • The Department is using independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest • Two peer reviewers who are evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation. Three peer reviewers who are subject matter experts will score selection criteria A, C, E, F and G. • Applications were assigned to panels by one of the four absolute priorities where possible. • Subject matter reviewers will determine whether any competitive preference priority points should be added For all three grant types… • Scale-Up finalists will be invited to an in-person interview in which they may clarify elements of their application to a panel of reviewers – finalists will not be able to present new information. Scale-Up Only…

4. What is expected of a Peer Reviewer? • Review the entire i3 application package and FAQs • Review and become thoroughly familiar with the selection criteria, factors, and notes • Familiarize yourself with the e-Reader system – used for inputting scores and comments • Participate in all scheduled conference calls • Provide appropriate comments that justify the score awarded and are helpful to the applicant • Revise comments as suggested by your panel monitor • Return all forms as required to ensure payment and completion of review process • Be available the entire review process • Draw upon your expertise • Maintain confidentiality throughout the review process Preparation Process Behavior

5. What must a Peer Reviewer really do? • Receive applications and panel assignments from Synergy • Identify any Conflicts of Interest, and notify Synergy or panel monitor immediately • Participate in Orientation webinar • Register in and become acquainted with e-Reader • Read applications (It may be helpful to you to take notes!) • Write draft comments, assign scores, and submit drafts in e-Reader before panel discussion (We recommend cutting and pasting from Word) • Receive feedback on comments from Panel Monitor • Revise draft comments based on feedback as appropriate • When comments/scores are acceptable to Panel Monitor, submit final comments/scores • Sign and return Technical Review Signature Forms to Synergy • Remember to submit All Necessary Forms to Synergy Prior to Panel Discussion After Panel Discussion

6. Schedule for Today

7. i3 Is One Part of UnprecedentedDirect Federal Investment in Education Race To The Top and Other Grants($9.7B in FY2009 Funding) FormulaGrants $26B • $4.35B - Race to the Top Fund, including $350MM for development of assessments • $3.5B* - School Improvement Grants • $650MM - Investing in Innovation Fund • $650MM -Education Technology • $300MM* - Teacher Incentive Fund • $250MM - Statewide LongitudinalData Systems SFSF $48.6B ARRA K-12 InvestmentAligned with Four Assurances * Includes regular FY 09 appropriations

8. Investing in Innovation (i3) Fund Summary • To provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on: • Improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates Purpose Funding $650 million to be obligated by September 30, 2010 • Eligible applicants are: • Local educational agencies (LEAs) • Nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools Applicants

9. Types of Awards Available Under i3 i3 Development Validation Scale-up

10. Applicants Eligible Applicants can be: A local educational agency (LEA) A nonprofit organization in partnership with (a) one or more LEAs or (b) a consortium of schools

11. Key Definition: Partners Why It Is Important Official partnermeans any of the entities required to be part of a partnership under section 14007(a)(1)(B) of the ARRA (i.e., a non-profit organization, an LEA, or a consortium of schools). In the case of an eligible applicant that is a partnership between a nonprofit organization and (1) one or more LEAs or (2) a consortium of schools, the partner that was the applicant, and became the grantee when the partnership was selected to receive an award, may make subgrants to one or more of the official partners Other partnermeans any entity, other than the applicant and any official partner, that may be involved in a proposed project.

12. All Eligible Applicants Must Implement Practices, Strategies, or Programs for High-Need Students High-need studentmeans a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient. MUST MUST

13. i3 Absolute Priorities Teacher and Principal Effectiveness Early Learning (0 or 1 point) Improved Use of Data Systems College Access and Success (0 or 1 point) Improve Achievementfor High-Need Students College- and Career-ready Standards and High Quality Assessments Serving Students with Disabilities and Limited English Proficient Students (0 or 1 point) Improving Achievement in Persistently Low-performing Schools Serving Students in Rural LEAs (0, 1, or 2 points) Required forall applications Must select one(Absolute Priority) May select one or more(Competitive Preference)

14. Notes on Absolute Priority 1Innovations that Support Effective Teachers and Principals • “…increase the number or percentages of teachers or principals who are highly effective teachers or principals or reducethe number or percentages of teachers or principals who are ineffective, especially for teachers of high-need students…” • “…by identifying, recruiting, developing, placing, rewarding, and retaining highly effective teachers or principals (or removing ineffective teachers or principals).” • “…teacher or principal effectiveness should be determined through an evaluation system that is rigorous, transparent, and fair; performance should be differentiated using multiple rating categories of effectiveness; multiple measures of effectiveness should be taken into account, with data on student growth as a significant factor, and the measures should be designed and developed with teacher and principal involvement.” Two Possible Routes Multiple Methods Multiple Measures of Effectiveness 14

15. Notes on Absolute Priority 2 Innovations that Improve the Use of Data Two Possible Areas of Focus • “…(a) encourage and facilitate the evaluation, analysis, and useof student achievement or student growth data by educators, families, and other stakeholders in order to inform decision-making and improve student achievement, student growth, or teacher, principal, school, or LEA performance and productivity; or(b) enable data aggregation, analysis, and research” • “…data must be disaggregated using the student subgroups described in section 1111(b)(3)(C)(xiii) of the ESEA…” Data Disaggregation 15

16. Notes on Absolute Priority 3 Innovations that Complement the Implementation of High Standards and High-Quality Assessments • “…standards and assessments that measure students’ progress toward college and career-readiness…” • “…may include, but are not limited to, … • increase the success of underrepresented student populations in academically rigorous courses and programs…; • increase the development and use of formative assessments or interim assessments, or other performance-based tools and “metrics” that are aligned with high student content and academic achievement standards; or • translate the standards and information from assessments into classroom practices that meet the needs of all students, including high-need students. • “…eligible applicant must propose a project that is based on standards that are at least as rigorous as its State’s standards…” Focus on College & Career Readiness Range of Allowable Projects RigorousStandards 16

17. Notes on Absolute Priority 4 Innovations that Turn Around Persistently Low-Performing Schools • “Whole-school reform, including, but not limited to, comprehensive interventions to assist, augment, or replace Investing in Innovation Fund Absolute Priority 4 schools, including the school turnaround, restart, closure, and transformation models of intervention … or …” • “Targeted approaches to reform, including, but not limited to: • Providing more time for students to learn core academic content by expanding or augmenting the school day, school week, or school year, or by increasing instructional time for core academic subjects • integrating ‘‘student supports’’ into the school model to address non-academic barriers to student achievement • creating multiple pathways for students to earn regular high school diplomas” Projects May Choose Either Approach 17

18. i3 Priority 4 Schools • Under Absolute Priority 4, the Department provides funding to support strategies, practices, or programs that are designed to turn around schools that are in any of the following categories: • persistently lowest-achieving schools (as defined in the final requirements for the School Improvement Grants program, see http://www2.ed.gov/programs/sif/faq.html); • Title I schools that are in corrective action or restructuring under section 1116 of the Elementary and Secondary Education Act (ESEA); or • secondary schools (both middle and high schools) eligible for but not receiving Title I funds that, if receiving Title I funds, would be in corrective action or restructuring under section 1116 of the ESEA. 18

19. i3 Competitive Preference Priorities Teacher and Principal Effectiveness Early Learning (0 or 1 point) Improved Use of Data Systems College Access and Success (0 or 1 point) Improve Achievementfor High-Need Students College- and Career-ready Standards and High Quality Assessments Serving Students with Disabilities and Limited English Proficient Students (0 or 1 point) Improving Achievement in Persistently Low-performing Schools Serving Students in Rural LEAs (0, 1, or 2 points) Required forall applications Must select one(Absolute Priority) May select one or more(Competitive Preference)

20. Notes on Competitive Preference Priority 5 Innovations for Improving Early Learning Outcomes • “…improve educational outcomes for high-need students who are young children (birth through 3rd grade) by enhancing the quality of early learning programs” Focus on High-need Children • “…(a) improving young children’s school readiness (including social, emotional, and cognitive readiness) so that children are prepared for success in core academic subjects (as defined in section 9101(11) of the ESEA); (b) improving developmental milestones and standards and aligning them with appropriate outcome measures; and (c) improving alignment, collaboration, and transitions between early learning programs that serve children from birth to age three, in preschools, and in kindergarten through third grade.” Projects Must Address All 3 20

21. Notes on Competitive Preference Priority 6 Innovations that Support College Access and Success • “… enable kindergarten through grade 12 (K–12) students, particularly high school students, to successfully prepare for, enter, and graduate from a two- or four-year college…” • “…(a) address students’ preparedness and expectations related to college; (b) help students understand issues of college affordability and the financial aid and college application processes; and(c) provide support to students from peers and knowledgeable adults.” Focus on College Graduation Projects Must Address All 3 21

22. Notes on Competitive Preference Priority 7Innovations to Address the Unique Learning Needs of Students with Disabilities and Limited English Proficient Students • “…address the unique learning needs of students with disabilities, including those who are assessed based on alternate academic achievement standards, or the linguistic and academic needs of limited English proficient students.” • “…must provide for the implementation of particular practices, strategies, or programs that are designed to improve academic outcomes, close achievement gaps, and increase college- and career-readiness, including increasing high school graduation rates (as defined in this notice), for students with disabilities or limited English proficient students.” Focus on Either Student Population Projects That Improve Specific Outcomes 22

23. Notes on Competitive Preference Priority 8Innovations that Serve Schools in Rural LEAs • “…focus on the unique challenges of high-need students in schools within a rural LEA…” • “…must include practices, strategies, or programs that are designed to improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or improve teacher and principal effectiveness in one or more rural LEAs.” Focus on Specific Locations Projects May Address Range of Outcomes 23

24. Schedule for Today

25. Schedule for Today

26. i3 Selection Criteria and Points * Development grants will be judged in two tiers: all eligible applications will be scored on Criteria A, C, E, F, and G and the competitive preference priorities; then high-scoring applications will be scored on Criteria B and D by a different panel of reviewers.

27. Subject Matter Reviewers Subject Matter Reviewers will Score Criteria A, C, E, F, and G (not including the Evidence and Evaluation Criteria)

28. Scale-Up GrantsA. Need for the Project and Quality of the Project Design (up to 15 points) • The Secretary considers the need for the project and quality of the design of the proposed project. In determining the need for the project and quality of the design of the proposed project, the Secretary considers the following factors: • The extent to which the proposed project represents an exceptional approach to the priorities the eligible applicant is seeking to meet (i.e., addresses a largely unmet need, particularly for high-need students, and is a practice, strategy, or program that has not already been widely adopted). • The extent to which the proposed project has a clear set of goals and an explicit strategy, with actions that are • aligned with the priorities the eligible applicant is seeking to meet, and • expected to result in achieving the goals, objectives, and outcomes of the proposed project.

29. Scale-Up Grants C. Experience of the Eligible Applicant (up to 15 points) The Secretary considers the experience of the eligible applicant in implementing the proposed project. In determining the experience of the eligible applicant, the Secretary considers the following factors: The past performance of the eligible applicant in implementing large, complex, and rapidly growing projects. The extent to which an eligible applicant provides information and data demonstrating that— (a) In the case of an eligible applicant that is an LEA, the LEA has— (i) Significantly closed the achievement gaps between groups of students described in section 1111(b)(2) of the ESEA, or significantly increased student achievement for all groups of students described in such section; and (ii) Made significant improvements in other areas, such as graduation rates or increased recruitment and placement of high-quality teachers and principals, as demonstrated with meaningful data; or (b) In the case of an eligible applicant that includes a nonprofit organization, the nonprofit organization has significantly improved student achievement, attainment, or retention through its record of work with an LEA or schools.

30. Scale-Up GrantsE. Strategy and Capacity to Bring to Scale (up to 15 points) The Secretary considers the quality of the eligible applicant’s strategy and capacity to bring the proposed project to scale on a national, regional, or State level. In determining the quality of the strategy and capacity to bring the proposed project to scale, the Secretary considers: The number of students proposed to be reached by the proposed project and the capacity of the eligible applicant and any other partners to reach the proposed number of students during the course of the grant period. The eligible applicant’s capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to bring the proposed project to scale on a national, regional, or State level working directly, or through partners, either during or following the end of the grant period. The feasibility of the proposed project to be replicated successfully, if positive results are obtained, in a variety of settings and with a variety of student populations. Evidence of this ability includes the proposed project’s demonstrated success in multiple settings and with different types of students, the availability of resources and expertise required for implementing the project with fidelity, and the proposed project’s evidence of relative ease of use or user satisfaction. The eligible applicant’s estimate of the cost of the proposed project, which includes the start-up and operating costs per student per year (including indirect costs) for reaching the total number of students proposed to be served by the project. The eligible applicant must include an estimate of the costs for the eligible applicant or others (including other partners) to reach 100,000, 500,000, and 1,000,000 students. The mechanisms the eligible applicant will use to broadly disseminate information on its project so as to support replication.

31. Scale-Up Grants F. Sustainability (up to 10 points) The Secretary considers the adequacy of resources to continue the proposed project after the grant period ends. In determining the adequacy of resources for the proposed project, the Secretary considers the following factors: The extent to which the eligible applicant demonstrates that it has the resources to operate the project beyond the length of the Scale-up grant, including a multi-year financial and operating model and accompanying plan; the demonstrated commitment of any other partners; and evidence of broad support from stakeholders (e.g., State educational agencies, teachers’ unions) critical to the project’s long-term success. The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the eligible applicant and any other partners at the end of the Scale-Up grant.

32. Scale-Up Grants G. Quality of the Management Plan and Personnel (up to 10 points) The Secretary considers the quality of the management plan and personnel for the proposed project. In determining the quality of the management plan and personnel for the proposed project, the Secretary considers: The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks, as well as tasks related to the sustainability and scalability of the proposed project. The qualifications, including relevant training and experience, of the project director and key project personnel, especially in managing large, complex, and rapidly growing projects. The qualifications, including relevant expertise and experience, of the project director and key personnel of the independent evaluator, especially in designing and conducting large-scale experimental and quasi-experimental studies of educational initiatives.

33. Evidence and Evaluation Reviewers Evidence and Evaluation Reviewers will Score Criteria B and D only

34. Scale-Up GrantsB. Strength of Research, Significance of Effect, and Magnitude of Effect (up to 20 points) The Secretary considers the strength of the existing research evidence, including the internal validity (strength of causal conclusions) and external validity (generalizability) of the effects reported in prior research, on whether the proposed project will improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates. Eligible applicants may also demonstrate success through an intermediate variable that is strongly correlated with improving these outcomes, such as teacher or principal effectiveness. In determining the strength of the existing research evidence, the Secretary considers the following factors: The extent to which the eligible applicant demonstrates that there is strong evidence (as defined in this notice) that its implementation of the proposed practice, strategy, or program will have a statistically significant, substantial, and important effect on improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates. The importance and magnitude of the effect expected to be obtained by the proposed project, including the extent to which the project will substantially and measurably improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates. The evidence in support of the importance and magnitude of the effect would be the research-based evidence provided by the eligible applicant to support the proposed project.

35. Strong Evidence: Scale-up

36. Strong Evidence: Scale-up (cont’d)

37. Scale-Up Grants D. Quality of the Project Evaluation (up to 15 points) The Secretary considers the quality of the evaluation to be conducted of the proposed project. In determining the quality of the evaluation, the Secretary considers the following factors: The extent to which the methods of evaluation will include a well-designed experimental study or, if a well-designed experimental study of the project is not possible, the extent to which the methods of evaluation will include a well-designed quasi-experimental study. The extent to which, for either an experimental study or a quasi-experimental study, the study will be conducted of the practice, strategy, or program as implemented at scale. The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes. The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings. The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively. The extent to which the proposed evaluation is rigorous, independent, and neither the program developer nor the project implementer will evaluate the impact of the project.

38. Schedule for Today

39. Schedule for Today

40. i3 Peer Review Process • The Department is using independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest • Two peer reviewers who are evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation. Two – three peer reviewers who are subject matter experts will score selection criteria A, C, E, F and G. • Applications were assigned to panels by one of the four absolute priorities where possible. • Peer reviewers will determine whether any competitive preference priority points should be added For all three grant types… • Scale-Up finalists will be invited to an in-person interview in which they may clarify elements of their application to a panel of reviewers – finalists will not be able to present new information. Scale-Up Only…

41. What is expected of a Peer Reviewer? • Review the entire i3 application package and FAQs • Review and become thoroughly familiar with the selection criteria, factors, and notes • Familiarize yourself with the e-Reader system – used for inputting scores and comments • Participate in all scheduled conference calls • Provide appropriate comments that justify the score awarded and are helpful to the applicant • Revise comments as suggested by your panel monitor • Return all forms as required to ensure payment and completion of review process • Be available the entire review process • Draw upon your expertise • Maintain confidentiality throughout the review process Preparation Process Behavior

42. What must a Peer Reviewer really do? • Receive applications and panel assignments from Synergy • Identify any Conflicts of Interest, and notify Synergy or panel monitor immediately • Participate in Orientation webinar • Register in and become acquainted with e-Reader • Read applications (It may be helpful to you to take notes!) • Write draft comments, assign scores, and submit drafts in e-Reader before panel discussion (We recommend cutting and pasting from Word) • Receive feedback on comments from Panel Monitor • Revise draft comments based on feedback as appropriate • When comments/scores are acceptable to Panel Monitor, submit final comments/scores • Sign and return Technical Review Signature Forms to Synergy • Remember to submit All Necessary Forms to Synergy Prior to Panel Discussion After Panel Discussion

43. Conflict of Interest: Direct vs. Indirect Direct Conflict of Interest A peer reviewer has a DIRECT conflict of interest if: • The individual’s financial interests were affected by the outcome of the Investing in Innovation competition; • An individual helped prepare an Investing in Innovation application, even if he or she has no financial interest in the outcome of that application; • An individual has agreed to serve as an employee or consultant, or otherwise provide assistance or advice, on any project for which funding is being sought in any Investing in Innovation application, or has been offered the opportunity to do so, and has not yet accepted or declined. A peer reviewer has an INDIRECT conflict of interest if any of the following individuals or organizations has a personal financial interest in the outcome of the competition: •  The reviewer’s spouse, his or her child, a member of his or her household, or any relative with whom he or she has a close relationship; • Any employer the reviewer has served within the last 12 months; a business partner; an organization the reviewer has served as an officer, director, or trustee within the last 12 months; or an organization that he or she serves as an active volunteer; • Any person or organization with whom the reviewer is negotiating for, or has an arrangement concerning, future employment; or • Any professional associate – including any colleague, scientific mentor, or student – with whom the reviewer is currently conducting research or other professional activities, or with whom the reviewer has conducted such activities within the last 12 months. Indirect Conflict of Interest

44. Conflict of Interest: What must a Peer Reviewer do? • Complete the conflict of interest questions that were provided by Synergy. • Review the applications and contact your panel monitor immediately if, while reading the applications, you identify an area in which you may have a direct or indirect conflict of interest. • Specifically, a peer reviewer should review the applicant, official and other partners, and, if applicable, sources for the private-sector match to ensure that he or she does not have a conflict of interest with any of the entities associated with each application. Before Receiving Applications After Receiving Applications

45. Other than the peer reviewer, what are the key roles within the peer review process? • Manages relationship with Contractor (SEI/Synergy) • Coordinates across all panel monitors • Supports all panel monitors • Contact reviewers to schedule panel meetings • Assist reviewers with any questions • Identify any conflicts of interest • Make sure reviewers enter comments/scores in e-Reader promptly • Keep panel discussions on track • Promptly send feedback on written comments/scores to reviewers and notify reviewers when scores and be submitted in final form • Assign peer reviewers to panels, and re-assign as required • Mail applications to peer reviewers • Processes paperwork and compensates peer reviewers • Provide peer reviewers resources on their website Competition Manager Panel Monitor Contractor (SEI/ Synergy)

46. Schedule for Today

47. Scoring Applications • Numeric scores indicate how well the applicant responded to the selection criteria • Use the entire range of points for each criterion. • For each of the competitive preference priorities, applicants do not earn a point simply for addressing the priority. Reviewers must award points based on how well applicants address the requirements of the priority. • Make sure that your comments are consistent with your numeric scores • If full points are awarded to a criterion, all comments should be in “strengths” section and “no weaknesses found” should be entered in the “weaknesses” section. • If partial points are awarded to a criterion, there should be points and comments under both the “strengths” and “weaknesses” sections.

48. Writing Comment - “Do’s & Don’ts”

49. Sample Comment – #1 “The application proposes to create a professional development program for teachers that will instruct them how to use performance data from class assignments and to use it to improve their teaching. It’s an excellent idea.” • Problems: The comment simply paraphrases the application, and it does not explain why the reviewer likes the idea. • Revision: “One of the strengths of the proposed professional development program is that it gives teachers direct practice in using the newly created online repository of student assignments that can be used “in the moment” when classroom assessments identify a learning deficiency. As the application demonstrates on p. 17, this technology-based approach aligns with research showing that learning deficits can be remedied if addressed immediately.” • Why is it Better?: It explains more clearly why the reviewer believes this is a quality program, and it includes specific references to the proposal.

50. Sample Comments – #2 “This management plan seems thorough and thoughtful, with appropriate LEA representation on the advisory council. The allocation of leadership and responsibilities seems appropriate to the tasks, and the schedule laid out in the chart seems reasonable.” • Problems: At first glance, these comments seem fine, but they are not as helpful as they could be. More detail is necessary to explain what about the management plan is “thorough and thoughtful.” What characteristics of the schedule or the allocation of responsibilities is the reviewer referencing? • Partial Revision: “…It makes sense in this case to split administrative leadership of the project from the coordination of the each of the school-based implementation sites and to assign these duties to different individuals…” • Why is it Better?: The revision makes more clear what specific aspects of the leadership plan are “appropriate” and “reasonable.”