How to Develop a Project Evaluation Plan

1. How to Develop a Project Evaluation Plan Pat Gonzalez Office of Special Education Programs Patricia.Gonzalez@ed.gov 202-245-7355

2. Initial Steps in Developing the Outcome Evaluation Plan • Identify the program or project’s mission and/or goals and objectives • Identify all relevant and important outcomes that should be evaluated • Select outcome indicators • Identify data sources and data collection procedures

3. Step 1: Identify mission and/or goals and objectives • Clarify the expectations and priorities of key stakeholders/collaborators • Get a reasonable level of agreement on goals, strategies or activities and outcomes • Develop Goals: Broad statements generally describing desired outcomes • Develop Objectives: Measurable statements about outcomes (target performance) expected to be accomplished in a given time frame

4. More on Objectives… Objectives require detail and must include a target group (who), what is to be done (activities), a time frame (when), and a target performance (how much). 80% (how much) of the 300 participating teachers (who) will indicate that the Transition Toolkit is useful, relevant and of high quality (what) on the second year follow-up survey (when).

5. Step 2: Identify relevant and important project outcomes • Short-term Outcomes typically involve learning: awareness, knowledge, attitudes, skills • Intermediate Outcomes typically involve action: behavior, practice, policies • Long-term Outcomes typically involve conditions: social, economic, civic, environmental

6. REMEMBER! Focus on Short and Intermediate-term outcomes that can be completed within the grant period

7. Outcomes are not Outputs! Outputs are the direct products of program activities, usually measured by “volume”, such as the number of classes taught or number of participants served

8. Sources of information on Program Outcomes • Legislation and regulations • Purpose statements contained in the RFP • Strategic plans, SPPs or APRs • State data systems • Program descriptions or annual reports • Discussions with stakeholders or collaborators • Complaint information • Performance measures from government agencies or other programs

9. Step 3: Select outcome indicators Each identified outcome needs to be translated into one or more outcome indicators that state specifically what is to be measured (e.g., the # of teachers passing a test)

10. Checklist for Outcome Indicators • Does each indicator measure some important aspect of the outcome? • Does each indicator start with a numerical designation such as : incidence, percentage, rate, or proportion of? • Does your list of indicators cover all your outcomes?

11. Checklist for Outcome Indicators-continued • Does your list indicators cover “quality” characteristics, such as timeliness of services? • Is the wording of your indicator sufficiently specific? Avoid terms like “appropriate”. • What is the feasibility and cost of collecting the indicator? Note that sometimes costly indicators are the most important and should be retained.

12. Step 4: Identify data sources and data collection procedures • Determine if a research design can be used to evaluate effectiveness. There are several quasi-experimental designs that can be readily applied to program evaluation. • Identify data sources, such as: extant agency/program records, performance assessments, surveys, observer ratings, and interview data.

13. Considerations in Determining Data Collection Procedures • When will data be collected? Consider your design… • When entering the program • When completing the program • Fixed interval after entering • Fixed interval after completing • Combination of above

14. Considerations in Determining Data Collection Procedures-continued • Who is considered a participant? • Include all participants or a sample? • Who will collect the data? • How will the data be analyzed?

15. The Evaluation PlanImplementation Questions Using the information concerning, goals, objectives, strategies/activities and outcomes, develop evaluation questions on implementation: Were the activities completed as intended, on time and did they result in the planned outputs? These questions provide a feedback loop for the purposes of on-going project monitoring.

16. The Evaluation PlanOutcome Questions Using the information concerning, goals, objectives, strategies/activities and outcomes, develop evaluation questions on impact/effectiveness: How well did the activities address the objectives as measured by the indicators? What changed for the target group either over time or in comparison to another group?

17. The Evaluation PlanMethods For each evaluation question: • Is a research design feasible/which ones? • What are the data sources? • What methods will be used to collect the data? • How might the data be analyzed and reported?

18. The Evaluation PlanTimelines For each evaluation question: When will data be collected? When will data be reported or used?

19. The Evaluation PlanPersonnel Responsible Who is responsible for data collection, analysis and reporting at each point in the timeline?

20. SPDG Evaluation Criteria • The extent to which the methods of evaluation are thorough, feasible, and appropriate to the goals, objectives and outcomes of the proposed project. • The extent to which the methods of evaluation provide for examining the effectiveness of project implementation strategies.

21. SPDG Evaluation Criteria - continued • The extent to which the methods of the evaluation include the use of the objective performance measures that are clearly related to intended outcomes of the project and will produce quantitative and qualitative data to the extent possible.

22. SPDG Evaluation Criteria - continued • The extent to which the methods of evaluation will provide performance feedback and permit assessment of progress toward achieving intended outcomes

23. Performance Measures • The percent of personnel receiving professional development through the SPDG Program based on scientific- or evidence-based instructional practices. • The percent of SPDG projects that have implemented personnel development/training activities that are aligned with improvement strategies identified in their State Performance Plan.

24. Performance MeasuresContinued • The percent of professional development/training activities provided through the SPDG Program based on scientific- or evidence-based instructional/behavioral practices. • The percent of professional development/training activities based on scientific- or evidence-based instructional/behavioral practices, provided through the SPDG Program, that are sustained through ongoing and comprehensive practices (e.g., mentoring, coaching, structured guidance, modeling, continuous inquiry, etc.).

25. Performance MeasuresContinued • In States with SPDG projects that have special education teacher retention as a goal, the Statewide percent of highly qualified special education teachers in State-identified professional disciplines (e.g., teachers of children with emotional disturbance, deafness, etc.), consistent with sections 602(a)(10) and 612(a)(14) of IDEA, who remain teaching after the first three years of employment.

26. Performance MeasuresContinued • The percent of SPDG projects that successfully replicate the use of scientific or evidence-based instructional/behavioral practices in schools. • The percent of SPDG projects whose cost per personnel receiving professional development on scientific or evidence-based practices is within a specified range.