February 22, 2012

1. 325K: Combined Priority for Personnel Preparation Webinar on the Annual Performance Report for Continuation FundingOffice of Special Education ProgramsU.S. Department of EducationWashington, DC February 22, 2012

2. Annual Performance Report (APR) • An annual report of your performance in meeting the approved objectives of the projectand responsible use of federal funds • Required for all active grants, including those in no cost extension (NCE) • OSEP reviews the report to determine if substantial progress has been made in order to grant continued funding or a NCE

3. Overview • Recognize strong project objectives that can be associated with high-quality performance measures • Develop relevant, measurable, outcome-oriented performance measures related to your objectives that maximize the potential for meaningful data and positive outcomes • Complete the ED Grant Performance Report (ED 524B)

4. Why Is This Important? • High-quality objectives and measures … • Make it easier for you to measure your progress for the purpose of grant management • Allow you to report progress easily and quantitatively • Establish targets (both short-term/annual and long-term) • Allow OSEP staff to gather evidence of program effectiveness Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

5. Goal – Objectives - Measures Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

6. Project Objectives • What are you trying to accomplish? • Objectives should answer this question. • Preferred format for objectives: • Begin the objective with a verb and define a desired outcome or condition

7. Attributes of High-Quality Project Objectives • Relevance • How relevant is the project objective to the overall goal of the program and/or the goal of your project? • Applicability • How applicable is the project objective to the specific activities that are being conducted through your particular project? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

8. Attributes of High-Quality Project Objectives • Focus • How focused is the project objective? • Measurability • Are there concepts in the project objective that lend themselves to measurement? If so, is measurement feasible? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

9. Project Objectives -- Examples • Recruit and retain high-quality scholars • Implement a high-quality personnel development program that is aligned with State and national professional organization standards • Prepare high-quality personnel to serve children with disabilities and their families • Disseminate information and outcomes of the program within State and nationally

10. Performance Measures • How are you measuring your progress in meeting your objectives? • Performance measures should answer this question.

11. Performance Measures are … • Measurable indicators used to determine how well objectives are being met. • Important to establish - • How progress will be assessed • How much progress will constitute success • How will it be known if an objective or part of an objective has been achieved Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

12. Performance Measures Taken from the Center for Evaluation & Educational Policy (CEEP) at Indiana University

13. 2 Types of Performance Measures • Program • All grantees funded under the Personnel Development Program (PDP) must report on the PDP program performance measures established by OSEP. • Project • Each grantee reports on the approved project performance measures established to meet their project objectives.

14. Program Performance Measures • Program • Measures were established by OSEP for the Personnel Development Program. • Measures apply to all grants funded under the Personnel Development Program. • Results on these measures are reported to Congress under the Government Performance and Results Act of 1993.

15. Personnel Development PROGRAM Performance Measures #1: The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. #2: The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. #3: The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. #4: The percentage of Special Education Personnel Development funded degree/certification program recipients who are working in the area(s) in which they were prepared upon program completion. #5: The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion, and who are fully qualified under IDEA. #6: The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years, and who are fully qualified under IDEA. #7: The Federal cost per fully qualified degree/certification recipient.

16. Program Performance Measures • OSEP collects data on the program performance measures each year. • Grantees should propose a project performance measure that addresses each program performance measure reported for the 325K program. • The focus of the project measure must be the same as and clearly aligned with the program measure.

17. Program Performance Measure #1. The percentage of Special Education Personnel Development projects that incorporate evidence-based practices into their curricula. Example of related projectperformance measure: By the beginning of Year 3, 100% of syllabi from core courses will be rated as “evidence-based” by the Project Advisory Board using a rubric designed by the Project.

18. Program Performance Measure #2. The percentage of scholars completing Special Education Personnel Development funded programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities. Example of related project performance measure: At the end of their program, 100% of scholars will achieve a minimum of 3 on a 4-point rubric for a portfolio that provides evidence of their competencies in providing interventions to children with disabilities.

19. Program Performance Measure #3. The percentage of Special Education Personnel Development funded scholars who exit preparation programs prior to completion due to poor academic performance. Example of related project performance measure: As measured annually, no scholars (0%) will have exited the program due to poor academic or field-based performances.

20. Program Performance Measure #4. The percentage of Special Education Personnel Development funded scholars who are working in the area(s) in which they were prepared upon program completion. Example of related projectperformance measure: By the end of Year 5 of the project, 90% of scholars will be working in the area for which they were trained following program completion.

21. Program Performance Measure #5. The percentage of Special Education Personnel Development funded degree/certification recipients who are working in the area(s) for which they were prepared upon program completion, and who are fully qualified under IDEA. Example of related projectperformance measure: By the end of Year 5 of the project, 90% of program graduates will be working in the area for which they were trained and fullyqualified under IDEA.

22. Program Performance Measure #6. The percentage of Special Education Personnel Development degree/certification recipients who maintain employment in the area(s) for which they were prepared for 3 or more years, and who are fully qualified under IDEA. Example of related projectperformance measure: By the end of Year 5 of the project, 90% of eligible program graduates will have maintained employment in the area for which they were trained for 3 or more years, and be fully qualified under IDEA.

23. Program Performance Measure #7. The Federal cost per fully qualified degree/certification recipient. (Efficiency Measure) Example of related project performance measure: The Federal cost is measured in Year 5 by dividing the total grant award (e.g., $250,000 x 5) by the number of scholars counted for Performance Measure #2.

24. Project Performance Measures • Project Measures that the grantee establishes to meet their project objectives • Project performance measures can address both the process of working towards an objective and the outcome related to meeting the objective • Ensure a mix of both process and outcome measures, but most will be outcome Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

25. High-Quality Performance Measures • High-quality performance measures show: • What will change. • How much change you expect. • Who will achieve the change. • When the change will take place. Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

26. Project Performance Measure • Process measure example - • Project staff (who) will implement 4 (how much) targeted recruitment activities (what ) during the first year of the grant (when).

27. Project Performance Measure • Outcome measure example – • Inquiries from potential candidates (who) about the program (what) will increase 20% from previous recruitment efforts (how much) during the first year of the grant (when).

28. Project Performance Measure • Outcome measure example – • At the end of each semester (when), 100% of scholars (who) will rate their mentoring support (what) as 5 or higher on a 7-point scale (how much) using an on-line survey developed for the project.

29. Project Performance Measure • Outcome measure example – • At the end of their program (when), 90% of scholars (who) will demonstrate advanced critical reflection skills (what) by achieving a proficiency score of at least 3 on a 4-point rubric (how much) developed by the project.

30. Project Performance Measure • Process measure example – • By the end of the project period (when), the project staff (who) will present on activities and outcomes (what) of the project at a minimum of 3 state and national meetings (how much).

31. Common Problems • Activities are NOT performance measures • If the best response is “Yes, we did that,” it is likely an activity (not a performance measure) • Examples of activities: • Scholars complete required coursework • Hold an Advisory Board meeting quarterly • Establish a mentoring program • Evaluate the project Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

32. Common Problems • Performance measures need to be measurable • Examples with measurement problems (activities rather than outcomes): • Maintain meaningful collaborative partnerships • Provide academic support in areas of concern to scholars • Establish field placements in low income schools Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

33. Need additional information on writing performance measures? • All grantees are strongly encouraged to seek training on writing performance measures. • For further information on developing performance measures and logic models, see - • http://www.tadnet.org/model_and_performance

34. Summary • Projects should have a few clear objectives that explain what the project is doing to support their overall goal(s) • Each objective should have a few, specific performance measures to demonstrate how progress toward meeting the objective is being measured • Both program and project performance measures are reported Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www.tadnet.org

35. Completing the ED 524B • The ED 524B is a required annual and final performance reporting form with specific instructions. • The form is used by all ED grants and has been approved by the Office of Management and Budget (OMB). Project Directors must follow the directions listed in the Dear Colleague letter and ED 524B Instructions provided by OSEP • Word or PDF versions of the forms are available at http://www2.ed.gov/fund/grant/apply/appforms/appforms.html

36. X x

37. ANNUAL PERFORMANCE REPORTS Reporting Period: For first year grants, the date is the beginning of the project year to February 29th. For grants in years 2-5, it is the date from the end of the previous reporting period to February 29th. 02 29 2012-__

38. ANNUAL PERFORMANCE REPORTS Budget Expenditures: Report the expenditures during the “Reporting Period.” Must be data or information from the business or grants office.

39. ANNUAL PERFORMANCE REPORTS Performance Measure Status: This will be checked “No” since OSEP is asking for data for the award reporting period, not for the budget period. The date entered here will be the due date for your Final Performance Report; which is 90 days after the end of the grant. Signatory must have authority to sign on behalf of the institution since the grant is from the Department to the institution and not to an individual. The Authorized Representative signs; not the Project Director.

40. OMB No. 1894-0003 Exp. 02/28/2011 OMB No. 1894-0003 Exp. 02/28/2011 EXECUTIVE SUMMARY SHEET H325 K - - - - - *** Provide highlights of the project's activities and the extent to which the expected outcomes and performance measures were achieved during the reporting period. Do NOT include the project abstract.

41. PROJECT STATUS CHART H325 K - - - - -

42. PROJECT STATUS CHART H325 K - - - - - Enter one of the project’s objectives; on subsequent pages, you will enter additional project objectives as submitted in your grant application.

43. PROJECT STATUS CHART Enter the PROGRAM performance measure(s) that align with the objective immediately followed by any PROJECT measure(s) developed to address that program measure. Enter any additional PROJECT performance measures that show you are measuring progress toward meeting the objective (e.g., 1c., 1d.,).

44. PROJECT STATUS CHART Here you identify if the performance measure is a PROGRAM measure, “PRGM,” or a PROJECT measure, “PROJ.” PRGM PROJ Note: Program measure refers to one of OSEP’s 7 performance measures for the Professional Development Program. Projectmeasures are unique to your grant.

45. PROJECT STATUS CHART QUANTITATIVE DATA Depending on your measure, enter either a raw number, or a ratio and percentage. Enter the target number identified in the performance measure and then the actual data for this year. If complete data are not available for the measure, enter “999” (if no baseline) or “NA” in the “Raw Number” or “%” column, as appropriate. Provide an explanation at the bottom of the page under “Explanation of Progress.” 8 6

46. Information to Include in the Explanation of Progress Section • Describe the data provided (e.g., what data collection methods were used, when were the data collected, how was a sample drawn, are there missing/incomplete data, what was the response rate, was a reliability measure taken). Your Project Officer should be able to understand and interpret the numbers in the chart from your description in this section. • What changes in the data occurred since last APR (i.e., trend)? • What activities were undertaken to achieve the targets? • If targets were not met, what are possible reasons? • How will activities that failed to meet targets be improved?

47. PROJECT STATUS CHART QUALITATIVE DATA If measure requires the collection of qualitative data, report the performance measure and type (PROG or PROJ) and then, enter “N/A” under the Raw Number and Percentage columns. N/A N/A N/A N/A In the “Explanation of Progress” section of the page, referencing the performance measure by number, report applicable qualitative data along with other information about how these data were collected, targets and activities –refer to previous slide for additional content requirements.

48. By Year 3, 20 20 18 20

49. H325 K - - - - - Implement a high-quality personnel development program that is aligned to State and national professional organization standards Explanation of Progress (Include Qualitative Data and Data Collection Information) (1b) The program faculty are committed to reviewing the entire curriculum of our personnel development program and ensuring evidence-based practices are integrated into core courses, including field experiences. As part of that plan, 100% of the syllabi would be thoroughly reviewed by the Project Advisory Board using an evidence-based practices (EBP) rubric developed for this Project to decide whether revision was necessary to meet our new state professional practice standards for licensure which include knowledge and skills on the use of EPB. By the end of Year 2, a full review of all 20 syllabi (100%) was completed and it was determined that 18 met the target (90%); 2 require a revision to include EBP. We attached the rubric to Section C of this report. (1c) Since this grant is designed with a focus on universal designs for instruction, a required seminar on this topic was identified for more in-depth evaluation. The seminar was offered during the Fall 2010 semester. All students in the course completed the rating. Given that the target was not met, students will be interviewed to see how the seminar might be improved for future scholars, with the goal of meeting the “usefulness” score target of 4.5 the next time the class is offered and evaluated.

50. Final Page of the Report H325 K - - - - - Section B: Refer to the instructions for Section B with the ED 524B Section C: Refer to the instructions for Section C with the ED 524B—put attachments here (e.g., recruitment material, evaluation instruments, journal articles)