The Program Review Process: NCATE and the State of Indiana

1. The Program Review Process: NCATE and the State of Indiana Richard Frisbie and T. J. Oakes March 8, 2007 (source:NCATE, February 2007)

2. Programs for which NCATE does not have standards • Art, music, dance, or drama education • Business, speech, and vocational education • Some advanced teacher education programs (e.g., Curriculum & Instruction • School counselor These programs undergo state review or are accredited by other accrediting agencies

3. NCATE Program Review-Specialized Professional Associations (SPAs) The national organizations that represent teachers, professional education faculty, and other school personnel who teach a specific subject matter, teach students at a specific developmental level, teach students with special needs, administer schools, or provide services to students.

4. State of Indiana Program Review The state review is similar to the NCATE process. For details on the state review, see http://www.doe.state.in.us/dps/teacherprep/program_review.html

5. Challenges for Institutions • Programs are having surprising difficulty demonstrating how assessments align with SPA standards • Programs have had difficulty selecting appropriate assessments

6. 6-8 Assessments: The Rules • Institution must submit a minimum of six assessments, unless the SPA specifies more than six required assessments • Institution may submit additional assessments when SPA does not specify all eight assessments • Five specific types of assessments are required by all SPAs

7. Required Assessments in National Program Review • State licensure exam for content • Assessment of content knowledge • Assessment of Planning (e.g., lesson plan or unit plan) • Student teaching/internship assessment • Assessment of candidate impact on student learning or providing a supporting learning environment • Other assessment

8. NCATE Unit Standard #1 • Candidates preparing to work in schools as teachers or other school personnel know the content of their fields, demonstrate professional and pedagogical knowledge, skills, and dispositions and apply them so that students learn. Assessments indicate that candidates meet professional, state, and institutional standards.

9. Required Program Assessments Content: • State licensure exam for program area (if available—otherwise another content based assessment) • Content Assessment Professional and Pedagogical Knowledge Skills and Dispositions: • Assessment of Planning (e.g., unit plan) • Student teaching/internship assessment

10. P – 12 Student Learning: • Assessment of candidate impact on student learning or providing a supporting learning environment Program is required to have at least one more assessment, can have a maximum of 8 to demonstrate candidate mastery of SPA standards Some SPAs have additional required types

11. Important things learnedfrom the pilots: Unit-wide assessments can be used as key program assessments, but faculty must demonstrate alignment with specific SPA standards

12. Three effective strategies • SPA-specific addendum • SPA-specific rubrics • Coding assessment • AND including information in narrative about context

13. Important things learned: It is critical to document alignment between assessments, scoring guides and SPA standards

14. Most Important Learning from Pilots: All Parts of Assessments Must Interrelate! STANDARDS

15. What does that mean? • Parts of the assessment are aligned with the SPA standards • These standards can be seen in the elements of the scoring guide • The data is broken down by the elements in the scoring guide

16. Developing Appropriate Assessments: What are the Criteria?

17. Working definitions of assessment The term “assessment” can refer to different aspects of the program review process • Any of the 5 required types of assessment • A specific element of an assessment • Assessment (assignment summary) • Scoring guide • Data & findings

18. The five key principles by which reviewers evaluate your reports • Are assessments aligned with standards? • Do assessments meet the cognitive demands and skill levels of the standards? • Are assessments free from bias? • Do scoring guides clearly describe the different categories? • Do data provide evidence for meeting standards?

19. Distributing assessments across standards • Do not take a “this assessment addresses all standards” approach • Think of assessments as “primary” vs. “supportive” in meeting standards • Typically, one to three assessments, taken as a whole, meet a single standard

20. The assessment instrument • Align to standards! Preferably with references to standards in the assessment itself. • If alignment is not embedded in the assessment, provide alignment chart in your discussion of the assessment, and a strong argument for how the assessment covers the standard. Be specific.

21. Characteristics of a strong assessment • The objectives candidates must meet correspond to the depth and breadth required by SPA standards • Assessments are scored primarily on standards-based criteria. • The required activities candidates must perform are clearly described. • Assessment shows no evidence of bias.

22. Scoring Guides • must be consistent with the assessment instrument • acceptable scores must target the level of competence required by a standard • must clearly describe differences among scored categories

23. Putting it all together (with data) • The data table is how you “tell your story” • Data should be collected and used in ways that help your program analyze its strengths and weaknesses. • Data should mirror the scoring categories for the assessment (e.g. if grades are used, data should be reported as grades).

24. Speaking of grades… • Grades in required courses are usually acceptable for Assessment #2. • Provide course descriptions and align courses to standards. • Course content must intensively and comprehensively address standards. • Show range of grades received by candidates, not just GPAs.

25. Other comments on data • Remove identifying #s and names. • Make sure the number of candidates in data sets are consistent across tables in reports, or explain inconsistencies that might puzzle reviewers. • Note and explain any conversions of data (e.g. numerical scores converted to grades in a data table).

26. Analyze your data • Make the most of the two pages you have per assessment to analyze and “defend” your assessments and data (Section IV). Analyze your data according to the SPA standards. Compare data across standards, cohort groups, time periods, etc. Tell what you learn from it. • In Section V, make sure to describe what you are doing or will do to address areas where data are poor or below average.

27. How Much Data? • Fall 2008 and spring 2009, two years of data • Three years of data are optimal – not necessary to provide more • Remember: Data are only as good as the assessment/scoring guide

28. Resources on NCATE web site • www.ncate.org, click on Institutions, click on Program Reviews • Program Report Forms, Guidelines, Instructions, Examples of Assessments, Examples of Report Sections, Examples of Reports • www.ncate.org, click on Program Reviewers • All training materials for program reviewers

29. Who to Contact at SPAs For questions related to the standards and what are appropriate assessments for a particular discipline: SPA Program Review Coordinators http://www.ncate.org/programreview/spacontact.asp?ch=88