NAGC-CEC/NCATE Program Reviewer Refresher Training

1. Welcome! January 29, 2010 Presenter: Susan Johnsen, NAGC/CEC-TAG Margie Crutchfield, NCATE When you join the session, an Audio Dialog Box will appear. To enable receipt and transmission of audio, please select from one of the following two options, either of which will immediately place you into the session: Enter a phone number and click on “Call Me” to receive an automatic call back (Preferred Option); or click on the drop down menu and select “I will call” and follow instructions. OR Select “Use Computer Headset” and click on “Call Using Computer” (Microphone-required Option) NAGC-CEC/NCATE Program Reviewer Refresher Training 1:00 pm- Session will begin

2. Webinar Information • This session is being recorded • How to listen to and view the webinar • How to participate in the webinar • Chat box and/or via telephone • How to access an archive of the event • How to receive the power point and other documents shown today 2

3. Orienting You to the System We are UsingMargie Crutchfield

4. Goals for Today’s Refresher Training Reacquaint yourselves with the review process Familiarize yourselves with the roles of reviewer and lead reviewer Become aware of developments in the review process Discuss elements of an actual report Ask questions 4

5. Recent policy/process changes Low enrollment programs Submission date for spring semesters Changes to template New options for program review Initial Continuing Data rule (to be covered later) Update from NCATE

6. Low Enrollment Programs In Spring 2010 and Fall 2010, NCATE will defer review of all low-enrollment programs (defined as 5 or fewer completers in the last 3 years). Over the next year, NCATE staff will work with states, SPAs, and institutions to develop a new strategy for review of these programs. It is essential to maintain the integrity of the SPA process to ensure that SPA standards and national recognition decision are consistently applied. It is also imperative to reduce the burden on both programs and SPAs.

7. Change in Submission Date for Spring Starting in 2010 submission date will be 3/15---no 4/15 submission date possibility

8. Changes to Template Current template In Section I deleting questions 3, 4 and 5 3 = policies for admission, retention and completion 4 = relationship between unit’s conceptual framework and program 5 = relationship between unit’s assessment system and program assessments Clarifying directions in Section IV Modified templates were not available for Spring 2010 submissions, but many programs understood that they didn’t need to respond to questions #3, 4, and 5. So don’t be surprised!

9. New Options for Initial Review Current Process (streamlined) Allow institutions to choose all of their own assessments (with some constraints) No more than 8 Must submit state test data Must demonstrate content, pedagogical content knowledge and skills, and impact on student learning Must meet SPA standards

10. Options for Continuing Review Reduce requirements for “continuing” national recognition so that only new assessments or assessments that have been substantially changed need to be approved and minimal data provided. Focus on what is now Section V—self study and continuous improvement Permit an institution to conduct validity studies of its assessments in lieu of other program report evidence requirements

11. How will the continuing review options be implemented? Some will be piloting new options in S10 More in F10 Have developed templates for all options Will develop guidelines this fall and modify as necessary after pilots

12. A Brief Review of Terms…. • Unit = School, College, or Department of Education plus other entities on campus • Program = Specific Discipline Area • Candidate = Preservice teachers • Students = K-12 students • Program Report = what the program submits • Recognition Report = what the reviewer completes 12

13. Purpose of the Program Review • Determine whether or not the program has in place a limited number (6-8) of comprehensive assessments that demonstrate candidate mastery of the SPAstandards. • Candidate performance on these assessmentsis appropriate to demonstrate mastery. • Provide information for unit to use to respond to Unit Standard #1—and for BOE members to determine if Unit Standard #1 is met 13

14. NCATE Unit Standard #1 Candidates preparing to work in schools as teachers or other school personnel know and demonstrate, the content knowledge, pedagogical content knowledge and skills, pedagogical and professional knowledge and skills, and professional dispositions necessary to help all students learn.Assessments indicate that candidates meet professional, state, and institutional standards. 14

15. 6 Required Assessments • State licensure exam for program area (if available— otherwise another content based assessment) • Content assessment • Assessment of ability to plan learning experiences (e.g., unit plan) • Candidate efficacy in practice (e.g., practicum or internship) • Assessment of candidate impact on student learning or providing a supportive learning environment • Assessment type & focus up to program’s choice 15

16. A Brief Review of the Recognition Report A pdf version of the report form was provided to you

18. Part A

19. Part A of the National Recognition Report Recognition decision Conditions The 80% pass rate criterion forAssessment #1 (licensure data) should show that 80% of candidates in the last year (or as much as the last three years) have passed the content test. If there are fewer than 10 program completers over three years, the pass rate is not applicable. Summary of strengths 19

20. If the decision is National Recognition with Conditions…

21. THE CONDITIONS BOX MUST BE FILLED OUT

22. A word about conditions Write in the SPECIFIC conditions that need to be met by the institution on the next report submission.

23. This report functions as a contract between NCATE-SPA and the Institution. In order for an Institution to retain Recognition after a Recognized with Conditions decision has been granted…

24. The team that looks at this when it is re-submitted will zero in on this conditions section to determine whether they were met or not High Stakes!

25. Part B

26. Part B of the National Recognition Report Decision on each standard: (Met, Met with Conditions, or Not Met) Any standard that is Not Met or is Met with Conditions should be accompanied by a comment that explains why the assessment(s) is insufficient to meet the standard. Example: “The program cites Assessments 3 and 4 as evidence for this standard. However assessments and rubrics do not align with standards; therefore the data from the assessments do not provide evidence of meeting the standard.” 26

27. Part C

28. Part C of the National Recognition Report Summary of how well the program assessments address components of NCATE Unit Standard 1: C.1 – content knowledge (subject matter knowledge or professional field of study knowledge) C.2 – pedagogical and professional knowledge (knowledge of concepts, theories and research about effective teaching of subject matter or professional field) C.3 – student learning (candidates’ ability to have a positive impact on student learning) 28

29. A Note on Part C BOE members look at Part C of the report for input into NCATE’s Standard 1, which requires evidence of candidate’s content knowledge, pedagogical and professional knowledge and skills, and effects on student learning. Example (C.3): “There is no explicit requirement in the case study assessment that contains data linking the candidate’s instruction to the gifted student’s learning new knowledge and skills.” 29

30. Part D

31. Part D of the National Recognition Report Provide a brief response to the narrative in Section V of the program report. The expectation is that the program will cite examples of how they have applied the findings of assessment data to make changes in the program. Example: “It is clear that the program and unit have processes in place for analysis and application of assessment data. However, the program has not yet collected enough data to analyze any strengths and weaknesses that might be indicated.” 31

32. Part E

33. Part E of the National Recognition Report Part E – Areas for consideration Use this section to provide any summative comments on assessments or standards. For recognized reports, provide guidance as to how they might improve assessments between now and the next report submission. For not recognized reports, provide guidance for what they need to address in their assessments in order to be successful. 33

34. Part F

35. Part F – Additional Comments Part F.1 – Comments on Section IUse for comments on Section I (programmatic concerns). This may also be used to convey comments on any problems with the report format or other general concerns that do not have to do specifically with standards or assessments. Part F.2 – Concerns for follow-up by BOE Only add a comment here if the report suggests there are serious problems that may have implications for the education unit. 35

36. Reviewing Revised or Response to Conditions Reports If possible, the report will be assigned to at least one reviewer from the original review. The program will only respond to unmet standards, concerns, or conditions to recognition noted in the previous report. Therefore, the review should only focus on those areas. Reviewers may not reverse previous decisions on met standards or add new concerns unrelated to concerns/standards addressed in the second report. 36

37. Writing Reports: Tips If you repeat a comment in the report (likely), it is best to say “See comment under Standard 1” rather than simply copying the text. Be sure that the gist of your comments would make sense to an outside reader as well as to the program faculty (i.e. a BOE member or state DOE personnel) who wouldn’t necessarily understand the IHE, state, or professional terminology and acronyms. Provide specific feedback on assessments (in Parts B, C or E), particularly if they need revision. 37

38. Making DecisionsSusan Johnsen

39. Decisions on Standards Standard is met when there is sufficient evidence from one or more assessment sources that the content of the standard is covered by assessments and candidates perform at a minimally acceptable level. Standard is met with conditions if the assessments are viable, but there are no data; OR if assessments are “on the right track” but still need some refining and data demonstrate candidate adequacy. 39

40. Guidelines for Standards Decisions 40

41. Recognition Decisions The decisions you make on each of the standards in Part B should drive your overall decision for the program. • Nationally recognized • Nationally recognized with conditions (conditions must be specified, and must relate to the standards; programs have 18 months to address conditions) • It is critical that the cited conditions are clearly written • These act as a contract between program and NCATE • Further Development Required/Recognized with Probation (program has 12-14 months to submit a revised report) • Not nationally recognized (for programs that have submitted a total of three times and have not managed to move to recognized with conditions or nationally recognized) 41

42. Criteria for Program Decisions 42

43. Advice from Experienced Reviewers/Auditors Reviewers frequently "forget" to review in light of a program's context. They may read that first section over too quickly in order to get to what we keep telling them is the "meat" of the report. Evidence does not exist in a vacuum - it is inextricably tied to the philosophy and structure of a program - and to the institutional and state restraints put upon it. Reviewers should carefully read the Context section of the report and refer back to it when they are having questions about evidence to see if the program has presented any information that could shed light on their questions. 43

44. More Advice Look for the evidence within the reports; be a detective. Look for evidence from the university that supports the specific NAGC-CEC standard. Know what is acceptable and what is not acceptable evidence. Grammatical and spelling errors in the report: what to do? 44

45. The “Life Cycle” of a Program Report: Six Steps 1. Report is submitted by institution 2. Report is reviewed by review team (in most cases, two reviewers and a lead reviewer) 3. The lead reviewer posts a team report. 4. The SPA audit team usually reviews all team reports and posts an audit report. 5. The audit report is edited by NCATE. 6. The final report is sent to the institution. 45

46. Roles of Reviewer and Lead Reviewer Review teams should use whatever approach works best. The lead reviewer should contact the other reviewers to discuss a strategy immediately after assignments are made. (Note: NCATE will pay for conference calls.) Example 1: Reviewers work independently, the lead reviewer compiles a team report based on composite work and decisions. Example 2: Reviewers work collaboratively to reach decisions and/or write the report – perhaps submitting only a team report. 46

47. Role of Reviewers Judge alignment of assessment and candidate data with the NAGC-CEC standards. Clearly communicate strengths and weaknesses in relation to the standards. Make a judgment with a clear and open mind. Make a judgment based on accepted criteria rather than personal bias. 47

48. Being Objective and Clear The job of the reviewer is not to pass or fail programs, but to make as objective an assessment as possible about the degree to which a given program meets the SPA standards. 48

49. SPA Audit Team Appointed by SPA, usually 6 to 8 experienced reviewers Review to some degree every Recognition Report Will review programs when team cannot reach a decision Will also review reports that have been flagged by NCATE staff

50. The Role of the Audit Team Assure that reports are interpreting standards and applying decisions consistently across programs. Verify comments in team’s report Assume that communications going back to the program is clear and descriptive The audit team is also the “tie-breaker” when a team cannot reach consensus on standard(s) or a recognition decisions.