Development, Implementation, and Effective Use of the Unit Assessment System at the University of Mississippi

1. Development, Implementation, and Effective Use of the Unit Assessment System at the University of Mississippi Dr. Kaye Pepper Dr. Fannye Love Mrs. Smitty Horne-Wood

2. Overview of University of Mississippi and the School of Education • University of Mississippi • One of 8 public universities in the state • Located in north Mississippi • Carnegie research extensive institution • Has experienced continuous growth • School of Education • Two departments • Department of Curriculum and Instruction • Department of Leadership and Counselor Education • Initial and advanced programs, including an alternative certification program • Candidates in 2006-2007 - 940 initial; 589 advanced • Faculty - 51 full-time and 14 part-time

3. The assessment system must collect and analyze data on: • Applicant qualifications • Candidate and graduate performance • Unit operations Results are used to evaluate and make data-driven decisions which improve the unit and its programs.

4. Development of the Unit Assessment System

5. Developing the Unit Assessment System at UM • Developed in 2001 with assistance of professional partners and has undergone continuous improvement • Designed to link performance of our candidates to: • The conceptual framework • National, state, and program standards • K-12 student learning

6. The Assessment Committee - 2001: • Charged with developing an assessment and evaluation system that: • Met all NCATE assessment and evaluation standards • Was feasible and useful to the SOE in monitoring candidate performance and improving SOE programs Included representatives from each department in SOE and professional partners

7. Unit Assessment System Design - School of Education - University of Mississippi Unit Evaluation Candidate Assessment Program Evaluation Curriculum Audit Database • Candidate Assessments within Courses Consisting of • Conceptual Framework Outcomes and associated • Course objectives • Assessments • Scoring Tools • SPA standards • Reflection Elements • Unit-wide data • Faculty evaluations • Student, staff, faculty diversity • Governance/ • Leadership Review of instruments for fairness, consistency accuracy, avoidance of bias • Decision Point Assessments: • Program Entry • Intermediate • Program Completion • Follow-Up Analyzed to develop Used in Feedback to individual candidates and to faculty for decision making purposes Data entered into CurriculumAlignment Audit Changes needed to improve unit performance Candidate Assessment Database containing individual candidate data Resulting in Used for Program improvement plans developed and implemented by program faculty and partners Reports presenting aggregated summaries of candidate strengths and weaknesses And for Analyzed to develop Summaries of formal candidate complaints and their resolution Individual program summaries included in Analyzed to produce Compilation of outcome data from all programs (including recruitment and retention data) • External data from • Employer surveys • Graduate surveys • Licensure tests • Results of research on student outcomes

8. Levels of the Assessment System Individual data on essential candidate knowledge, skills, and dispositions Aggregated candidate data and results of curriculum audit used to assess programs Program and other data used to assess the unit

9. To accomplish goals in 2001, the Assessment Committee: • Was divided into subcommittees by program areas (elementary, secondary, special education, educational leadership, counselor education) • Subcommittee chairs worked with program faculty and members of the professional community to provide the information needed to build the databases and in establishing the assessment system components at initial and advanced levels.

10. Components of the system developed for each program / level • Alignment of course content/objectives with the SOE conceptual framework • Selection of decision-point assessments • Committee validation of decision-point assessments • Identification of external assessments and other sources of unit data

11. Components of the system (cont.) • Development of timelines, policies, and procedures for • candidate mid-program and program completion reviews • program review • unit review • Compilation of outcome data on candidate performance on decision-point assessments • Compilation of data in aggregate form for program evaluations

12. Evolution of the Candidate Assessment Database • Consists of decision-point data for each candidate in each program • 2002-2005 was a period of alternating development and updating of the database design. • In 2005, decision point data entry was completed and reports were generated at the program level • In Spring 2006, initial programs were transitioned into Tk20. • Advanced programs are scheduled to transition to Tk20 by spring 2008.

13. Curriculum Audit Database • Consists of all course objectives for each SOE course. Curriculum Alignment Audit every 2 years • Each objective is linked to: • Measure of learning addressed (knowledge, skills, or disposition) • Relation to the conceptual framework • SPA standards relevant to the objective • Indication of whether the objective entails candidate reflection • General content and form of the assessment, including reference to the scoring tool • Program changes related to additions/deletions and reasons for the changes

14. Systematic Process Flowchart for Decision-Making Process Internal/External data Director of Assessment Annual Reports developed Assessment Committee Dean’s Leadership Team Program/Dept Faculty and Staff Review Reports / Plans for Improvement Teacher Ed Committee Unit Evaluation Committee Dean Strategic Planning Implementation of Changes

16. Important Documents • Faculty Unit Assessment Handbook • Overview of the Unit Assessment System • Frequently Asked Questions • Copy of each decision-point assessment for each program • Annual Individual Program Assessment Report Notebooks • Copy of decision-point assessments • Aggregated results of decision-point assessments and data from other sources • Program assessment reports • Program Plans for Improvement

17. Implementation of the Unit Assessment System

18. Oversight of the Unit Assessment System • Fall 2005: a Director of Assessment and a Database Manager were hired to facilitate the assessment process. • Responsibilities Director of Assessment include: • Management of data collection • Compilation, summarization, and analysis of data • Report development • Disseminates reports to program faculty, department chairs, assessment committees, and professional community for development of plans for improvement

19. Oversight (cont) • Responsibilities of Database Manager include: • Tk20 system setup each semester • Training for candidates/faculty on Tk20 use • Request reports from Tk20 • Development and maintenance of other databases

20. Data Collection • Candidate Assessment Database – capture of decision-point assessment results • Initial Programs – use of Tk20 • Advanced Programs – results entered manually into spreadsheets • Curriculum Alignment Audit – Summary of additions/deletions to programs developed • External sources of data – Summarized in reports

21. Data Analysis • Candidate Assessment • Decision-point Assessments (entry, intermediate stage, and exit) • GPA • Standardized test results • Course-based and noncourse-based assessments • Dispositions • Follow-up surveys

22. Data Analysis (cont.) • Program Evaluation • Decision-point assessments organized by outcomes of the conceptual framework • Candidate formal complaints and resolutions • Input from candidates, school/university partners, employers, and program faculty • Curriculum Alignment Audit • External reports (SPA reports)

23. Data Analysis (cont.) • Unit Evaluation • Program Evaluation Reports • Assessments with “like-items” for all programs in the SOE • Input from candidates, school/university partners, employers, and SOE faculty • Candidate formal complaints and resolutions • External reports (External Review / MDE Process and Performance Review) • Unit leadership and governance reports (faculty evaluations, diversity, technology, etc)

24. Program Plans for Improvement • Developed by program faculty/staff with assistance from members of the Assessment Committee and Dean’s Leadership team • Final plan is reviewed and approved by the Teacher Education Committee • Dean/Department Chairs prepare for strategic planning to implement changes

25. Unit Plan for Improvement • Developed by members of the Assessment Committee and Dean’s Leadership team • Reviewed by SOE faculty/staff for input • Final plan is reviewed and approved by the Unit Evaluation Committee • Dean/Leadership Team prepares for strategic planning to implement changes

26. Ongoing Activity • Assessment is a never-ending, cyclical process • Annually • Decision-point assessment data • Formal candidate complaints and resolutions • Other internal data (faculty evals, etc • Results of external assessments(MDE • Every 2 years • Curriculum Alignment Audit • Every 3 years • External Review

27. Systematic Process Flowchart for Decision-Making Process Internal/External data Director of Assessment Annual Reports developed Assessment Committee Dean’s Leadership Team Program/Dept Faculty and Staff Review Reports / Plans for Improvement Teacher Ed Committee Unit Evaluation Committee Dean Strategic Planning Implementation of Changes

28. Results of an effective, systematic assessment process allows for data-driven decision-making which results in continuous improvement