SACS Reaffirmation of Accreditation:

1. SACS Reaffirmation of Accreditation: The Assessment of Educational Programs

2. What are we doing and why do we have to do it?

3. Institutional Effectiveness • The institution identifies expected outcomes for its educational programs and its administrative and educational support services; • Assesses whether it achieves these outcomes; • And provides evidence of improvement based on analysis of those results.

4. Assessing Educational Programs • Determine what you want your program’s graduates to know. • Measure that. • Use results to shape program improvement.

5. Is there an easy way to do this?

6. Using Student Assessment Data for Program Assessment • Kill two birds with one stone • Use a process with which faculty are already familiar (student assessment) • Save ourselves some work!

7. Student vs. Program Assessment • Similarities • Unit of measurement (the student or some product produced by the student) • Tools/Instruments of assessment • Differences • Overall purpose of assessment • Primary utilization of results

8. Student Assessment • Purpose – Students assessed with respect to mastery of material presented; ensure capability or competency of each student • Assessment tools – written examinations, essays, papers, presentations, etc. • Use of results – determine grades or initiate some remedial action (tutoring, extra work assignment, etc.) • Unit of remediation – individual student

9. Program Assessment • Purpose – judge and improve overall instructional performance; provide data for making program improvements • Assessment tools – written examinations, essays, papers, presentations, etc. BUT, data are aggregated • Use of results – program improvements; ensure that the program is accomplishing its objectives • Unit of remediation – program

10. Program Remediation Strategies • Curricular changes • New courses • Revised courses • Revised course sequence • Admission criteria changes • Instructional methodology changes

11. Determine learning objectives ↓ Design core curriculum to teach objectives ↓ Test students on objectives ↓ Use test results to improve their program The Faculty:

12. What’s this capstone stuff the Provost is talking about?

13. Capstone Course • End of the program • Comprehensive • Exams can be program assessments

14. Sociology Learning Objectives Theory Research Methods Statistics 1. Courses on theory, methods, & statistics • Test theory, methods & statistics during capstone course • Analyze aggregate test responses • Use analysis for improvement

15. Average Percent Correct

16. Analyze the Problem Only 10 of 50 theory questions answered correctly Classical (25) Contemporary (25) 2 correct 8 correct

17. Solve the Problem • Devote more time to classical theorists in required theory class • Add a second required class exclusively on classical theory • Lower expectations for student learning

18. Statistics • Program-BS in Statistics • Five outcomes are identified. • Comprehensive exam in capstone course will be used to measure these outcomes. • This process combines student assessment with program assessment.

19. Outcomes of Statistics Program • Knowledge of sampling and descriptive statistics. • Knowledge of estimation techniques. • Knowledge of statistical inference. • Knowledge of modeling techniques. • Ability to apply statistics.

20. Statistics Comprehensive Capstone Exam • Link exam questions to outcomes • Questions 1 – 10 assess Outcome 1 • Questions 11 – 20 assess Outcome 2 • Questions 21 – 30 assess Outcome 3 • Questions 31 – 40 assess Outcome 4 • Questions 41 – 50 assess Outcome 5

21. Comprehensive Statistics Exam • Given in Capstone Course. • Consists of 50 multiple choice questions worth 2 points each. • Criteria for success: • Average student score will be greater than 69. • Average outcome score will be greater than 13 points.

22. Exam Results

23. But my discipline is more complicated and nuanced than Sociology or Statistics. We use ESSAY EXAMS.

24. Essay as Means of Student Assessment • Course objective is to evaluate a student’s ability to provide a persuasive argument that clearly supports a given resolution to an ethical dilemma. • Means of Assessment • Written essay given in a course at the senior level or other appropriate point in the curriculum • Criterion for Success – Individual Student • Determined by the course instructor

25. Essay as Means of Program Assessment • Assume ethics is identified as a programmatic outcome. • Outcome – Ethics • “Graduates of the program will be able to provide a persuasive argument that clearly supports a given resolution to an ethical dilemma.”

26. Essay as Means of Assessment • Criteria for Success – Program • Average score (as determined by a panel of faculty) will be at least 70 out of a possible 100 points on a rubric designed to score the written essay • On no individual rubric category will the average score be less than 60% of the possible points

27. Essay as Means of Assessment • All essays need not be scored for program assessment; a sample may be sufficient. • A group of faculty should be involved in scoring the essays. • The same rubric used to score each essay for individual student assessment may also be used in program assessment.

28. Essay as Means of Assessment • The same student product (essay) is used twice: once for individual student assessment and again for program assessment. • Course-imbedded assessment overcomes student motivation problem.

29. Essay as Means of Assessment • Key: • The means of student assessment (e.g. essay) MUST relate to or measure in some way the PROGRAM OUTCOME!

30. We are a more applied and practical discipline. We have INTERNSHIPS.

31. Internship as Means of Assessment • Internship experience required in many programs • Extremely useful for student and program assessment • Relies on “external” evaluator

32. Internship as Means of Assessment • Key Issue • Feedback MUST relate to outcomes to be useful in program assessment • Standardized assessment form tied to program objectives (rubric) • Can develop overall program criteria for success with aggregated data • Recognize “multi-rater” consistency issues

33. What about PERFORMANCES or ORALS EXAMS?

34. Juried Performance/Oral Exam as Means of Student Assessment • Programs such as those in music and art frequently require senior recital or exhibition. • Most graduate programs and some undergraduate programs incorporate oral exams. • Use a standardized collection instrument tied to program objectives – data collection, performance evaluation.

35. Music - Juried Performance • Outcome: • “Program graduates will be able to deliver a polished and technically correct performance, i.e., student is expected to demonstrate proficiency in tone, intonation, accuracy, rhythm, technique, and interpretation or phrasing.” • Means of assessment: • rubric completed by jury of evaluators

36. Music - Juried Performance • Criteria for student success: • Example: Student must achieve average rating of 3.0 over all 6 elements in order to pass final performance course. • Criteria for program success: • Average ratings will be at least 3.0 across all elements on the rubric. • On no element will the average be less than 2.5.

37. Music - Juried Performance

38. Regardless of the testing method…

39. Determine learning objectives ↓ Design core curriculum to teach objectives ↓ Test students on objectives ↓ Aggregate test results to improve their program The Faculty:

40. But we have to have multiple assessment methods.

41. Other assessment methods Icing on the cake • Focus groups • Exit surveys • Alumni surveys • Licensure exams

42. Importance of Assessment • Perhaps as much as half our total SACS efforts and time will be related to assessment and institutional effectiveness • Should not be “busy work” • Work towards making it a permanent, ongoing process

43. Van Gray Associate Vice President, Strategic Planning and Improvement Phone (254)710-8468 Tom Bohannon Assistant Vice President, Institutional Research and Testing Phone (254)710-2061 Gina Green Assistant Professor of Information Systems Phone (254)710-6210 Tiffany Hogue Assistant Provost Phone (254)710-3545 Tim Logan Director, Electronic Libraries Phone (254)710-6665 SACS Task Force Larry Lyon, Senior Vice Provost, Chair of SACS Task Force, Dean of the Graduate School, Phone (254)710-3588 http://www.baylor.edu/sacs sacs@baylor.edu