Mission Possible : Assessing graduate and professional Programs

1. Mission Possible: Assessing graduate and professional Programs Dr. Timothy S. Brophy Director of Institutional Assessment University of Florida Gainesville, FL

2. Today’s Goals • Part 1: To introduce, describe, and explain the basic elements of student learning outcomes and program goals, their development, and measurement • Part 2: To share a structure for assessment planning and reporting for graduate and professional programs and review an example • Part 3: Review and discuss graduate sample academic assessment data reports

3. Common Challenges

4. How accreditors define educational programs

5. Expectations for academic assessment for accreditation

6. Expectations for academic assessment

7. Part 1: Student Learning Outcomes, Program Goals, and Outputs

8. Define and disseminate the terms

9. consider a categorical organizing framework for SLOs

10. Thee characteristics of SLOs:Recency, Relevance, and Rigor

11. distinguish Outputs from outcomes • An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement - quantification - of performance. • Outputsdescribe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output.

12. Outcomes and outputs: What is the difference?

13. Exercise 1: Are these results statements Outputs or outcomes?

14. Our program graduated 25 students in Spring 2013. • Output • Outcome

15. 75% of our students achieved Level 4 (out of 5) on our presentation assessment rubric. • Output • Outcome

16. We recruited 10 additional students in 2013-14. • Output • Outcome

17. In 2013, Our doctoral students published 10 papers in The International Journal of Psychology. • Output • Outcome

18. distinguish SLOs and Program Goals • Program Goals describe the unit’s expectations for programmatic elements, such as admission criteria, acceptance and graduation rates, etc. • Student Learning Outcomes (SLOs) describe what students should know and be able to do as a result of completing an academic program. • Program faculty set targets for their SLOs

19. Exercise 2: Student learning outcomes or program goals?

20. We will lower our attrition rate to 10%. • Program Goal • Student Learning Outcome

21. Music Education Students discriminate musical quality based on sound musical reasoning. • Program Goal • Student Learning Outcome

22. We will reduce the average time to degree in our graduate program from the 2012-13 rate of 6.5 years to 5 years in 2014-15. • Program Goal • Student Learning Outcome

23. Students analyze experimental data and interpret results in the cellular and molecular sciences. • Program Goal • Student Learning Outcome

24. ensure the outcome is measurable EFFECTIVE SLOS: • Focus on what students will know and be able to do. • All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings. • Describe observable and measurable actions or behaviors. • Effective SLOs present a core set of observable, measureable behaviors. Measurement tools vary from exams to complex tasks graded by rubrics. • The key to measurability: an active verb that describes a observable behavior, process, or product • A framework for developing SLOs: Bloom’s Taxonomy

25. Verbs and phrases that complicate measurability

26. Developing Measurable SLOs: A Three-level Model (carriveau, 2010)

27. Exercise 3: Are the following outcome statements measurable?

28. Students understand good writing style. • Yes • No

29. Students sight-sing a 16-measure melody with no errors. • Yes • No

30. Students explore and learn about geological formations. • Yes • No

31. Students Define the ethical responsibilities of business organizations and identify relevant ethical issues. • Yes • No

32. Balance Direct and Indirect Assessments • Indirect assessments are those that ascertain the opinion or self-report of the extent or value of learning experiences (Rogers, 2011) • Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators.

33. Exercise 4: Direct or indirect Assessments?

34. Course Final exam • Direct • Indirect

35. SERU or NSSE survey data. • Direct • Indirect

36. Final paper, Performance, or Presentation graded by a faculty developed rubric. • Direct • Indirect

37. Senior exit interview. • Direct • Indirect

38. Part 2: Planning and Reporting

39. Academic Assessment Planning

40. Why Plan?

41. Graduate and Professional program Assessment Plan

42. Academic Assessment Plan Approval Process

43. Sections of the plan Template: Figure 4 – pp. 3-4 in your handout Rubric: Figure 1, p. 2

44. Mission

45. Student learning outcomes • The complete file of graduate and professional program SLOs is on the Institutional Assessment website – http://assessment.aa.ufl.edu • UF’s Graduate SLO categories are Content, Skills, and Professional Behaviors • Online resources at the UF Institutional Assessment website: • “Writing Measurable Student Learning Outcomes” PowerPoint • “Guide to Writing Student Learning Outcomes”

46. Research

47. assessment timeline

48. Assessment timeline Program_____________________________________College ____________________________________

49. assessment Cycle

50. Assessment Cycle Program College _ Analysis and Interpretation: [Enter date or time frame here] Improvement Actions: Completed by [Enter date here] Dissemination: Completed by [Enter date here]