Learning Outcomes and Assessment APCC

1. Learning Outcomes and Assessment APCC April 2013 Peter Wolf pwolf@uoguelph.ca

5. Scholacticus Exemplaris

6. Outline • Perspectives on developing a programme outcomes culture • Four key questions • Culture change Language: Outcomes, objectives, attributes, expectations…

7. What are the programme outcomes that we want to foster in our students? Where in the curriculum are students encouraged to develop the programme outcomes? How do we know whether our programmes are successful? How do we embed continuous improvement? Key Questions

8. What are the programme outcomes that we want to foster in our students? Where in the curriculum are students encouraged to develop the programme outcomes? How do we know whether our programmes are successful? How do we embed continuous improvement? Key Questions

9. ‘Ideal’ Curriculum Development Process University Learning Objectives ProgrammeOutcomes Areas of Specialization / Foundational Content Educational Experiences Programme Structure Course A Objectives Course B Objectives Course C Objectives Course D Objectives

10. Attributes of the ‘Ideal Graduate’ • University Learning Outcomes • Existing Program Outcomes • Professional Organizations • Accreditation • Other Similar Programs

11. OCAV - Degree Level Expectations • Depth and Breadth of Knowledge • Knowledge of Methodologies • Application of Knowledge • Communication Skills • Awareness of Limits of Knowledge • Autonomy and Professional Capacity http://www.cou.on.ca/content/objects/Undergrad%20Degree%20Expectation%20FINALen1.pdf

12. University of Guelph Learning Objectives (2013) • Critical and Creative Thinking • Inquiry and Analysis, Problem Solving, Creativity, Depth and Breadth of Understanding • Literacy • Information Literacy, Quantitative Literacy, Technological Literacy, Visual Literacy • Global Understanding • Global Understanding, Sense of Historical Development, Civic Knowledge and Engagement, Intercultural Competence • Communicating • Oral Communication, Written Communication, Reading Comprehension, Integrative Communication • Professional and Ethical Behaviour • Teamwork, Ethical Reasoning, Leadership, Personal Organization and Time Management

13. Bachelors of Arts & Sciences Sample Program Outcomes • Pose & solve problems by drawing on & integrating diverse protocols & methods • Oral & written communications, for academic & general audiences • Conduct traditional & electronic research, in humanities/social sciences & scientific contexts

14. Embedding Outcomes OCAV DLE University Outcome Degree Outcome Course Outcome

15. What are the programme outcomes that we want to foster in our students? Where in the curriculum are students encouraged to develop the programme outcomes? How do we know whether our programmes are successful? How do we embed continuous improvement? Key Questions

16. Semester Maps

17. Intentions Map • Survey Questions : • Which instructional methods do you use in this course? • Which assessment approaches do students engage in for this course? • For each listed knowledge, skill and value, please indicate which, if any, you intentionally foster in this course? At what level of sophistication? • For those indicated, please specify how /whether each KSV is taught and/or assessed in this course? • How are the total marks available to students distributed over the course of the semester? • Comments? • Course-by-course info: • Outcomes • Assessment • Learning methods

18. Research Methods Instructional Approach Lecture – 15 Readings – 14 Presentations – 7 Research – 6 Seminar – 4 Groupwork - 3

19. Research Methods Assessments Test/Quiz/Exams – 14 Projects/Assignments – 13 Presentations – 5 Participation - 1

20. Outcome Focus Not Taught; Assessed Civic engagement & advocacy Taught & Assessed: 4 courses Taught; Not Assessed: 1 course Not Taught; Not Assessed: 25 courses Taught; Not Assessed Not Taught; Not Assessed Taught & Assessed

21. Outcome Focus Application of theory in the world Not Taught; Assessed Taught & Assessed: 25 courses Not Taught; Assessed: 1 course Not Taught; Not Assessed: 4 courses Taught; Not Assessed Not Taught; Not Assessed Taught & Assessed

22. What are the programme outcomes that we want to foster in our students? Where in the curriculum are students encouraged to develop the programme outcomes? How do we know whether our programmes are successful? How do we embed continuous improvement? Key Questions

23. Programme Assessment Effective? Efficient? Appropriate? Effective? Efficient? Appropriate? Content Programme Outcomes Assessment Learning methods Effective? Efficient? Appropriate?

24. Four Levels of Evaluation Level 4 Results Level 3 Behavior Level 2 Learning Level 1 Reaction Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

25. Four Levels of Evaluation Key Question: What was the student reaction to learning environment? Timing:Usually done immediately or soon after the learning event(s) Level 1 Reaction Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

26. Four Levels of Evaluation Key Question: Did students achieve desired learning objective(s)? Timing: Usually done immediately or soon after learning Level 2 – Learning Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

27. Four Levels of Evaluation • Key Question: Are newly acquired skills, knowledge, or attitude used by learner after learning event is completed? • Timing: Usually done 1-3 months after learning Level 3 Behavior Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

28. Four Levels of Evaluation Key Question: Did students achieve desired outcomes of program of study? Timing: Usually done 3 months – 2 years after learning experience Level 4 - Results Kirkpatrick, D. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler

29. Programme Assessment • Student work – Tests, assignments, etc. • 360° consultations • Curriculum mapping • External reviews, accreditations, etc. • Benchmarks, research, literature, etc. • Studentreflection – (e)Portfolios, journals, etc • Analytics

30. Student Reflection • 40+ courses • 100’s of assignments, projects, etc • Life experiences • TIME to … • Sort • Reflect • Self-assess

31. What are the programme outcomes that we want to foster in our students? Where in the curriculum are students encouraged to develop the programme outcomes? How do we know whether our programmes are successful? How do we embed continuous improvement? Key Questions

32. Change…

33. Recommended Practices • Data-informed • Faculty-driven • Student-engaged • Stakeholder-informed • Culture- & context-specific • Rigorous & authentic • Sustainable continuous improvement processes • Resourced & recognized A Guide to Developing and Assessing Learning Outcomes at the University of Guelph - http://www.uoguelph.ca/vpacademic/avpa/pdf/LearningOutcomes.pdf

34. Keep in Sight • Refining the blend of people, processes & resources • Transition requires investment • Keep momentum • Continuous improvement is at least as important as ‘reporting’ • Mistakes will happen

35. Worth the Effort! • Superior graduating students • Evidence of graduating student quality • Opportunity for individual student & programme autonomy • Enhanced time & resource usage