Assessing and Assuring Graduate L earning Outcomes

1. Assessing and Assuring Graduate Learning Outcomes A practical workshop on assessment with alverno college Sydney 28 - 29 May 2013

2. Collaborative national project supported by ALTC / OLT with strategic priority funding • What are the assessment tasks in a range of disciplines that generate convincing evidence of achievement of graduate learning outcomes? • What are the assurance process trusted by disciplines in relation to those assessment tasks and judgements? Situational analysis, literature review, expert reference group, institutional visits, interviews with 48 academics 28 universities in 7 disciplines Ten key issues papers, endnote library, submissions to government discussion papers, research report – 5710 unique page views…and counting. Full story at: http://www.itl.usyd.edu.au/projects/aaglo/

3. Some AAGLO Headlines1. The nature of graduate learning outcomes 1. Coherent assessment of program level graduate learning outcomes requires coherent institutional and discipline statements of outcomes - but what that statement ‘is’ matters. • Notwhat skills are included – but the nature of the skills, and what that says in terms of traditional or contemporary conceptions of knowledge. Some of the ways we have articulated GLOs are limiting - taking us backwards to outdatedexpectations of the goals of higher education – not forwards to the contemporary outcomes and purposes we espouse.

4. Some AAGLO Headlines2. Assessment choices 2. What we assess and how we assess it shapes curriculum and teaching as much as it does learning; we do not have to assess everything that is taught but: • Assessment prioritizes teaching of some GLO and ignores others • Communication skills – privileged • Information literacy - privileged • Research and inquiry – (less) privileged • Ethical social professional understandings - neglected • Personal intellectual autonomy - neglected Consequences for students

5. Some AAGLO Headlines3. Assessment design 3. Core features of effective assessment practices identified by respondents • interconnected multi-component • authentic, relevant • standards-based with effective communication of criteria • involve multiple decision makers – including students • …..coupled with effective assurance processes around the quality of these tasks and the judgments made

6. Some AAGLO Headlines4. Quality Assurance of Assessment 4. Variety of assessment quality assurance strategies • Quality of task: Pre-implementation (policies, curriculum approval processes for new and revised tasks) Post- implementation (evaluation by students, examiners, peers, curriculum audit & review) • Quality of judgment: Pre-judgement - (calibration) Post-judgement – (consensus moderation) • Confidence? Combination of effective task and assurance was less common than it could be Evidence of ‘assurance’ engaging staff and leading to ‘enhancement’ was rare

7. FUTURE RECOMMENDATIONS • Recommendation 1: That Universities review their choice of assessment tasks in programs and units of study to ensure these include design features that provide convincing evidence of achievement of GLOs. • Recommendation 2: That universities provide professional learning opportunities and support for academics to develop effective discipline based assessment for the assurance of GLOs 3,4,5,6,….. • Recommendation 7: That the OLT continue to support initiatives that engage academics in productive, evidence-based dialogue around the issue……

8. Thank you to the OLT for their wonderful support in bringing you Alverno College! Introducing Kathy Lake - Alverno JeanaAbromeit - Alverno Clair Hughes – AAGLO

9. From Principles to Practice Jeana Abromeit Associate Vice President for Academic Affairs, Professor of Sociology Kathy Lake Associate Vice President for Academic Affairs, Professor of Education Alverno College

10. Outcomes for Session • To have a sense of each other’s educational principles and their meaning for current and future practices -- both unique and shared • To have an initial understanding of Alverno’s educational program to start a meaningful conversation • To explore principles for design of practice: curriculum, teaching, learning, and assessment

11. Shared Values and Principles Our Goal: Reach agreement on a set of important underlying principles of higher education

12. Individual and Group Activity Principles I hold: Discuss your list with others at your table. Principles which our group holds: Discuss shared principles with all participants

13. In 1970, one of the questions the President asked each department to address: • What are you teaching that is so important that students cannot afford to pass up courses in your department? Which led to questions like: • How do you want your students to be able to think as a result of studying your discipline? • What do you want your students to be able to do as a result of studying your discipline?

14. In 1973, new curriculum initiated based on eight institutional abilities or outcomes • Graduation requirements based on demonstration of outcomes rather than distribution requirements.

15. PRINCIPLE Education GOES BEYOND Knowing to being able to DO WHAT ONE KNOWS

16. PRACTICE • Performance in the context of content

17. PRINCIPLE Educators are RESPONSIBLE for making learning more available by ARTICULATING OUTCOMES and making them PUBLIC

18. PRACTICE • Effective Communication • Analysis • Problem Solving • Valuing in Decision Making • Social Interaction • Developing a Global Perspective • Effective Citizenship • Aesthetic Engagement

19. Abilities that • Involve the whole person • Are teachable • Can be assessed • Transfer across settings • Are continually re-evaluated and • re-defined

20. PRINCIPLE Abilities need to be defined in a way that our teaching of them can be DEVELOPMENTAL

21. The Alverno Impact • As you watch the clips of student performance, how have students changed? • Students are being assessed for communication ability. What other abilities are also demonstrated?

22. PRACTICE Analysis • Level 1 – Show observational skills • Level 2 – Draw reasonable inferences • Level 3 – Perceive and make relationships • Level 4 – Analyze structure and organization • Level 5 – Employ frameworks from major and support disciplines in order to analyze • Level 6 – Independently employ frameworks

23. PRINCIPLE Designing for learning involves INTEGRATINGabilities with disciplines and across disciplines

24. PRACTICE • Coherent curricular design – Sociology example • Fusion of content and abilities • Self Assessment and Feedback

25. PRACTICE in AUSTRALIA • Draft Standards for Learning Outcomes from Australian Higher Education Standards Panel: Standards 3, 4 & 6

26. PRACTICE Examples from Outcomes for Chemistry • Communicates effectively, using language, concepts, and models of chemistry • Uses methodology of chemistry to define and solve problems individually and collaboratively

27. PRACTICE Examples of Outcomes for Business and Management • Uses discipline models and theories to analyze interdependence among systems, organizations, individuals, and events (Systems Thinking and Analysis) • Applies business and management principles to develop and deliver quality products or services (Enterprising and Problem Solving) • Uses team and organizational skills to work effectively with diverse individuals, teams, and organizational units to meet stakeholder and organizational goals (Interacting and Leading)

28. PRINCIPLE Abilities must be carefully IDENTIFIEDand COMPARED to what CONTEMPORARY LIFErequires

29. PRACTICE • Ability departments • Scheduled Institutes • Ongoing revisions of outcomes

30. PRINCIPLE INTEGRAL to LEARNING is ASSESSMENT

31. Institutional and Program Assessment Student Assessment-as-Learning

32. Student Assessment-as-Learning • A process in operation at Alverno College, integral to learning, that involves observation and judgment of each student’s performance on the basis of explicit criteria, with self assessment and resulting feedback to the student. • It serves to confirm student achievement and provide feedback to the student for the improvement of learning and to the instructor for the improvement of teaching.

33. Student Assessment-as-Learning Essential to learning and assessment are: • Public, Expected Outcomes/Criteria • Performance • Feedback • Self Assessment

34. Institutional and Program Assessment • Processes that yield patterns of student and alumna learning, development, and performance on a range of educational outcomes. They provide meaningful feedback to faculty, staff, and various publics for improvement, shared learning, and demonstrated effectiveness. • These processes ensure comparisons to standards (faculty, disciplinary, professional, accrediting, certifying), and enable evidence-based judgments of how students and alumnae benefit from the curriculum and college culture.

35. PRINCIPLE The effectiveness of assessment for everyone involved depends on the existence of a total dynamic system that contributes to the coherence and continuous improvement of the curriculum

36. PRACTICE • Matrix structure • Discipline departments • Office of Research and Evaluation

37. INSTITUTIONAL/PROGRAM ASSESSMENT EXPECTED OUTCOMES STUDENT LEARNING (INSTITUTION: PROGRAM/COURSE: COURSE/UNIT) MISSION ASSESSMENT TEACHING EDUCATIONALPRINCIPLES

38. as . sess ( ses’) v.t.[late ME <ML <L assess (us) ptp. of assidere (ad + sedere)]to sit down beside e e

39. The Role of Outcomes in Creating a Coherent Curriculum

40. Goals For This Session Consider the relationships among institutional outcomes, program outcomes, course outcomes, and criteria.

41. Definition of Outcomes Outcomes are a set of statements that describe what a student can do with what s/he knows as a result of a set of learning experiences.

42. Characteristics of Outcomes • Generic abilities, not tasks • Multidimensional and holistic • Cognitive, affective and kinesthetic dimensions • Transferable

43. Sources of Outcomes • Institutional mission • Accreditation standards • Professional requirements • Needs of society • Nature of Discipline • Aspirations of learners

44. Increasing Degrees of Specificity Institutional Outcomes Major Outcomes & Accreditation Standards Course Outcomes Assessment of Course Outcomes Criteria (more specific than Outcomes) Judgment of Student Performance Using Criteria

45. Benefits of Using Outcomes • Provide direction for learning • Provide continuity • Provide departments with standards to judge university level learning • Assure accountability by making teaching/learning public and explicit

46. More Benefits of Using Outcomes • Help students transfer learning because learning experiences are embedded in multiple performance contexts • Foster cross-discipline communication: creates a community of faculty with common goals

47. Continuum of Increasing Specificity • Outcomes become more specific and contexualized the closer they come to the student and his or her performance • Criteria are outcomes at their most specific and most contextualized

48. Connections to the Disciplines • Outcomes help students translate experience into learning • Outcomes help facilitate efficient, coherent, consistent assessment procedures

49. Examples of Outcomesfor English • Communicates an understanding of literary criticism, questions its assumptions, and uses its frameworks to analyze and evaluate works. • Reads and interprets diverse cultural expressions in works of literature, film, and other media.

50. Examples of Outcomes for Computing and Information Technology • Effectively uses computing frameworks, both independently and collaboratively, to analyze, develop, implement, and evaluate solutions to diverse problems. • Appling the principles of usability to the design and implementation of components, products, and systems.