Setting graduate capability standards and evaluating outcomes

1. Shining a light on the ‘new black’ of Higher EducationMark Reedman, Design for Learning Project Manager, Office of the DVC(A)1st June, 2012

2. A short time ago in an Institution not that far away…. It is the period pre-TEQSA. The Bradley Review of Higher Education has been accepted by the Government. During the review, one University develops an ambitious plan to to improve the quality of undergraduate student engagement, learning, and academic success…. TEQSA • Setting graduate capability standards and evaluating outcomes

3. Design for Learning at La Trobe • La Trobe University Setting graduate capability standards and evaluating outcomes • Design for Learning (DfL) is an: • Ambitious project that directly supports La Trobe University’s Strategic Objective 1 to “Transform student lives through learning”. • Institution-wide effort involving the review and, where necessary, redesign of the Bachelor-level curricula and of related student engagement (First Year Experience) and student academic support services

4. Setting Graduate Capability Standards and Evaluating Outcomes • La Trobe University Setting graduate capability standards and evaluating outcomes • In relation to Graduate Capabilities, La Trobe will: • Identify and define measurable Graduate Capabilities (GCs). • Identify, develop and embed learning activities and assessment tasks for evaluating GC outcomes into the normal activities and assessment tasks associated with content. • Develop transparent and measurable evaluation standards for student achievement at three quality levels – Standard Not Met, Standard Met and Standard Exceeded – for each GC at three points in a course - Cornerstone, Mid-pointand Capstone. • Develop rubrics to allow easy evaluation of those quality levels as part of the normal marking of assessment tasks with easy provision of feedback to students

5. Setting Graduate Capability Standards and Evaluating Outcomes 1 Barrie, S. (2005). Rethinking generic graduate attributes. Higher Education Research and Development Society of Australasia News, 27(1), 1–6 • La Trobe University Setting graduate capability standards and evaluating outcomes • In 2005, Simon Barrie1 noted: • “ . . . every university in Australia now has a statement of the generic attributes of its graduates. However, one wonders if every graduate of each of these universities lives up to the rhetoric of these statements1”

6. What are the core GCs applicable to all disciplines which can actually be explicitly evaluated? • La Trobe University Setting graduate capability standards and evaluating outcomes • Writing • Speaking • Critical Thinking • Problem-solving • Teamwork • Inquiry/research • Ethical Practice/Awareness • Social Responsibility • Global Citizenship/Perspectives • Sustainability • Lifelong Learning

7. What standards should Graduates be expected to achieve by the end of their course and how can those standards best be developed? • La Trobe University Setting graduate capability standards and evaluating outcomes

8. How can student’s best be assisted to meet those standards within existing curriculum? • La Trobe University Setting graduate capability standards and evaluating outcomes

9. How can student’s best be assisted to meet those standards within existing curriculum? • La Trobe University Setting graduate capability standards and evaluating outcomes • The ‘Stairway to Success’ Model

10. At what level in the Institution can GCs be agreed and implemented? Based on the American Association of Colleges and Universities. Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics. http://www.aacu.org/value/ • La Trobe University Setting graduate capability standards and evaluating outcomes

11. What should the relationship be between GC Standards and the AQF Level Standards? Adapted from: Australian Qualifications Framework, First Edition July 2011. Bachelor Degree qualification type descriptor, p46. • La Trobe University Setting graduate capability standards and evaluating outcomes

12. Are the policies, procedures and guidelines in place to support the intended outcomes? Policies can be found at: http://www.latrobe.edu.au/policy/all-policies • La Trobe University Setting graduate capability standards and evaluating outcomes • Policies, procedures and guidelines are boring! • Helps to support an Institutional approach. • Helps to provide direction. • Helps to describe requirements. • Graduate Capabilities – definitions; standard setting; embedding; evaluation; recording; and reporting. • Curriculum Design – base requirements; incorporating ILOs & GCs; T&L resources; assessment; feedback; accessibility • A policy framework is essential to make things happen in conjunction with Senior Management support and Faculty/Discipline Champions.

13. In which assessments can we embed GC evaluation? 2 Biggs, J., & Tang, C. (2007). Teaching for Quality Learning at University : What the Student Does. Maidenhead, UK: Society for Research into Higher Education: & Open University Press. • La Trobe University Setting graduate capability standards and evaluating outcomes • Evaluation of GCs must: • be part of normal activities and assessment. • be embedded within and evaluated as part of the normal assessment of discipline content. • Requires Curriculum Mapping • Requires ‘Constructive Alignment’2 of assessment, activities, Subject Intended Learning Outcomes, Course Intended Learning Outcomes, and the GC descriptors. • Students will not take GCs seriously if it is an ‘add-on’. • Staff will not accept evaluating GCs if it will lead to more work (e.g., double-marking) and is not seen to benefit students.

14. What feedback should students receive? • La Trobe University Setting graduate capability standards and evaluating outcomes • Feedback should be meaningful for students but not necessarily onerous for staff. • Students should be provided with grading rubrics prior to submission or completion of any assessment task in which GC evaluation is embedded. • How GC standard is evaluated must be clear and transparent (or students will appeal!). • Feedback will depend on the rubrics developed to assess the discipline-specific content as evaluation of any GC standard should be part off the same rubric so it is marked at the same time. • Feedback may simply extend to the GC outcome being displayed in the LMS along with the Grade for the assessment, and the completed grading rubric being provided back to the student.

15. What if a student is evaluated for the same GC in more than one subject (unit)? • La Trobe University Setting graduate capability standards and evaluating outcomes • Part of initial framework discussion and the policy discussion, and is dependent upon the level at which the standard is set and interpreted. • The standard set for a given GC in one discipline is likely to be different to another (e.g., the expected ‘writing’ standard will be different in sociology from that expected in accounting). • A student undertaking a Bachelor of Accounting who takes a sociology subject should be provided in the sociology subject with feedback associated with the expected standard for ‘writing’ in sociology • The same student should be provided in the accounting subject with feedback associated the expected standard for ‘writing’ in accounting.

16. How should outcomes be recorded and reported? • La Trobe University Setting graduate capability standards and evaluating outcomes • Is it enough to simply show the constructive alignment of assessment & ILOs for us to say that a student can demonstrate a skill to a particular level? • Recording outcomes should be at the subject level in the same way and at the same time the discipline-specific content grade is recorded. • How the achieved outcome translates to some sort of transcript or record is dependent upon the level at which the standard is set and interpreted. • If set at a discipline-level, then providing a record of achievement against the standards set for that discipline (Major) would be appropriate. • A record of some sort (perhaps in the Australian Higher Education Graduate Statement) could provide students with a copy of the standards and evidence of what they can demonstrate.

17. What are the benefits of this recording approach? • La Trobe University Setting graduate capability standards and evaluating outcomes • Graduates will be able to provide through a record of achievement, evidence of the generic skills they can demonstrate and will be provided with matrices detailing the standard(s) or quality to which they can demonstrate those skills. • Employers can also be provided with these easily understood matrices of demonstrable skills and can identify in a timely manner, whether or not the Graduate has and can perform the generic skills to the standard required by the employer. • Staff get (hopefully) better students because of formative feedback mechanisms and scaffolding. ‘At Risk’ students can be identified through this mechanism and provided with appropriate support. • Management get to prove to TEQSA (or other) that quality standards are embedded within curriculum and can prove the number of students able to demonstrate skills to those standards. The ‘At Risk’ identification also provides a mechanism for increasing retention and participation.

18. QUESTIONS? Mark Reedman Tel: +61 3 9479 6496 Email: m.reedman@latrobe.edu.au Web: www.latrobe.edu.au/ctlc/dfl