Assessment Standards, Feedback & Capacity: Carpe Diem BMAF Conference 21 April 2010

1. Assessment Standards, Feedback & Capacity: Carpe DiemBMAF Conference 21 April 2010 A/Professor Mark Freeman Learning and Teaching Academic Standards Project The Australian Learning and Teaching Council has received funding from the Australian Government for this project. The views expressed in this material do not necessarily reflect the views of the Australian Learning and Teaching Council or the Australian Government.

2. Australian context • Big country = 31 x UK • HE is 3rd largest export • ~200 HEI with 25% uni • Uniself-accrediting & mostlylarge, publicly funded • Quality regulator (~QAA) • Active deans council (~ABS) • Quality support org (~HEA) = ALTC

3. Business, Management and Economics

4. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

5. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

6. Roles of assessment Assessment is the "making of judgements about how students' work meets appropriate standards" (Boud, 2010) • “always defines the actual curriculum” (Ramsden, 03) • generates feedback to close the performance gap (Nicol & McFarlane-Dick, 2006) so design for more frequent and specific (Gibbs & Simpson, 2004) • enduring capability to make judgments about the standard of one’s own and others' work (Boud, 2000)

7. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

8. Student Experience: CEQ 2006-8 Assessment % Agreement • Source: www.graduatecareers.com.au

9. Student Engagement: AUSSE 2008 Received prompt written/oral feedback from teachers/tutors on academic performance Discussed your grades or assignments with teaching staff Source: ausse.acer.edu.au

10. Funding constraints I don’t see any effect of reduced real funding. Neither do I. • Double Blind study reveals NO impact of reduced funding on standards Adapted from avilian.co.uk/wp-content/uploads/2009/11/cartoon.jpg

11. Expand HE for economic growth • More & diverse • students • providers

12. Tougher new regulator

13. Greater transparency and scrutiny Plans for ‘MyUniversity’ website Similar to ‘MySchool’ website • Released in Feb 2010 • Covers ~10,000 Australian schools • Publish nationally comparable data • Popular with parents

14. Media spotlight Standards: MPs upbraid sector 6 August 2009 …more 'Radical change' is needed to reassure public on standards 1 October 2009 …more

15. Learning outcomes not measured

16. Peer discussion: Agree/Disagree? “We have evidence that could convince critics that our programmes only graduate those that meet expected programme outcomes” For example, the ability to effectively • communicate • work in teams • apply knowledge critically to disciplinary problems HAND UP if agree

17. International trends

18. Assessment reform • Assessment standards: a Manifesto for Change and Feedback: an Agenda for Change (ASKe, 2007) • Assessment 2020 “Universities face substantial change in a rapidly evolving global context. The challenges of meeting new expectations about academic standards in the next decade and beyond mean that assessment will need to be rethought and renewed” (Boud & Associates, 2010)

19. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

20. Literature review • Changing students understanding of assessment requires active engagement (O’Donovan et al, 2003) • Change in HE is slow (Elton, 2003) • Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke, 2004) • Cultivate an assessment community of practice through informal exchanges seeded by specific activities (O’Donovan et al, 2008) • Engage distributed leaders actively before introducing new policies or technologies (Freeman et al, 2009) • Multi-institutional collaboration combined with planned follow-on actions supports sustainability (Sykes et al, 2010)

21. Active student engagement 3. The Social Constructivist Model Actively engaging students in formal processes to communicate tacit knowledge of standards • 4. The ‘Cultivated’ Community of Practice Model • Tacit standards communicated through participation in informal knowledge exchange networks ‘seeded’ by specific activities. The Future Formal activities and inputs Informal activities and inputs The Past 1. The Traditional Model Tacit standards absorbed over relatively longer times informally and serendipitously 2. The ‘Dominant Logic’ Explicit Model Standardsexplicitlyarticulated (with limitations) and passively presented to students Passive student engagement O’Donovan, Price & Rust (2008)

22. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

23. National HE change agent: supports collaboration of individuals, groups and institutions

24. National business deans council: Carpe Diem funded collaborations

25. National business deans council: Carpe Diem funded collaborations

26. Self & tutor assessment enabler - ReView

27. Evaluation - Thompson et al (2008) Students “Using the review system showed us which parts we were lacking in and [need to] develop” Survey % agreement  > 12 % across all 3 units of study “Feedback on assessment assisted my learning in this unit of study” Academic “Have spent the day marking essays which has been painless, efficient and productive. I have been adding marks and little comments in the comments box. It has been the most enjoyable marking experience I have had in 15 years of teaching. :-)”

28. Evaluation – Taylor et al (2009) Task 1: p<0.01 Self Tutor Task 2: p<0.01 Self Tutor

29. Evaluation – Soyref (2010) “As a student feedback was about comments from my marker to help understand my grade. As a peer-learning facilitator my role was to help newer students to interpret the expected standard and how to get there. As a tutor in ‘Business Context A’ now, my role is to help students to develop the capacity to work out the standard themselves and then strategise getting there. The Unit Coordinator’s assessment design is crucial, and ReView is the tool to make it happen efficiently.”

30. Example 2: Discipline support strategy Single focus eg. in-class team based learning expert Build on previous eg. 5 states hosting collaborations across Australia Meet with multiple levels of distributed leaders eg. a/deans, programme directors, teaching academics Promote proven learning activities eg. self & peer Promote proven enablers eg. self & peer

31. Peer quiz – 1 minute Which of the following is true about short in-class quizzes? • Teams underperform individuals on quizzes • Redoing an individual quiz as a team precludes peer learning • Fixed teams outperform changing groups • Immediate feedback after each quiz question is desirable but impractical

32. Self and peer assessment enabler Team completed quiz using IFAT

33. Evaluation – Dihoff et al (2003)

34. Self & peer assessment enabler - SPARK WA = Well Above team average AV = Average

35. HEI change Example 3: disciplines setting standards 2012 Peers audit Samples archived 2011 HEI prepare Next discipline Final submitted Accounting engaged Drafts developed Accounting engaged 2010 Disciplines chosen Disciplines engaged 2009 Policy announced

36. Evaluation • Supportive professional bodies • High attendance and engagement in briefings • Many applicants to draft Accounting threshold LO • Implementation concerns • teach to the test (22%) • negative impact on teacher attitudes (27%) • bureaucracy costs outweigh benefits (13%) • lower standards overall (12%).

37. Participant Learning Outcomes • Distinguish different roles of assessment • Identify drivers for assessment reform • Review relevant change literature • Describe collaborative Australian efforts • Evaluate potential assessment enablers

38. Conclusion • Carpe Diem: five years • Sustainable change is possible, although slower than desired • Change driven by a new assessment policy or enabling technology is less enduring without engagement of leadership and communities of practice • Further challenges to assessment standards, feedback and capacity remain • Eagerly await the outcomes of others’ research • BMAF Conference 2010 • BMAF journal

39. Thank you A/Professor Mark Freeman ALTC Discipline Scholar: Business, Management and Economics (+612) 90365030 mark.freeman@altc.edu.au Learning and Teaching Academic Standards Project