Technology-enabled learning-oriented assessment

1. Technology-enabled learning-oriented assessment Professor David Carless, @CarlessDavid University of Hong Kong, KU Leuven Keynote, May 29, 2019 The University of Hong Kong

2. Overview Learning-oriented assessment Designs for large classes Technology-enabled learning Implications & questions The University of Hong Kong

3. Competing assessment functions The University of Hong Kong

4. Learning-oriented assessment Summative vs formative Effective student learning as a key aim of all assessment The University of Hong Kong

5. The University of Hong Kong Productive assessment task design Development of student feedback literacy Development of student evaluative expertise Learning-oriented assessment framework

6. Assessment task design Iterative sequences of rich tasks Mirror real-life applications of disciplines Engage students productively & steadily The University of Hong Kong

7. Designing for feedback Assessment task 1 Interlinked task 2 Interlinked task 3 The University of Hong Kong

8. Student evaluative expertise Developing a nose for quality The University of Hong Kong

9. The University of Hong Kong Student feedback literacy Understandings, capacities & dispositions needed to use feedback for improvement (Carless & Boud, 2018).

10. EXAMPLE PSYCHOLOGY OF HEALTH BEHAVIOR – 1300 STUDENTS DEAKIN UNIVERSITY, AUSTRALIA The University of Hong Kong

11. The University of Hong Kong

12. Assessment design Online quizzes 10% Three linked journal entries 45% Final exam 45% (applying knowledge learnt in journals) The University of Hong Kong

13. Aligned assessment design Students’ use of feedback is built into the design The University of Hong Kong

14. Use of exemplars Video explanation of rubric (flipped element) + Online exemplar The University of Hong Kong

15. 3 forms of feedback Grade aligned with rubric 2. Written explanation linked to LOs 3. 5 minute audio file (25 sessional staff) The University of Hong Kong

16. Audio feedback Timely: 7 days before next task is due Enables rapport, nuance & care Explanation of how to fix a problem The University of Hong Kong

17. PEER LEARNING & TECHNOLOGY The University of Hong Kong

18. Audio peer feedback Activating student role Enabling comparison with work of peers (Filius et al., 2019) The University of Hong Kong

19. Peer assessment Peer reviewed assignments + detailed rubrics as key feature of MOOCs (Admiraal et al, 2015; Huisman et al., 2016) The University of Hong Kong

20. Reliability of PA Moderate reliability of peer assessment (Admiraal et al., 2015) Use of multiple peer assessments to mitigate variance in judgments (Hew, 2016) The University of Hong Kong

21. Peer & Self-evaluation Students did 5 peer reviews then self-evaluated own work (Hew, 2016) The University of Hong Kong

22. Composing peer feedback Providing feedback more cognitively engaging than receiving feedback (e.g. Nicol et al., 2014) The University of Hong Kong

23. TECHNOLOGY-ENABLED FEEDBACK The University of Hong Kong

24. The University of Hong Kong

25. Technology-enabled feedback Chapter 4 Literature synthesis Examples of practice Resources Questions for debate The University of Hong Kong

26. Selected strategies 1. Automated online quizzes 2. Audience response systems 3. Dashboards – learning analytics 4. Learning Management Systems (LMS) 5. Feedback storage & synthesis The University of Hong Kong

27. Online quizzes Quizzes with instant automated feedback Hints rather than just correct answers (Försteret al., 2018) The University of Hong Kong

28. Audience response systems Eliciting student participation in large classes (Ludvigsen et al., 2015) Clickers, Mentimeter, Poll Everywhere Hunsu et al. 2016 The University of Hong Kong

29. Learning analytics & feedback Using learning analytics to scale the provision of personalised feedback https://www.ontasklearning.org/ Pardo et al., (2017) The University of Hong Kong

30. LA for feedback at scale Automated feedback messages based on engagement & performance in LMS activities Sent one week before mid-term exam (Pardo et al., 2017) The University of Hong Kong

31. The University of Hong Kong

32. LA at Leuven Early feedback opportunities Self-directed learning Dashboards (Sedrakyan … Verbert et al., 2018) The University of Hong Kong

33. LMS Using or reconfiguring LMSs to enable access & retrieval for feedback purposes (Carless & Boud, 2018) The University of Hong Kong

34. FEATS portfolio https://www.youtube.com/channel/UCKOsEuHFPbWJ-Q6YUa062KQ The University of Hong Kong

35. FEATS functions Feedback review & synthesis tool Skill development resources Action planning & dialogue tool (Winstone, 2020, in press) The University of Hong Kong

36. Implications The University of Hong Kong

37. Sustainable feedback Enhancing student role to seek, generate & use feedback (Carless et al., 2011) The University of Hong Kong

38. Social constructivism Action on feedback is developed through learner agency, interaction & co-construction The University of Hong Kong

39. Key recommendations Pedagogy driving technology use Coherent assessment design Active student involvement Development of student feedback literacy The University of Hong Kong

40. Ed Tech follow-up issues Q1. How can technology be harnessed to promote worthwhile student learning? Q2. How does technology add further value? Q3. How does technology involve & empower students? The University of Hong Kong

41. Broader follow-up issues Q4. What is the vision for effective student learning at KU Leuven & how is it communicated to staff and students? Q5. What are the key institutional & disciplinary drivers and barriers for assessment change? The University of Hong Kong

42. QUESTIONS & COMMENTS The University of Hong Kong

43. References Admiraal, W., Huisman, B., & Pilli, O. (2015). Assessment in massive open online courses. The Electronic Journal of e-Learning, 13(4), 207-216. Broadbent, J., Panadero, E., & Boud, D. (2018). Implementing summative assessment with a formative flavour: A case study in a large class. Assessment & Evaluation in Higher Education, 43(2), 307-322. Carless, D. (2015). Exploring learning-oriented assessment processes. Higher Education, 69(6), 963-976. Carless, D. & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback.Assessment & Evaluation in Higher Education, https://doi.org/10.1080/02602938.2018.1463354. Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing sustainable feedback practices. Studies in Higher Education, 36(4), 395-407. Filius, R., R. de Kleijn, S. Uijl, F. Prins, H. van Rijen& D. Grobbee (2019). Audio peer feedback to promote deep learning in online education. Journal of Computer Assisted Learning. https://onlinelibrary.wiley.com/doi/full/10.1111/jcal.12363 Förster, M., Weiser, C., & Maur, A. (2018). How feedback provided by voluntary electronic quizzes affects learning outcomes of university students in large classes. Computers & Education, 121, 100-114. Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320-341. The University of Hong Kong

44. References Huisman, B., Admiraal, W., Pilli, L., van de Ven, M., & Saab, N. (2016). Peer assessment in MOOCs: The relationship between peer reviewers’ ability and authors’ essay performance. British Journal of Educational Technology, 49(1), 101-110. Hunsu, N., Adesope, O. & Bayly, D. (2016). A meta-analysis of the effects of audience response systems (clicker-based technologies) on cognition and affect. Computers and Education, 94, 102-119. Ludvigsen, K., Krumsvik, R., & Furnes, B. (2015). Creating formative feedback spaces in large lectures. Computers & Education, 88, 48-63. Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102-122. Pardo, A., Jovanovic, J., Dawson, S., Gasevic, D., & Mirriahi, N. (2017). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology. Sedrakyan, G., Malmberg, J., Verbert, K., Jarvela, S. & Kirschner, P. (2018). Linking learning behaviour analytics and learning science concepts: designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior. Winstone, N. (2020, in press). Facilitating students’ use of feedback: Capturing and tracking impact using digital tools. In M. Henderson, R. Ajjawi, D. Boud & E. Molloy (Eds), The impact of feedback in higher education. Basingstoke: Palgrave Macmillan. Winstone, N. & Carless, D. (2019, in press). Designing effective feedback processes in higher education: A learning-focused approach. London: Routledge. The University of Hong Kong

45. Resources 10 ideas for enhancing feedback with technology https://www.heacademy.ac.uk/system/files/resources/10_ideas_for_enhancing_feedback_with_technology.pdf Case of Clickers in Y1 Science http://y1feedback.ie/wp-content/uploads/2017/01/AIT-1-Anne-M-OBrien-5.pdf FEATS tutorial http://tinyurl.com/FEATSPortfolio The University of Hong Kong

46. Student feedback literacy The University of Hong Kong Appreciating Feedback Making Judgments Managing Affect Taking Action (Carless & Boud, 2018)

47. Information  action (Winstone & Carless, 2019) The University of Hong Kong

48. Two-stage exams Carl Wieman: Nobel prize-winning physicist 1. Individual exam (80-90%) 2. Small group (10%-20%) The University of Hong Kong

49. Two-stage exam processes Groups complete part of exam; negotiating answers; submitting 1 script per team Peer feedback Timely The University of Hong Kong

50. The University of Hong Kong