Technology-enhanced assessment and transformational change

1. Technology-enhanced assessment and transformational change David Nicol, Project Director, REAP www.reap.ac.uk Deputy Director Centre for Academic Practice and Learning Enhancement (CAPLE) University of Strathclyde Nottingham Trent University, 3rd November 2006

2. The REAP Project • 3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University) • Focus is on large 1st year classes • Pedagogy: assessment for learner self-regulation • Range of technologies: online tests, simulations, discussion boards, e-voting, e-portfolios, peer/feedack software, admin systems, VLEs, offline-online • Goals: Learning quality and teaching efficiencies • Outputs: case studies of redesign: advice to support strategic change in institutions (transformation). • Dissemination across HE/FE sector

3. My Plan • Briefly discuss issues relating to first year experience and assessment • Review literature on formative assessment and 1st year experience (synthesis) • Provide case study of first year course redesign (drawn from REAP project) • Analysis of case study in relation to assessment principles from literature • Transformational and strategic change

4. First Year: The academic experience Why students leave academic programmes early: Flawed decision-making in initial choices Events that impact on students’ lives outside the classroom Students’ experiences of the programme and the institution Failure to cope with the academic demands of programmes Yorke and Logden (2004) Retention and student success in higher education.

5. First Year: The academic experience What is important? • Coping with transition • Understanding what is required • Engagement with academic programmes • Support and feedback • Experiences of success • Agents in own learning • Belief in self (ability) and motivation • Social dynamics of learning (belonging) Yorke and Tinto’s research

6. Gibbs and Simpson (2004) Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 3-31. See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl/

7. Gibbs and Simpson (2004) • Assessment tasks [Conditions 1-4] • Capture enough study time (in and out of class) • Are spread out evenly across timeline of study • Lead to productive activity (deep vs surface) • Communicate clear and high expectations

8. Gibbs and Simpson (2004) • Feedback [Conditions 5-11] • Is sufficient (in frequency; detail) • Is provided quickly enough to be useful • Focuses on learning rather than on marks • Is linked to assessment criteria/expected outcomes • Makes sense to students • Is received by students and attended to • Is acted upon, to improve work and/or learning

9. Nicol and Macfarlane-Dick (2006) • Literature review: Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218 See: • Student Enhanced Learning through Effective Feedback [SENLEF] project funded by HE Academy • REAP project: www.reap.ac.uk

10. Research on Assessment in HE Teaching/learning paradigm Transmission Constructivist [student-centred] Assessment paradigm Transmission [teacher-centred]

11. Some key research findings • Students are always engaged in self-assessment/self-regulation of their own learning (Winne, 2005; Black & Wiliam, 2005). Logically entailed by constructivist thinking. • The act of using teacher feedback implies that self-assessment must be present (Sadler, 1983, Nicol and Macfarlane-Dick, 2006) • Feedback in HE is being significantly reduced so how are students still learning? • The question is: how can we scaffold students learning so they become better at self-regulation of learning? (Lajoie, 2005)

12. Scaffolding self regulation: 7 principles of good feedback • Clarify what good performance is (goals, criteria, standards). • Facilitate reflection and self-assessment in learning • Deliver high quality feedback to students: feedback that enables students to monitor and self-correct • Encourage peer and tutor dialogue around learning • Encourage positive motivational beliefs & self esteem through assessment • Provide opportunities to close the feedback loop • Use feedback information to shape teaching Source: Nicol and Macfarlane-Dick (2006) Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education

13. Re-engineering Assessment Practices [REAP] Psychology Case Study Context: • 560 first year students • Mixture of psychology majors (130) and those taking psychology only for one year (430) • 6 topic areas, 48 lectures, 4 tutorials, 12 practicals • Assessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

14. Psychology Case Study Problems identified with the course: • Students got no practice in writing skills but required in the exam • No feedback except on MCQs (percent correct) • Didn’t want to increase staff workload • Wanted to improve overall exam marks • And standard of entrant to second year

15. Psychology re-design Stage 1: Question 1: moderate difficulty (50 words) Individual response – post it –discuss answer in groups (of 6) Timed release: model answer to self-evaluate their response Stage 2: Question 2: difficult (100 words) Group response – discuss (online) – agree – post response Model answer released for stage 2 Stage 3: Question 3: complex (300 word essay) Group response – discuss (online) – agree – post Model answer released for stage 3

16. Psychology • Task 1: Define and describe structural encoding, phonological encoding and semantic encoding. Provide an example of each construct. (50 words, individual) • Task 2: Describe the serial position effect and its two separate components. Discuss the specific structural components of memory that are responsible. (100 word, group response) • Task 3: Summarise the ‘stage theory’ of memory. To what extent does it provide an adequate theory of memory? (300 word, group response)

17. Psychology re-design: key features • Writing tasks related to current lecture delivery (on human memory). • 78 volunteers –induction task • Question of increasing difficulty and length of answer (50 words, 100 words and 300 words) • Moving from individual to group responses. • ‘Scaffolding’ of learning through peer discussion (and cognitive growth thro conflict resolution) • Comparing work against model answers • Use of assignment/ discussion tools in (WebCT)

18. Planned developments • Replace half the lectures with discussion tasks • Tasks will become progressively more demanding within and across topics as the year progresses. • Participation will be compulsory and monitored • Individual responses will precede online discussion and response for all tasks • Once a final group response has been agreed each student will submit a copy to VLE (WebCT) • Purpose is to deal with free rider-effects Source: Jim Baxter, Psychology, Strathclyde University

19. The seven feedback principles • Standard format and model answers provide progressive clarification of expectations (clear goals, principle 1) • Students encouraged to self-assess against model answer (self-assessment, principle 2) • Online peer discussion aimed at reaching consensus about response (dialogue, principle 4) • Staged complexity and focus on learning rather marks (motivation, principle 5) • Repeated cycle of topics and tasks (closing gap, principle 6) • Tutors can monitor progress and adapt (shaping teaching, principle 7)

20. Gibbs & Simpson’s four assessment conditions • Assessment tasks require significant study out of class (capture sufficient study time, condition 1) • They are distributed across topics and weeks (are spread out evenly, condition 2) • They move students progressively to deeper levels of understanding (productive/ deep learning, condition 3) • There are explicit goals and progressive increase in challenge (communicates clear and high expectations, condition 4)

21. What can we learn from this case study? • Use of a simple technology (discussion board) • Considerable thought gone into the pedagogical design [which is transferable]. • Supported by robust assessment/learning principles • Coordinated approach within the department • Evaluation showed a lot of supportive socialisation during induction task • Intention to compare year on year examination performance.

22. Assessment and the the first year Two super principles that need to be balanced within the first year and across years • Engagement: Organised and structured academic activities (so students know they are making progress). • Empowerment: Opportunities for students to internalise academic standards and supervise themselves as they study in relation to these standards (self and peer assessment processes)

23. First Year: The academic experience What is important? • Coping with transition • Understanding what is required • Engagement with academic programmes • Support and feedback • Experiences of success • Sense of agency in own learning • Belief in self (ability) and motivation • Social dynamics of learning (belonging) Yorke and Tinto’s research

24. Questions and discussion

25. Transformational change • SFC definitions (pace/scale, cost-quality, levels) Enablers • Motivation • Powerful pedagogic designs • Readiness factors (benchmarking) • Links to other strategy developments and external drivers • Scenario planning • Dissemination

26. Case Study 2 • Large class taught through lectures, tutorials • Exam – series of multiple choice questions to test breadth of understanding • Sometimes essay or short answer questions in exam.

27. Redesign • Students introduced to MCQs – how they work • After lectures but before tutorial/problem solving class students in pairs prepare MCQ including feedback for right and wrong answers. • In tutorial pairs swap MCQs and get feedback – revise - post in VLE. • Refinement: students identify which level of Bloom’s taxonomy they questions are testing • Final exam teacher selects some students’ questions but has them provide reasons for answers. • Producing questions compulsory

28. Benefits of Redesign • Students develop questioning skills • Creating feedback develops writing skills and critical thinking – giving reasons for correct and wrong answers - deep learning • Sharing in class encourages peer feedback • Identifying Blooms level leads to further reflection • Use in final exam encourages class to share work

29. Relation to feedback principles • Students create MCQs (actively create criteria, principle 1) • Evaluating content against criteria & Bloom categories (self-assessment/reflection, principle 2) • Tutor monitoring and general feedback (principle 3) • Peer feedback during creation and in tutorials (dialogue, principle 4) • MCQs used in exam and the creation process is engaging (motivation, principle 5) • Cyclical development of MCQs (closing gap, principle 6) • Teaching could be shaped by results (principle 7)

30. Gibbs & Simpson’s four assessment conditions • Writing MCQs as preparation for tutorials (capture sufficient study time, condition 1) • Task could be a regular requirement and built into tutorials (are spread out evenly, condition 2) • MCQs could move to deeper levels matching Blooms taxonomy (productive/ deep learning, condition 3) • The goals are clear and there is progressive increase in challenge (communicates clear and high expectations, condition 4)

31. Task • Do you have a learning design that you could share with the group or a specific method? • Using the seven principles/questions and/or four conditions can you identify ways of improving some aspect of your own class teaching?

32. Other relevant papers Nicol, D (1996), Increasing success in first year courses: assessment redesign, self-regulation and learning technologies, Paper being presented at ASCILITE Conference, Sydney, Dec 3-6, 2006. Nicol, D (in press), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology (to be published 2007). Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.