Evaluating and improving student achievement based on pre-set criteria

1. Evaluating and improving student achievement based on pre-set criteria Brian Whalley (b.whalley@sheffield.ac.uk) (For more information and resources see:http://brianbox.net/pedagogies/criteria/criteria.html) Various things in the notes for each slide plus a few references at the end

2. We’re going to look at • Assessment • Feedback • Supporting students and, by the way, Talk about a variety of aspects of education - whatever you want! (But we shall need to use a bit of ‘jargon’)

3. What is it like to be a student?(with apologies to Chiroptera and Thomas Nagel)

4. Please discuss with a neighbour What you mean by: 1. Feedback 2. Feedforward What do (you think) students mean (ie think wemean) by these terms? Outcomes (Constructive) alignment

5. Alignment thus … Learners construct meaning from what they do to learn. [This concept derives from cognitive psychology and constructivist theory, and recognizes] the importance of linking new material to concepts and experiences in the learner's memory, and extrapolation to possible future scenarios via the abstraction of basic principles through reflection. The teacher makes a deliberate alignment between the planned learning activities and the learning outcomes. This is a conscious effort to provide the learner with a clearly specified objective, a well designed learning activity or activities that are appropriate for the task, and well designed assessment criteria for giving feedback to the learner.

6. Feedback • Should be incorporated into the assessment • Students learn most by ‘feedback’ • Geoff Petty So we need to bring feedback into the alignment process

7. when to give feedback When can you? When shouldyou? When do you? Do others in your department agree on when you do? Do others in your department give ‘proper’ feedback anyway (module scheme)?

8. The Ten Principles of Assessment and Feedback • 1. Help clarify what good performance is • via aims, criteria, standards • 2. Encourage ‘time and effort’ on challenging learning tasks. • 3. Deliver high quality feedback information that helps learners self-correct. • 4. Encourage positive motivational beliefs and self-esteem. • 5. Encourage interaction and dialogue around learning (peer and teacher student) (Nicol 2007)

9. 10 principles continued • 6. Facilitate the development of self-assessment and reflection in learning. • 7. Give learners choice in assessment – content and processes. • 8. Involve students in decision-making about assessment policy and practice. • 9. Support the development of learning communities. • 10. Help teachers adapt teaching to student needs. How do we use exams (and coursework) to deliver these?

10. It’s all about assessment …’Students can escape bad teaching but they cannot escape bad assessment’’(David Boud, 1995) And thus about Feedback and a bit about Feedforward So we’ll need to clarify things a bit and explain something about jargon and so we need a plan ….

11. Hmmm….. I think we do this too often Constructive Alignment is the underpinning concept behind the requirements for programme specification, declarations of Intended Learning Outcomes (ILOs) and assessment criteria, and the use of criterion based assessment.

12. Towards a better learning system(‘Abolish feedback’) It depends what you mean by…… • Information as ‘thing’ or ‘concept’ • Feedback as ‘thing’ or ‘concept’ • We are not explicit in this, students do not often/always know what we mean? • And do we know….? Avoid: Positive feedback = good Negative feedback = bad Feedforward = information in advance

13. Feedbackin a mechanical control system The centrifugal governor Is used to keep shaft speeds constant - but, for students - we want to increase the ‘rate of learning/education’, call this ‘attainment’

14. Feedback and Feedforward in a control system sense Do ‘exams’ provide a reference input (or even output?) A MCQ might do this. Criterion Referencing might too.

15. Ee Marks, no remarks for Exam events Ee Delayed marks and remarks for Essay event Ee Ese Task set with Criterion Referenced assessment T attainment Educational ‘ramp up’ To raise the level of ‘attainment’ We need controlled input, ramping up. • Output, achievement, results

16. start end Ee ‘Attainment’ M MR MR Delayed marks and remarks for Exam event, Ee Ee Ee R T T A A Ee MR MR T A T A Can I be more constructive? Bringing feedback into modules How do we increase the gradient? Task Activity Marks Remarks Two Activity c/work Two Activity c/work + exam Gradient = ‘learning rate’

17. The ‘connoisseur approach’ to assessment They want ‘the highest standards’ - achievements How do competitors know they have achieved? feedback

18. What do you (think you) mean by: • Standards • Criteria • Rubric Remarks: ‘You did quite well …’ (to an American) = ‘You did very well’ ‘Marks’ and ‘Remarks’ (my usage) Moral: students and tutors need to make sure that the terms they use are understood by everyone concerned; this includes terms like ‘feedback’

19. ‘Marks’ and ‘Remarks’ • Marks - are just that, for a piece of work • (This begs the Q of what the work is….) • Remarks - the comments that go with it • The trick is to get appropriate remarks to go with the marks, as quickly as possible [PreflightsandAfter Action Reviews help • Increases the frequency of control ‘shoves’ • Helps avoid ‘sticking points’ (Troublesome knowledge etc) • Increases student-tutor contact]

20. A quick look at ‘Standards’ • UPPER SECOND • Students are able to demonstrate the following, with respect to the criteria relevant to their discipline: o a thorough understanding of key concepts and knowledge, and of a range of supporting evidence; o informed consideration of key issues and interpretation of evidence; • ability to integrate material from a variety of sources, and to deploy established techniques of analysis and enquiry, accurately and effectively; etc University of Sheffield, 29_Assessment.doc

21. Also ‘Grade Descriptors’ • These are from the tutor’s point of view. • How do students know how to achieve designated grades? • How does ‘feedback’ help with this process? • How are standards maintained? Eg by various markers? ‘I know a 2:1 when I see it; understanding criteria for degree classification in franchised university programmes’ Ecclestone, K. (2001) Journal of Further and Higher Education 25: 301-313

22. Assessment criteria are descriptions of what the learner has to do in order to demonstrate that the learning outcomes have been achieved. [threshold standard; minimum expectation to achieve credit] Dumbing down?

23. Actually, you are coaching(mentoring, providing scaffolding etc) • By providing exemplars of how to achieve the standards • By showing students their attainments (or not) • To show how students can better achieve and • Gain confidence in their work (and achievements)

24. ‘Everyone’ needs trainers Umm, sometimes it works …… Teamwork Feed forward It is not judgmental. The negative connotations of past failures are banished There is no such thing as failure just Feedforward. It is much easier to deliver. People are less defensive when discussing future performanceFeedforward is taken less personally, provokes less resistance. Marshall Goldsmith

25. Charting • DIDET project • Allison Littlejohn • Flexible learning Undergraduate-postgraduate- PhD and the wider community

26. charting Supporting employers and employment using another 3Cs Slides from: Littlejohn, JISC Presentation, 2008

27. Using criterion referenced assessment • Avoid the ‘connoisseur’ approach • Show students what is needed to accomplish tasks • Provide remarks with the marks • Marks and remarks go hand in hand • Give a sense of achievement • Thus, criteria need to be assignment specific and so • Link all this back to JITT (Just in Time Teaching/Preflights) and AAR (After Action Reviews) • Providing proper feedback and feedforward

28. Examples etc: In general: 1. Determine the Task and set the Activity 2. Set and refine your Objectives and Learning Outcomes 3. Set the Assessment Criteria (4 - 7 are usually enough) 4. Note potential problems, ‘troublesome knowledge’ etc and provide assistance with this (eg ‘preflights*’ before the main activity) 5. Students do the activity, write it up, paying attention to the criteria 6. You mark it according to the criteria (if you have helpers this makes it easier for them as the criteria are easily defined and adhered to) 7. Record marks for each criterion in a spreadsheet row (and add a few comments in a cell), use these to calculate the task’s mark profile (more feedback) 8. E-mail out the row of marks and remarks, separately, to each student, in one go using a nifty-tifty bit of php scripting§.

29. Learning outcomes At the end of the module students will be expected to be able to: ･Plan and structure a major piece of work ･Use referencing appropriate to a published paper Assessment Task • In week 9 you are required to submit a plan for the dissertation. The plan should include the following: ･a summary of the main argument of the dissertation (200 words); ･an indication of the structure, section headings and outline of the content of each section (500 words); ･indication of, and full references for, texts consulted and texts to be discussed in the dissertation.

30. Assessment Criteria Writing your criteria 1. Consider the learning outcomes being tested. 2. Consider the assignment you wish to set. 3. Brainstorm requirements for, or attributes of, successful performance in the assignment. 4. If necessary, specify the level of difficulty and the particular context for the assignment. 5. Focus on what is essential and categorise the requirements into clear criteria. 6. Check that criteria are assessable (valid and reliable) and that they are clearly worded and unambiguous (ask students to explain what they think you mean). (based on an example in Gosling and Moon, 2002)･

31. Marking and criteria(an example from the dissertation preparation case) Evidence of study/reading in preparation of the plan (20%) Articulation of a clear idea/theme to be discussed in the dissertation(30%) Design of a workable and defensible structure for the dissertation (30%) Identification of texts to which the dissertation will refer using the Harvard referencing system (20%) Although the criteria here are quite broad it at least shows how much effort should be expended on each section, e.g. don’t overload it with a literature review.

32. Practical (poster) assessment Take this poster; how might it be assessed? ‘hmm, a low 2.1, perhaps 61%?’ Not very a good assessment method is it? Malcolm Knowles was doing better in 1975 with a few criteria and a Likert scale for each. This ‘connoisseur approach’ is also used for: Essays; essay exams; dissertations; projects (The connoisseur approach does not do well in fulfilling the good practice of Gibbs, Nicol and McFarlane-Dick.)

35. Criteria in the mark scheme for Experiment 1 of the practical 1 from the previous slide The table below shows marks achieved for each of 10 criteria of an exercise. Each student gets an individual e-mail with the marks for each components and some remarks.

36. Setting out a learning design shows not only how the task should be set up but where criteria for assessment can be identified. Task Task Problem specified Task Manipulation of system Problem solution S Assessment T R R Rules System Examples Procedures Clues Guidance Help Provision of Feedback on solutions Support Support Resources T Tacit knowledge required S Sticking point(s) likely A temporal sequence describing a rule-based learning design, In Oliver et al. 2007. Describing ICT-based learning designs that promote quality learning outcomes

37. A final thought:adding the veg We have lots of ideas Technology is helping greatly but What we need to do is keep reviewing the teaching-learning process: ‘ …… the time, energy and resources students devote to activities designed to enhance learning at university.’ (Kerri-Lee Krause, 2005)

38. A few extra references to get the basic concepts • University of Plymouth Assessment handbook (www.plymouth.ac.uk/files/extranet/docs/TLD/Assessment guidelines March09.pdf) • University of Bath, ‘How to use assessment criteria’ (http://www.bath.ac.uk/learningandteaching/themes/assessment-criteria/) • Stevens, D. D., & Levi, A. J. (2005) Introduction to Rubrics, (Sterling, VA: Stylus).