Setting and Marking Long Answer Qs (Essays)

1. Setting and Marking Long Answer Qs (Essays) Rachel Davis, Vicki Dale, Mandy Nevel Examiner training day, 6th January 2011

2. Overview • Why assess? • Essay/Long answer questions • Definition, examples, excercise • Advantages, limitations and misconceptions • Constructing your own questions • Construct, evaluate and improve • Model Answers • Giving feedback • appropriate, effective, efficient • Supporting students

3. Why assess? Discuss ...

4. Assessment ... • OF learning • To classify or grade students • To determine fitness for entry to programme/practice • To set/measure (course) standards • To get feedback on our teaching quality • FOR learning • To motivate students to learn • To help students choose modules/electives • To guide students’ improvement • AS learning • To allow students to monitor their progress (in relation to course standards & other students) (Bloxham and Boyd 2007, Race 2007)

5. What is the purpose of an Essay/LAQ? • Discuss…

6. Purpose of Essay/LAQ • Allow for individuality and expression • Can reflect depth of student understanding • A measure of a student’s written style (Race 2007)

7. A Useful Definition…? A test item which requires a response composed by the examinee, usually in the form of one or more sentences, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject. Stalnaker (1951)

8. Our Focus Today… • A question where the student has to • create an answer rather than select an answer • construct a response that requires more than a sentence • Allows for different or original answers • Requires subjectivity in marking • Requires more systematic and in-depth thinking than other constructed response items (SAQ) • N.B. We will not focus on writing skills here

9. Example A • List the 7-step path to making “ethical decisions”. List them in their correct progressive order. • Is this an effective essay/LAQ ?

10. Example A: Not Effective • Doesn’t require complete sentences or more than one sentence – can be a list • No room for originality of response • Requires a list of steps rather than systematic and complex thinking about ideas • Does not require expertise in marking (because does not require depth and original thinking from student)

11. Example B • Explain in what ways a person’s failure to apply step 5 of the 7-step path for making ethical decisions will impact on his or her ability to make ethical decisions. Provide an example that illustrates this impact.

12. Example B: Effective • Requires a constructed response of several sentences in order to sufficiently meet demands of task • Allows for different responses or patterns of responses (examples used, ordering of response items) • Due to above – accuracy, quality and relevance of responses needs to be judged by an expert

13. Exercise 1 What is an Essay/LAQ?

14. Appropriate Use of Essay/LAQ • Advantages • Assess higher-order critical thinking skills • Evaluate thinking and reasoning • Provide authentic experience (closer to real-life decision making) • Limitations • Assess a limited sample of the range of content • Are difficult and time consuming to grade • Provide practice in poor or unpolished writing

15. Common Misconceptions • Essay/LAQs: • Assess high-order critical thinking skills regardless of how responses are written • Are easy to construct • Eliminates the problem of guessing (bluffing!) • Benefit all students by placing emphasis on written communication (e.g. ESL/SPL/WP) • Encourage more thorough preparation (is often related to style of teaching than style of exams)

16. Review: What is an Essay/LAQ? • The student has to • create an answer rather than select an answer • construct a response that requires more than a sentence • Allows for different or original answers • Requires subjectivity in judging accuracy and quality of responses (expert) • Requires more systematic and in-depth thinking than other constructed response items (SAQ)

17. Essay/LAQ or MCQ/SAQ? • Student ranking same for both so your choice • Learning outcomes/objectives often contain verbs alluding to the ability/abilities you intend to assess • These are sometimes called directive verbs and can help you choose your method • For Example: • “Explain” requires construction of an answer • “Identify” requires selection • “Interpret” could be construction or selection Chase & Jacobs (1992)

18. Choosing Your Method • Can students demonstrate the action and thinking processes indicated by the directive verb by selecting and/or by constructing answers? • If both possible then consider your personal skill set and time available for setting/marking • How critical are original and authentic responses? Is it important to make students’ thinking explicit?

19. Exercise 2 Directive verbs

20. Exercise 3 Construct your own Essay/LAQ

21. Exercise 3: Constructing Qs In pairs, write a long answer question relevant to your area of teaching that would simply require a student to reiterate material covered on the course to satisfy the mark descriptor ‘Sound answer’. Sound answer (55) Basic coverage of main aspects of topic. Statements supported by facts but limited evidence of critical ability.

22. Exercise 3: Modifying Qs • In pairs, write a long answer question relevant to your area of teaching that would simply require a student to reiterate material covered on the course to satisfy the mark descriptor ‘Sound answer’. • Modify the question so that the “sound answer” answer would require more than just reiteration of the material. Sound answer (55) Basic coverage of main aspects of topic. Statements supported by facts but limited evidence of critical ability.

23. Evaluating Essay/LAQs • How do you know if your question is: • Valid - measures what it claims to • Reliable (Inter-rater, Intra-rater) • Transparent (criteria are clear to students) • Feasible • Aligned with learning outcomes? • Where might you look for further information/advice/support?

24. Sources of IAG • Internal Resources • 17-point marking scheme! • Course learning outcomes/objectives • Past papers • Colleagues • Students (reflect on your questions post-exam) • Our checklist! • Workbook(s) • will be posted on examiners website • Learning Theory (e.g. Bloom, Race) • Online resources

25. The RVC 17-point marking scheme

26. Bloom’s (1956) taxonomy

27. Tips on Setting Effective LAQs • Keep useful documents in sight • intended learning outcomes • 17-pont marking scheme • Definition of effective essay/LAQ • Include data/info to reduce reliance on memory • Make question layout clear and easy to follow • Keep sentences short, bullet points or separate sections • Assign marks (or suggested time) in brackets • Proofread Qs carefully • Write out the answer to your own question • Get feedback from colleagues and students! • Do they understand it the same way you do? • Ask them to provide sketch answers

28. Model Answers

29. Discussion Point: Model Answers • Many of you will have experience of writing/using model answers • Discuss the SWOT of preparing and providing model answers with a partner.

30. Exercise 4 Preparing Model Answers

31. Exercise 4: Model Answers • In pairs, write a model answer for each of the essay questions you wrote for the previous exercise.

32. Exercise 4: Model Answers • In pairs, write a model answer for each of the essay questions you wrote for the previous exercise. • How do the answers differ?

33. Exercise 4: Model Answers • In pairs, write a model answer for each of the essay questions you wrote for the previous exercise. How do the answers differ? • How do the answers differ? • In your groups, discuss the implications of question and answer setting for other people marking your assignment. (You may wish to share questions and model answers with others in your group.)

34. Feedback

35. Discussion Point: Feedback • Many of you have experience of marking LAQs/essays/coursework • How do you go about it? • What difficulties have you had? • How do you ensure consistency? • Discuss your experiences with your group and report back

36. Feedback should be: • Timely, relevant and meaningful • Formative vs. summative • Comments vs. grades • Individual vs. generic • Feed-forward vs. feedback • Fair for all students • Use a variety of assessment methods • Allow students to practice these • Make assessment criteria explicit

37. Marking fairly - During • Be familiar with the assessment criteria (‘knows’ versus ‘knows how’) • Read through the first few answers without assigning a mark • Have a copy of the marking scheme close to hand • When you have an idea of the standard, put in a provisional mark with brief notes on why • Leave any ‘difficult’ answers to the end

38. Marking Fairly - After • Plot your marks – where is the mean? What is the range of marks and are they spread evenly • Take into consideration the number of answers – if less than 12, this is less reliable. • Put answers in groups according to mark and check they are of a similar standard • Check failing answers deserve to fail • Compare with other assessors

39. Coping with ‘Difficult’ Answers • In your experience, what makes an answer particularly difficult to mark?

40. Giving Good Feedback • Set aside plenty of time for this part of the process • Use marking scheme to signpost opportunities for improvement • Phrase comments as questions – less critical • Balance ‘what was good’ with ‘what could be improved’ • Illustrate general comments with specific examples – e.g. if you comment on the structure of the answer, give specific areas that could be improved • Where there are particular weaknesses, try to point the student towards additional material

41. Supporting Students • Make feedback opportunities explicitly clear to them (previous, current, future) • Think about providing an (pre- and post) exam session that covers both the HOW and WHAT of taking exams (with help?) • Suggestions • Undertake past papers and mark each others work • Devise own essay questions and share with friends • Book 1-2-1 session with learning support • Use library books & online materials • Go to Learning Development web pages (to be relaunched)

42. References Bloxham, S. and P. Boyd (2007). Developing Effective Assessment in Higher Education: a practical guide. Maidenhead, Open University Press. Brown, G. (2001). "Assessment: a guide for lecturers." Retrieved 30 December, 2010, from http://www.palatine.ac.uk/files/980.pdf. Jolly, B. (2010). Written examinations. Understanding Medical Education: Evidence, Theory and Practice. T. Swanwick. Chichester, Wiley-Blackwell: 208-231. Race, P. (2007). Designing assessment and feedback to enhance learning. The Lecturer's Toolkit. London & New York, Routledge: 27-94.

43. References • Reiner, Bothell, Sudweeks & Wood (2002) Preparing effective essay question.New Forums Press. • Jacobs LC, Chase CI (1992) Developing and using tests effectively: A guide for faculty. San Francisco: Jossey- Bass.