Standards, criteria and assessment of the Geography Dissertation in the UK

1. Assessment for Learning Conference, June 22nd 2009 Standards, criteria and assessment of the Geography Dissertation in the UK Dr Margaret E. Harrison University of Gloucestershire Dr Dawn T. Nicholson Manchester Metropolitan University Professor W. Brian Whalley Queen’s University of Belfast

2. Geography dissertations • 'Dissertations have had a long history in geographical higher education, being widely regarded as the pinnacle of an individual's undergraduate studies and the prime source of autonomous learning' (Gold et al., 1991) • Often comprise a large proportion of the overall weighting for undergraduate degrees • Performance may be used to adjudicate degree award in borderline cases • Demonstration of independent working therefore grade awarded of interest to employers

3. Wider background • Marking reliability, marking to the full range • Maintenance of standards in HE (grade inflation) • Accountability (litigation) • Increasing evidence of new assessment culture emerging (Rust 2007) • Assessment criteria (e.g. Harrison and Whalley 2008) • Grade descriptors • Formative assessment • Feedback • But still poor practice that go unchallenged (Rust 2007)

4. Inconsistency, ambiguity etc • Inconsistent approaches to use of criteria (including weightings and standards) for dissertation marking (Hand and Clewes 2000) • Inconsistent use of terms in feedback (Pepper et al 2001) • Confusion among teachers over concepts of ‘criteria’ and ‘standards’ (Sadler 2005) • Considerable ambiguity concerning the use, meaning and application of assessment criteria (Webster et al 2000)

5. Students and criteria • Students don’t automatically understand the meaning of criteria (Penny and Grover 1996, Rust et al 2003, Woolf 2004) • Criteria need to be explicit and understood by students to effectively support learning (Gibbs and Simpson 2005) • An important principle is that students and markers are aware of and understand the assessment criteria and/or schemes that will be used to mark each assessment task. (QAA 2006, p16-17).

6. Some key research questions • What assessment criteria are used, how are they established and approved? • How are students assisted to interpret assessment criteria? What is the role of supervision in this? • What grade descriptors and marking schemes are used? • What are the procedures for double marking, anonymous marking, blind marking? • What happens in the event of a disagreement between first and second markers? • Identify ‘good’ practice and showcasing of innovative practice

7. Methods • Questionnaire survey sent to all GEES Departments – a mixture of closed and open questions • Follow-up telephone interviews with selected respondents • Number of responses: 22 • This represents 20 HEIs - 2 separate responses from 2 HEIs. • Responses include institutions throughout the UK with an even split in terms of pre-1992 and post-1992 HEIs.

8. Credit-rating and word length

9. Study period and product format Most dissertations are submitted as printed documents, but the handbook provides students with ‘the freedom to be creative about the form of the exercise.... the student can undertake a practical.... or a performance project.’

10. Elements that are assessed * PowerPoint poster on progress and literature review

11. Assessment criteria Reflective ability Use of English language Originality and perceptiveness Use of ICT applications Personal initiative and responsibility Use of referencing conventions Clarity of aims, topic identification Critical ability Awareness of literature Standard of presentation Integration of illustrative materials Research design and methodology Risks and ethical considerations Quality of database Cognitive, practical and personal skills Data presentation, analysis and interpretation Compliance with requirements Ability to work independently Theoretical understanding Conduct and competence during practical work Conclusions justified Flair, innovation Think creatively Engagement with administrative processes Use of complex academic terminology Sustained argument

12. Assessment criteria • ‘We used to have weightings for each bit but we now use a more general mark sheet. Every time we get a new external examiner they have different ideas and we tend to change our marking strategy fairly regularly’. (post 1992 HEI) • ‘Established by tradition … some of our descriptors are rather woolly. … The bottom line criteria for our dissertations is that it must be original’. (pre 1992 HEI

13. Assessment criteria • ‘Assessment scheme developed by a departmental working party … and then approved by a team meeting’ (equal weighting of 6 aspects including research skills, analysis/discussion, and presentation) (post 1992 HEI) • ‘I’m not sure what you’re asking here – do you want to know the university’s quality assurance system?’(pre 1992 HEI) • ‘It was laid down at the start of the Modular Course in 1974; I’m not aware that it has ever changed. Indeed it is the only feature of the course that has never changed’. (post 1992 HEI)

14. Grade descriptors and marking schemes • “See attached. These are departmental descriptors” (pre-1992 HEI). • “Attached – this is institutional” (post-1992 HEI). • “The grade structure and broad descriptors are university wide; the detailed interpretations were drawn up by the department” (pre-1992 HEI). • “We don’t have this by level. The University keeps promising it!” (post-1992 HEI).

15. “All departmental marking is anonymous, but clearly supervisors are likely to recognise the dissertations of ‘their’ supervisees” (post-1992 HEI) 95% marked by the supervisor ALL are double marked (64% blind and 36% ‘seen’) Six are marked anonymously…….?! One uses viva (supervisor and second marker) to inform mark Qualifications of the second marker – an issue? Marking

16. Disagreements about marks • Disagreement generally >5% or different class • Markers encouraged to discuss in first instance “Two markers would meet and discuss the reasons for their grading, which may reveal something not noticed or unknown to one of them…..” • If no resolution, introduce third marker • Use median mark from three • Discuss and agree • Discuss at internal exam board • If no resolution (still) • Module leader decides or external examiner adjudicates

17. Supervision Examples of other activities: peer led work, presentations, field and laboratory training

18. Feedback • Only three (14%) supervisors discuss content but don’t actually read, edit or comment on specific written material. • Some specify no. chapters or pages, % of work, specific chapters (eg lit. review) or specific date. • Feedback on complete draft rare - as work not available! “[Feedback] comments ‘should not compromise the independence of the project by modifying its principal organising ideas or otherwise adjusting its conceptual structure’.”

19. Supporting students • ‘There are [also] group sessions…… and formal supervisory meetings where it is discussed’ (post-1992 HEI) • ‘We top and tail the research design course with sessions on ‘What is research’ and ‘Writing up’, both of which include a discussion of expectations / features of a good dissertation / assessment criteria / grade descriptors’(pre-1992 HEI) • ‘Done as part of the supervision process’ (post-1992 HEI) • Scheme leader and tutors brief students on what is expected’ (pre-1992 HEI).

20. ‘There is a lecture at the start of the final year, but this mainly comes through tutors.’ (pre-1992 HEI) ‘It is part of the tutorial support’ (post-1992 HEI) Dissertation Guide – explanation of criteria Students peer-mark past dissertations using criteria Criteria discussed during Research design module Whole cohort lecture given to explain criteria Supporting students Examples of good practice

21. More good practice • “...introduction of the preliminary literature review as part of the Research Design [level 5] course”. • Students.... deliver oral presentations to a panel of supervisors and their peers”. • “Dissertation topics are not prescribed”. • “After a period in which dissertations were downgraded to an option, I think there is every chance that they might.... return to being compulsory”.

22. Conclusions • Requirements for dissertations vary but there is broad consensus across the sector • Consistent range of themes embedded in criteria, eg: • Employability: Skills, conduct, reflection • Research: Design, methods, data analysis • Higher skills: Criticality, flair, innovation, originality • Lot of variation in the way that assessment criteria are established: • Passive: Tradition, evolution, shared experience • Reactive: Compliance with University generic criteria • Proactive: Designed collaboratively and reviewed

23. Conclusions • There is some confusion in use of the terminology • Evidence of rigor and transparency in marking • Disagreements are dealt with in a variety of ways and the role of the external examiner varies considerably • Supervision time is consistent but effective learning can be achieved through other means • A range of measures are in place to support students interpret assessment criteria • How do we know learning achieved? • Feedback on drafts is variable – implications for maintenance of standards and student independence

24. Implications • Care needed in the formulation of criteria that are explicit • HEI staff need to adhere closely to criteria to achieve equivalence and consistency • Given the confusion over terms and the widely differing practices, this has implications for staff development needs • All of this is irrelevant if students are not party to understanding criteria and expectations

25. Bettany-Saltikov, J., Kilinc, S. and Stow, K.(2008). Bones, boys, bombs and booze: an exploratory study of the reliability of marking dissertations across disciplines. Assessment and Evaluation in Higher Education 1, 1-19. Gibbs, G. and Simpson, C.(2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education 1, 3–31. Gold, J. R., Jenkins, A., Lee, R., Monk, J., Riley, J., Shepard, I., and Unwin, D.(1991).Teaching Geography in Higher Education. Basil Blackwell, Oxford. Hand, L. and Clewes, D.(2000). Marking the Difference: an investigation of the criteria used for assessing undergraduate dissertations in a business school. Assessment and Evaluation in Higher Education25(1), 5-21. Harrison, M. E. and Whalley, W. B.(2008). Undertaking a dissertation from start to finish: the process and product. Journal of Geography in Higher Education 32(3), 401-418. Partington, J.(1994). Double-marking students’ work. Assessment and Evaluation in Higher Education 19(1), 57–60. Penny. A. J. and Grover, C.(1996). An analysis of student grade expectations and marker consistency. Assessment and Evaluation in Higher Education21(2), 173-184. Pepper, P., Webster, F. and Jenkins, A. (2001). Benchmarking in Geography: some implications for assessing dissertations in the undergraduate curriculum. Journal of Geography in Higher Education25(1), 23-35. Quality Assurance Agency(2006).Code of Practice for the Assurance of Academic Quality and Standards in HE: Section 6 - Assessment of Students. Available online at: http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/section6/COP_AOS.pdf. [Accessed 4th March 2009]. Rust, C. (2007). Towards a scholarship of assessment. Assessment and Evaluation in Higher Education32, 229-237. Rust, C., Price, M., and O'Donovan, B.(2003). Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education28, 146-164. Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education30, 175-194. Webster, F., Pepper, D. and Jenkins, A. (2000). Assessing the undergraduate dissertation. Assessment and Evaluation in Higher Education25(1), 71-80. Woolf, H. (2004). Assessment criteria: reflections on current practices. Assessment and Evaluation in Higher Education29(4), 479-493. References

26. Thank for your attentionAny questions? This project is being supported through the GEES Subject Centre Small-Scale Project Funding scheme We would like to acknowledge Karen Logan, our Research Assistant from Queen’s University Belfast, for her assistance with this project.

27. Survey Questions 1 • Do all your students undertake a Dissertation as part of their Geography Honours degree programme? • How many calendar months do your students have to work on their Dissertation? • What is the credit rating (CATS or ETCS credits) for a Dissertation in your Department? • Does any element of a Dissertation contribute to credit at less than final year?

28. Survey Questions 2 • Does your Department specify the number of hours of supervisory contact permitted per student? • What form does the supervisory contact take and who initiates supervisory sessions? • What is your Department position concerning feedback on drafts? • What form may a Dissertation take? • Please indicate sub-elements of the Dissertation and their relative weightings.

29. Survey Questions 3 • Please provide a copy of your Department marking schemes and/or assessment criteria. • Please indicate how this scheme / criteria is (a) established and (b) approved. • Please comment on any assistance given to students in interpreting the assessment criteria provided. • Please provide a copy of the grade descriptors or 'standards' scheme for the Dissertation.

30. Survey Questions 4 • Which marking procedures are adopted for Dissertations (e.g. double marking; blind marking; anonymous marking). • What procedures are followed in the event of a disagreement between markers. • Please comment on whether external examiners are asked to adjudicate. • Please provide an example of good practice in Dissertation assessment.