Tutors: Who Needs them?

1. Tutors: Who Needs them? Self assessment using Grade Related Criteria http://www.rgu.ac.uk/celt/learning/page.cfm?pge=7347#Tutor

2. Aims of this Session • To Review the literature on Self assessment • To outline the research design of an “experiment” with final honours year students • To review the results • To discuss the implications of the research for assessment

3. Aims of Assessment • Assessment has traditionally four main roles: • Formative, to provide support for future learning; • Summative, to provide information about performance at the end of a course; • Certification, selecting by means of qualification and • Evaluative, a means by which stakeholders can judge the effectiveness of the system as a whole.

4. Role of Assessment • Assessment as an aid to learning? • Assessment as a means of identifying ways of improving? • Assessment as a skill to be acquired by students?

5. Aims of Self assessment • An aid to student learning • To help students understand more clearly the basis on which they are assessed • To develop self assessment skills • To encourage students to be more self critical about their work • To give students more effective feedback • To improve the efficiency of the assessment process

6. The Context • Module Economics of Taxation and Corporate Taxes (BS4214) • Part of Accounting Module on taxation • Accounting students used to assessments which involve a known or correct answer • How do you assess an economics answer? • Session 2002/3 • 48 final year accounting honours students • Using the Common Grading scheme with grade related criteria • If GRC Scheme is as transparent as it claims then students should be able to use it to assess themselves accurately

7. Literature Review • How Reliable is Self Assessment? • What factors influence Self Assessment?

8. Literature Review • Mabe and West (1982) • Review of 55 studies from 1942-1977 involving 267 correlations • Findings: • Poor relationship between students and tutor ratings • R= 0.29 SD 0.25 (high degree of variability)

9. Literature Review • Boud and Falchikov (1995) • Total of 68 studies from 1932 -1994 • Do students over-rate themselves? ( 17 studies) • Good students better at rating themselves than bad students? (11 studies) • Advanced PG students better than “freshmen” (7 studies found PG better at self assessment) • More practice made self assessment better? (7 studies found that it did not!) • Gender differences? (6 studies – 3 showed women more accurate than men; the rest no evidence)

10. Literature Review • Larres, Ballantine and Whittington (2003) Self Assessment with Accounting Students • Computer Literacy in two UK universities with sample of accounting students • “Vast majority” over-estimated their computer knowledge • Conclusion: • “Self Assessment is not an appropriate means of determining computer literacy” • But it did provide: “a useful adjunct into students’ attitudes to computing and stimulated reflection on their abilities.”

11. Literature Review • Self Assessment as an aid to learning • Fitzgerald (1997) Found significant improvements in learning amongst medical students when self assessment introduced

12. Literature Review • Self Assessment as an aid to learning • Rust (2003) • Developed the use of grade related criteria with a group of 290 second year undergraduate students • 140 attended a workshop in which they used the criteria to assess work by students from previous years; 150 did not attend: Experimental v control; • Performance monitored and samples controlled for ability • Results: significant gains both short run and long-run

13. Methodology • Set three topics for students to choose • Set up iNET discussion forum using Salmons Five stage model Salmon (2003) • Gave out the criteria and ran a one hour workshop on what was meant be each of the criteria • Conducted an online Q & A session • Students handed in coursework and completed a Self assessment proforma using the same criteria as the tutor. • Students completed an evaluation of the work. Ways in which it could be improved. Analysis of self evaluation feedback + comments on iNET(the qualitative data) • Proformas submitted but not read by tutor-assessor • Coursework assessed internally, double marked • Comparisons made student v tutors assessment

14. Criteria • Presentation 10% • Research 10% • Knowledge and Understanding 20% • Analysis 30% • Evaluation 30% • Overall Grade obtained by “averaging” + profile of grade

15. Accuracy of Self Assessment • Actual Grades • Degree of Match • Combinations of both • Tests used • Correlations • Kruskall Wallis, Mann Whitney • Pearson Chi Square

16. Results

17. Results

18. Results

19. Hypotheses • H1: There will be no statistically significant differences in the degree of match in grade tutor v student • H2: There will be no statistically significant gender differences in the degree of match between tutor and students assessments. • H3 There will be no statistically significant differences in the degree of match made by good and poor students and the tutor • Three definitions of a good student • 1. Grade 5 and above (Broad) • 2. Grade 6 only (Narrow) • 3. First (Honours Narrow) • 4. First + 2:1 (Honours Broad)

20. Hypothesis 1

21. Hypothesis 2

22. Hypothesis 3

23. Conclusions on Quantitative Data • There were statistically significant differences between the grades by the tutor and grades by students. Students rated themselves significantly below the tutor on all dimensions except “Evaluation” • There were no significant gender differences on any of the dimensions with the exception of “Research” where female students underscored themselves on this in comparison to male students • No strong evidence that the “best” students rated themselves better than weaker students

24. Qualitative Evidence • Used quotes from the self evaluation form + iNET discussion forums to ascertain whether students were more aware of the criteria against which they were assessed • Did a content analysis of responses

25. Qualitative Evidence • “ I feel I have enhanced my ability to perform research and critical analysis through this assignment”Female student Grade 4 • “ I feel the strengths of this report was (sic) the research conducted as well as the knowledge and understanding I gained from this” Female student overall grade 5 • “As a result of this work I have learned that tax can be interesting!. The strengths of this work is (sic) in its presentation, application of knowledge and analysis of the issues identified” Male student overall grade 4

26. Feedback on the exercise • “ Thank you! I cant believe I got that mark ( I’m still shaking!) I honestly did think this was one of my poorer pieces of coursework, but I’m very glad you didn’t agree! Thanks also for such a detailed feedback, it’s not often we get this and I found it very useful” Female student Grade 6 • Quote from Paper • Biggest mismatches on Research and Evaluation

27. Conclusions • Using a university–wide grade related criteria scheme improves the accuracy of self assessment • Self Assessment: • helps students “unpack” the criteria by which they are assessed • improves feedback that tutors can give to students • identifies criteria that need to be made clearer in the future with more detailed briefings (or giving students actual coursework from previous years to practice assessing) • has the potential to change the role of tutor from “front-line” assessor to “ moderator” of the assessment process. • has the potential to improve both effectiveness and efficiency in assessment