How honest are the students when assessing each other’s work? Diogo Montalvão

How honest are the students when assessing each other’s work? Diogo Montalvão d.montalvao@herts.ac.uk Senior Lecturer, Dipl MSc PhD CEng School of Engineering & Technology 2012 Annual L&T Conference, 3rd May

Outline Why student’s peer-assessment? Description of a case study at IST Some of the students’ works Peer-assessment results Conclusions

Why student’s peer-assessment? Student benefits source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

Why student’s peer-assessment? Teacher benefits source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

Why student’s peer-assessment? • Disadvantages may be encountered when students: • Lack the ability to evaluate each other • Do not take it seriously • Fear discrimination source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

Description of a Case Study at IST* *Instituto Superior Técnico, Technical University of Lisbon • Modules: • Technical Drawing and Geometrical Modelling I & II • 1st year students • 2 groups/module, ~50 students/module • BEng / MEng in Mechanical, Aerospace and Marine Engineering • 2010/2011 session

Description of a Case Study at IST Technical Drawing and Geometrical Modelling I & II - Assessment “Other” tests 10% *CAD – Computer Aided Design

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 • Goals: • To engage the students in practice and study • Without significant increase of the marking load 0 1 Wrong Right How do we define the boundary in a binary marking scheme for drawings? This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 • Goals: • To engage the students in practice and study • Without significant increase of the marking load 0 .5 1 Wrong Very good Acceptable How to differentiate a student that does not deliver the assignment from the others? This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

(“Other” tests – 10%) Every week in-class 15~30 mins assignments marked from 0 to 1 • Goals: • To engage the students in practice and study • Without significant increase of the marking load 0 .25 .5 .75 1 Zero Poor Good Very good Accept. 2 levels of fail 3 levels of pass • Problems: • Poor feedback • “Ethical” question: marking is based on comparison This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

(“Other” tests – 10%) Examples of “other” tests 1 0.75 0.5 This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

Description of a Case Study at IST Technical Drawing and Geometrical Modelling I & II - Assessment Group presentations 10% Lecturer 5% Peer-assessment 5% *CAD – Computer Aided Design

Description of a Case Study at IST About the public presentation • Done in-class, during last week • 10~15 minutes/group • Students had to attend all the presentations • Marks ranged from 1 to 10 (0 for no shows) • Peer-assessment marks kept confidential by the lecturer • Non-differentiating peer-assessment marks were rejected 1-2 3-4 5-6 7-8 9-10 Very Poor Poor Good Very good Accept.

Some of the students’ works Reactive suspension Leonor Neto SoraiaRibeiro Boring mill Wilson Carreira Hélder Camacho Filipe Moreira Mower Rodrigo Ferreira

Some of the students’ works A Racing bike!!! (Solidworks) (Author deliberately left anonymous)

Peer-assessment results • First observation: • Both the teacher’s and students’ marks follow a Gaussian distribution.

Peer-assessment results

Peer-assessment results The students did not fail their colleagues

Peer-assessment results The two students with very poor presentations are given a pass by their peers...

Peer-assessment results ... and the four students just below pass were given a comfortable pass by their peers

Peer-assessment results 72% of the marks given by the students are in the 7-8 (Good) range

Peer-assessment results However, the students seem to be afraid to overrate too much their colleagues

Peer-assessment results • The averaged values are not very different • The teacher’s standard deviation is practically twice as much as the peers’

Peer-assessment results The marks are concentrated around the same mode, although the teacher’s assessment “leaks” towards the whole marking range.

Conclusions • Overall conclusions from this study: • Students have a tendency to overrate their “mates” • They are able to make a correct relative judgement between their peers’ performances (and thus themselves) • Most often, they will avoid to: • Fail their “mates” • Give their peers topper marks

Conclusions How honest are the students when assessing each other’s work? Qualitatively honest Quantitatively biased Students move by their own code of conduct and live in a competition environment.

Acknowledgments are due to Prof.AntónioRibeiro, Assistant Professor at IST, for sharing the data from two of their groups of students. THANK YOU Diogo Montalvão d.montalvao@herts.ac.uk Senior Lecturer, Dipl MSc PhD CEng School of Engineering & Technology

How honest are the students when assessing each other’s work? Diogo Montalvão