The use of Web-PA as a formative peer assessment instrument for team working

1. The use of Web-PA as a formative peer assessment instrument for team working Tom Joyce, Nuala Davis* and Clare Hopkins School of Mechanical & Systems Engineering *Information Systems and Services Newcastle University 24th June 2013

2. Outline • Context for the use of Web-PA • Use of online peer assessment tool (Web-PA) as a means of identifying and addressing team difficulties • Technical outline of the use of Web-PA

3. School of Mechanical and Systems Engineering, Newcastle University • Offer BEng and suite of MEng degrees • ‘Typical’ engineering student – male, post ‘A’ level, UK national • Increasing number of international students (24% of Stage 1 2012/13) • Like many engineering Schools, progression is an issue

4. School of Mechanical and Systems Engineering, Newcastle University

5. School of Mechanical and Systems Engineering, Newcastle University

6. Introduced in 2009/10 academic year Pre-selected group of 5 first-year students ‘Balanced’ in terms of academic ability Overlap with tutorial system Work together on Team projects in two modules (“Design & Manufacturing” and “Professional Skills” – total 45 credits) during first year A proposed solution – “Engineering Teams”

7. Why Engineering Teams? Increase in student integration – students get to know other students more quickly Informal peer instruction and learning, sharing skills and knowledge Team-working on projects gives shared goals

8. School of Mechanical and Systems Engineering, Newcastle University

9. School of Mechanical and Systems Engineering, Newcastle University * intervention year

10. School of Mechanical and Systems Engineering, Newcastle University 82% 91% * intervention year

11. Students’ views on team working • In 2012/13 academic year 91% of students said that they enjoyed/enjoyed very much working in an Engineering Team • 86% said that their Engineering Team had formed a good working relationship • 71% said that working in an Engineering Team had helped them to feel part of the School

12. But when there are problems • Problems occur for a small number of Engineering Teams when members fail to contribute in a variety of ways • “Some people in the team don't do anything at all. They don’t turn up to any meetings or labs and I’m surprised they even know we're making a wind turbine. They should get no marks at all if they haven’t contributed to the team.”

13. Peer–moderated marking • Used in Design and Manufacturing module • Teams decide division of marks at end of 1st and 2nd terms • Anecdotally, we hear of some heated discussions • Some groups want to avoid face-to-face conflict

14. Measures taken to further address team-working (2012/13) • Enhanced team building exercises on second day of induction period • An Engineering Team handbook with information and advice about team working, including ways of addressing non-contribution • Use of Web-PA peer assessment to facilitate feedback between team members

15. WebPA • What contribution did each member make to the group effort? • Average = Teacher awarded mark

16. Tutor I • Tutor creates a form with assessment criteria • Defines the start and end times for the assessment • Reminds students to fill it out

17. Students • Students fill the form out online • Confidential

18. Tutor II • Enters group scores in WebPA • Generates a “Mark Sheet” based on their chosen weighting for peer marked element • Examines and moderates results

19. Using Web-PA as formative peer assessment • Took place eight weeks into first term • 111 students took part 15 of whom were women and 22 international students • self and peer assessment using freetext • Conducted in computer lab where teams could sit separately from one another • Feedback given to teams in anonymised form under the guidance of tutors

20. Key learning points from the use of Web-PA • Of the 537 pieces of feedback given, the majority (92%) were constructively written • Large degree of agreement between students’ self assessment and the feedback provided by their peers • Only 43 comments (relating to 20 students) 8% of the responses were negative

21. Common reservations about using peer assessment • Students may be unprofessional and give highly critical feedback, leading to acrimony and breakdown within teams • Students may assess themselves more positively than they assess their peers as a way of gaining advantage • Students may use anonymous feedback as a way of expressing personal resentments

22. Reservation 1 – students may give unprofessional/critical feedback What we found: • Of the 537 pieces of feedback 92% was balanced and constructive • Four students who had contributed to a lesser extent for a reason that their peers believed to be valid received supportive, uncritical feedback

23. Reservation 2 – students may be over-positive about themselves What we found: • A very high level of agreement between team members self assessment and how they were rated by their peers (only 2 cases where this was not the case) • In 15 cases self assessment was deprecatory. In only 7 of these was this negative assessment confirmed by peers

24. Reservation 3 – criticism on the basis of personal feelings What we found: • Only 43 comments were negative (related to 20 people) – 8% of responses • Criticisms were related to team performance – absence; personal disorganisation; failure to contribute to the team; poor quality of work.

25. Fear 3 – criticism on the basis of personal feelings (cont) • Even when the majority of comments were negative, in more than half there was also an acknowledgement of some positive aspect “attended all meetings” “however he didn’t lack effort” “but he does try” • In a very small percentage (5%) there was a disparity between single negative comments and other feedback

26. Issues to be addressed before next use of Web-PA formative assessment • The reluctance/avoidance of a small number of students Addressed by – provision of information about use of Web-PA during induction period/EngineeringTeam handbook • The tendency of some international students to be tentative/self-deprecatory Addressed by – targeted measures to engage them in the process/help them overcome reluctance

27. Students’ perceptions of Web-PA • 91% of those who completed it rated the system as technically easy to use • 49% said that they were able to be more honest in their ratings because they did not have to share them directly with their team mates “It is good to be able to rate your team mate, and share your opinion of them without having the embarrassment of a face to face confrontation”

28. Students’ perceptions of Web-PA • It was seen as a good way of addressing problems within a team “It allowed me to focus on the aspects of the team that were good but more importantly, it helped to highlight the shortcomings of the team so that they could be addressed and remedied”.

29. Students’ perceptions of Web-PA • As a useful way of bring up issues without embarrassment “Allows people to bring up any issues anonymously. Without worry.”

30. Thank you Any questions?