Practically Assessed Structured Scenarios PASS

1. Practically Assessed Structured Scenarios PASS

2. Objectives for this session • To disseminate some of the academic and planning issues of PASS • To give an appreciation of the need to integrate the PASS activity into an ALT strategy and to make the assessments relevant and rigorous • To enable consideration to be taken of the challenges that this assessment method presents • To provide an opportunity for you to create structured scenarios for their own teaching disciplines.

3. “All examinations and assessments should be fair, valid, reliable and feasible and have a positive effect on student learning” (Wakefield, Newble, Wilson, Carter and Helms, 2000; Wakeford, 1999).

4. The development of an Objective Structured Clinical Examination (OSCE) for event management students • OSCE is widely used and considered as a reliable format which can be applied to large groups of students (Harden and Gleeson, 1979) • OSCE is a check list based marking assessment and Newble (2004) believes that this enhances consistency between multiple assessors, and the inclusion of multiple stations increases the number and range of competences that can be tested • There is a wide range of research to support these views, with Newble and Wilkinson being the main authors, and cited most often.

5. Our objectives for undertaking PASS included:- • Replace outdated formal examinations and a report writing assessment • Ensure validity and relevance for industry • Encourage a deeper understanding of learnt material and its application • Minimise time for marking; moderation and recording of marks • Ensure that the assessment is realistic and challenging and reflects pressures of decision making in industry.

6. Importance of matching desired learning outcomes to what is being assessed • Mapping of learning outcomes to structured scenarios • Result was a series of practical tasks, some individual and some group based, which could be completed within 20 minutes.

7. PASS May 2005 • 2 modules were involved in the PASS assessments:- • Operations and Project Management • Event Production • Qualitative research was undertaken to confirm if it was successful in terms of: • Relevance and cohesion • Student feedback: fairness and appropriateness • Management of the day • Speed of marking, moderating and recording marks.

8. PASS May 2006 and May 2007 • 3 modules used the PASS vehicle to assess components of their modules:- • Operations and Project Management • Event Production • Mgt of Human Resources in the Event Industry • A more formally researched approach (Wilkinson et al 2000) was taken to consider the importance of the assessment to events management students • The marks allocated to each station and the time duration were closely monitored • The numbers of stations per module was in accordance to the assessment requirements within the module.

9. Practicalities 1 • 2007 there were 17 groups of 6 or 7 students 9 groups in the morning, 8 in the afternoon. • There were 5 stations, which were duplicated, twice in the morning, twice in the afternoon • Total assessment time on the day = 2.5 hours PLUS 1 hour for multiple choice question examination • 19 members of the event tutor team were involved • 12 stewards were appointed to manage the groups as they moved through the rooms • Facilities Management were involved in preparing the rooms for each station.

10. MORNING SCHEDULE: 9 GROUPS 070517schedule of groups.doc 9.00 ALL 9 GROUPS TO REGISTER IN FG03

11. Practicalities 2 • Each group followed a clear schedule of rooms and the route was clearly signposted • For each room there was a full list of all equipment / stationery required / clock • At each station there were clear instructions for the candidates and for the assessors … the lead assessor and the “buddy” in the duplicate room • Each station had a clear marking scheme • Consideration of arrangements for dyslexic students.

13. Pilot tests • There should be a pilot test in advance with the students to familiarise staff and students with:- • The format • Logistics • Timing • Room requirements.

14. Management of the day • Person/s to co-ordinate the event, registration of each student etc • A steward to accompany each group around the rooms • Lunch break for staff and all stewards • Supervised 15 minute break for students between PASS and MCQ exam • Video recording of the proceedings by Media Services • Photographs taken during the day.

15. Registration of students:- Holder for student card Issue protocol for the day Plastic bag for pen and calculator Storage of personal belongings Groups allocated a steward.

17. PART 1 • To develop a site layout on a scale plan of a large open event space for a music concert • Students placed scaled ‘cut outs’ of facilities e.g. stage; mixer tower; catering; toilets on the plan • They had to consider sight lines and dead space etc to estimate the safe capacity of the event.

18. Tutor asking questions to assess justification of decisions made

19. Students analysing the hazards arising from the mixer tower used in Part 1 (shown in a photograph) PART 2

20. Human resources scenario … individual students watch a 4 minute DVD structured around a staff discussion of recruitment, selection and training The students each identify bad practice and recommend good practice on a pre-printed mark sheet.

21. Student team interviewing the entertainments’ supplier for Leeds Great Student Run, and applying the findings to Tum’s evaluation cycle and making recommendations for following year

22. Individual students were provided with data files on use of toilet facilities and how much coke was drunk at an event to provide data for future events; they had to complete a cross tabulation in SNAP and test it using the Chi-squared method The correlation and regression problem used Excel, and they also had to comment on the reliability of their predictions.

23. Individual students identify hazards from edited video of a live event and complete a “suitable and sufficient” risk assessment form.

24. Validation and evaluation of PASS performance • The tasks / activities were mapped onto a learning outcomes matrix in advance of the day • Station results moderated e.g. by photograph or marks to give interstation reliability / internal consistency • Inter-assessor reliability was gauged using co-efficient of correlation • All module results for the level were compared statistically at the Exam Board • In addition, at the end of the MCQ exam all students completed, a short questionnaire which was entered onto SNAP • Student focus groups were held on the same day and the following day.

25. Feedback – staff and students • Student feedback provided:- • Student perception • Authenticity of assessment • Fairness and timing • Staff feedback provided:- • Student performance • Station performance and appropriateness • How to improve logistics and process.

26. Management of Human Resources in the Event Industry Module

27. The chart shows the generally lower marks for PASS and the variation in marks between students and between components for each student.

28. There was general agreement that the assessment is testing vocational application of the skills and knowledge learned.

29. One problem of using PASS to test several modules at once is that poor performance on the day can produce stress. In our case the assessment covers 50% of one module, 50% of Event Production, and 35% of HR. It is therefore not surprising that some students find it stressful.

30. PASS and presentations are seen as the most relevant in their future careers .

31. Your turn now!! In small groups:- • Which modules could you use PASS for? • Create 1 or 2 scenarios for a module • What benefits can you see from PASS? • What challenges can you envisage?

32. Thank you! Any more questions or points to be made?