The Royal College of Pathologists – the overall assessment system

1. The Royal College of Pathologists – the overall assessment system Dr. Trevor Gray Director of Examinations and Assessment

2. The Assessment System • Overview of PMETB and assessment strategy • How WBA fits with other assessments • Advantages and disadvantages of WBA compared with other assessments • Local organisation of WBA and how it is administered by the College • Future prospects

3. Overview • PMETB (and the MMC process) has laid down significant challenges to medical postgraduate training (elegantly dissected in the Tooke report) • PMETB principle is that training and assessment should be much more integrated – an assessment system • Workplace-based assessment is an essential element of the assessment of trainees undertaking specialist medical training • The specialty curriculum, training, exams and assessments are core College business (in the charter) • The College’s aim is to develop a validated assessment system which is fit for purpose

4. PMETB strategy on assessment • PMETB principles for assessment 2004 • Nine principles for assessment systems • Now backed up with technical documentation • Principles based on “overarching strategy” • Workplace-based assessment • Examinations of knowledge • Examinations of Clinical Skills (“Workplace-based assessment” PMETB, Jan 2005) • Specialty curriculum and assessment system now approved by PMETB • (i.e. match strategy)

5. Workplace-based Assessment • Assessment based on performance in the workplace and measured there • Assessed locally by supervisors as well as staff of all grades in contact with the trainee • Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance • Results put into trainee portfolio for their annual review and hence important for trainee development and ultimately their employment contract

6. College Assessment Systemapproved by PMETB • Workplace-based assessment • Year 1 assessment • MRCPath Part 1 (test of knowledge) • MRCPath Part 2 (test of skills and applied knowledge) • Feedback on regular basis (WBA) • Educational supervisor’s report • ARCP (Annual Review of Competency Progression)

7. The College Assessment system: Miller’s pyramid WBA - MSF Does WBA e.g. Mini-CEX, DOPS Shows MRCPath Part 2 Practical Knows how MRCPath Part 1 Knows

8. Value of multiple assessments • Key is several modes of assessment – triangulation • A single exam system (MRCPath) does not provide adequate coverage of clinical practice • WBA is also not sufficient in itself (Assessment System Guidance – PMETB)

9. Examinations and Assessment • Examinations are simply one particular form of assessment • Assessments are: • formative (involve feedback and are intended to guide training) or • summative (assess whether reach given standard) e.g. driving test, MRCPath • All assessments should be “blueprinted” to the curriculum to demonstrate that the most appropriate method is being used

10. How Workplace-based Assessment fits with MRCPath • Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee • Examinations provide an independent assessment of knowledge and practical skills to national standards • Workplace-based assessment provides a local insight into candidates’ performance in routine tasks and interaction with staff and patients

11. How Workplace-based Assessment fits with MRCPath • Workplace-based assessment is a local process governed by deaneries albeit to PMETB standards • Examinations are organised nationally by the College • Comments, complaints and appeals for WBA go to deanery, for exams go to College

12. Workplace-based Assessment – Advantages • Measurement occurs in more authentic environment than the simulated one of the examination room • Trainees can take assessment in their own time and are not limited to exam timetable • The assessment tools are set up to provide extensive feedback to the candidate (formative) • Use of computerised forms makes them easy to record

13. Workplace-based Assessment – Disadvantages • To avoid time-consuming “busy” assessment forms, the assessments are often simplified if not simplistic • A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties • They have only been validated in a number of clinical settings but not in all pathology settings • Computerised forms can over-simplify complex judgements and may not reward excellence appropriately

14. Year 1 Assessment (OSPE) • Developed to provide evidence of progression in run-through training in the absence of further recruitment and selection • Designed to assess progress in first year and be taken after 7 months training • It is a test to “indicate the capability and potential of a trainee through a test of applied knowledge and skills relevant to the year 1 curriculum”

15. Training and Assessment Plan ARCP F1 F2 ST1 ST2 ST3 ST4 ST5 Foundation Stage A Stage B Stage D Stage C WBA Workplace-based assessment MSF Y1A MRCPath CCT Part 1 Part 2

16. The training environment • Workplace-based assessment utilises the skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff • The College cannot train or quality assure all assessors • Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised and training is cascaded from those attending events such as this

17. Workplace-based Assessments and the College • WBA is part of assessment system for Specialist Trainees from 1st August 2007 • Assessments need to be undertaken from early in training and recorded for ARCP • 2007 – Launch of tools and paper-based assessment forms • College is researching web-based systems similar to the ones used in foundation

18. Workplace-based Assessments and the College • April 2008 – pilot web-based system • By August 2008 – launch of web-based system • Use data generated by all assessments including MSF and OSPE for evaluation of assessment strategy • Ensure compliance with MMC Gold Guide and planned ARCP

19. Vision for College Assessment system • College will have integrated specialty curriculum and assessment system endorsed by PMETB • College will have fully validated workplace-based assessment tools delivering feedback on progress to trainees throughout course • Valid and reliable tests will underpin MRCPath examination – Part 1 test of knowledge, Part 2 test of applied knowledge and skills

20. Factors affecting future assessment strategy • Selection into specialty may not be an annual process in future compromising ability to deliver year 1 assessment (except maybe in Histopathology) • Multiple recruitment may require more flexibility in arrangements for WBA • Selection into specialty may involve a national set of tests (Tooke and Academy) displacing Year 1 assessment

21. Factors affecting future assessment strategy • Workplace-based assessment is likely to continue but its use must be evidence-based • Workplace-based assessment is not cast in stone – work in progress and subject to improvement following evaluation • Place of individual components of assessment in MRCPath, Year 1 assessment and workplace-based assessment may change

22. Summary • The College’s education and training, assessment and examinations departments have worked hard to produce curricula and assessments acceptable to PMETB • This training day is the first for workplace-based assessment and is intended to assist training centres to develop assessment locally • Workplace-based assessment will be subject to a full programme of validation and optimisation to ensure fitness for purpose • The College is looking to source a web-based system to support workplace-based assessment

23. Glossary ARCP Annual Review of Competence Progression CbD Case-based Discussion DOPS Direct observation of practical skills ECE Evaluation of clinical events MSF Multi-source feedback (ePATH-SPRAT) Mini-CEX Mini clinical evaluation exercise OSPE Objective structured practical examination PMETB Postgraduate medical education and training board WBA Workplace-based assessment