Overview

1. Workplace-Based AssessmentProcedure Based (PBA)These slides have been prepared to facilitate discussion on the use of the PBA. The suggested practical group exercise at the end is designed to be used with a training video.

2. Overview PBA Blueprint How many PBAs? PBA validity and reliability Preparation for the PBA During the PBA After the PBA Group exercise Further Reading Acknowledgements • What is the PBA? • Purposes of the PBA • PBA Form • PBA Worksheet • PBA Domains • Completing the form • PBA Global Summary • PBA vs elogbook levels • How does the PBA work? • Who should use the PBA?

3. What is the PBA? • Mandatory workplace assessment for learning • An observation of the trainee performing a practical procedure usually in the operating theatre • Available for advanced specialty index procedures • Covers patient safety, consent, pre-operative planning and preparation, intraoperative technique, communication with the team and post-operative management

4. PBA vs DOPS DOPS: • The counterpart of the PBA • For the simpler index procedures for Core Training • Set in Outpatients or Treatment Room • Adaptable as a portion of a more complex procedure • Should be developed alongside the PBA

5. Purposes of the PBA • To provide feedback to help the trainee improve their technical and procedural skills, decision making, leadership and professional skills • To encourage a reflective approach to learning • To help the trainee build evidence of their progress • PBAs form one of the indicators that inform the AES’s end of placement report for the ARCP • Individual PBAs are not designed to sign off independent practice.

6. PBA Form

7. PBA Worksheet Provides examples of positive and negative behaviours that are associated with each competency Serves as an internal check of the validity of the competency Provides a useful guide for assessors and trainees

8. PBA Domains Six core domains: • Consent • Pre-operation planning • Pre operative preparation • Exposure and closure • Intra operative technique • Post operative management

9. Completing the form • Rated to the standard of Certificate of Completion of Training (CCT) • Duration of the procedure allows analysis of progression • Duration of the assessment includes feedback element • Operation more difficult highlights unusual problems • Simulated setting allows PBAs to be used in skill labs etc. • Competencies marked Gfor generic allow cross specialty analysis • Competencies marked T are procedure-specific • 3-point rating scale plus Not assessed for areas outside remit or not observed • Area for written feedback • Trainee can add reflection when PBA is uploaded • Trainee can link to syllabus topics when PBA is uploaded

10. PBA Global Summary • Four levels describing of the trainee’s overall competence on that occasion • Maps to surgical logbook levels • Level 4 is equivalent to a specialist in NHS practice • Level 4 in any one procedure does not permit a trainee to perform a procedure unsupervised

11. PBA vs elogbook levels

12. How does the PBA work? • The trainee selects an advanced procedure, representative of their normal workload • The setting is in the operative theatre, scrubbed • The assessor observes the trainee carrying out the procedure OR parts of procedures • The assessor and trainee discuss the procedure using the PBA form as a guide • The assessor provides feedback on progress, covering strengths, exploring development needs and agreeing action plans within a feasible timescale. • The assessor records their feedback on the PBA form and provides a rating against competencies where applicable • Takes as long as the procedure with 10-15 minutes feedback

13. Who should use the PBA? Trainee • The PBA is for higher specialty training - ST3 and above Assessor • Consultants • Trainee’s current AES • Senior trainees depending on training level and complexity of procedure • Senior SAS grade surgeons Assessors must have received training in the tool and be expert in the clinical problem/task

14. PBA Blueprint Assessment blueprints help to ensure that assessment methods are mapped to the domains of the curriculum and Good Medical Practice. A single method will cover some domains and the range of assessments should cover all domains ** Directly assessed * Indirectly assessed

15. How many PBAs? • To be used throughout surgical training • Ideally performed every time the trainee is supervised undertaking an index procedure • Should be of a sufficient number at level 4 but will vary according to the complexity of the procedure and experience of the trainee • Should be agreed with the trainee’s AES • The JCST Quality Indicators state that training posts should offer at least 40 WBAs per year, of which the PBA should be of an equal proportion to other methods. This is checked at the ARCP

16. PBA validity and reliability Validity and reliability is enhanced by: • Assessor training and trainee induction • Linking to Learning Agreement objectives • Use as an assessment for learning • Using frequently throughout training • Using a range of assessors who are qualified and trained • Use across index procedures • Increasing the complexity according to the trainee’s competence and progression • Feedback addressing trainee development needs • Triangulation with the surgical logbook • Trainee reflection on feedback

17. Preparation for the PBA Trainee: • Arrange the case and PBA with the assessor well in advance • Arrange to have the form to hand, either printed or online • Reflect on the procedure beforehand Assessor: • Receive training in the tool • Ensure it is a procedure in which you have expertise Both: • Ensure the patient has given consent • Make suitable time for the procedure and feedback • Agree the appropriate level of supervision

18. During the PBA Trainee: • Carry out the procedure as you would normally • Explain what you intend to do throughout • Ensure the patient suffers no increased risk or discomfort Trainer: • Observe the encounter • Intervene if patient safety is at risk • Immediately afterwards and in a suitable environment, discuss with the trainee, giving the trainee the opportunity to give their view • Summarise the feedback including what was done well and what needs to be improved • Expand on areas that were less than satisfactory, including suggestions for remedial steps and actions • Ensure the form captures feedback and areas observed

19. After the PBA Trainee: • Upload feedback to the portfolio accurately in good time • Reflect on performance and feedback received • Follow up action plans Trainer: • Validate the assessment in good time • If necessary, follow up on any actions agreed • Discuss the trainee’s needs with the AES if necessary

20. Group exercise Purpose: To practice rating PBA, discuss best practice and enhance consistency Technique: With a group of new and experienced trainers: • Group discusses the meaning of the PBA competencies • Group views a PBA video (see ISCP PBA example videos) https://www.iscp.ac.uk/surgical/assessment_observation.aspx(pause at point of feedback in order to discuss the trainee’s performance) • Each trainer completes the PBA Form ratings and writes feedback comments based on the video • Group compares ratings and discusses written feedback • Each trainer reflects on whether they would adjust their ratings/comments in light of discussion Repeat the exercise with another video (alternatively a role play). Further discussion: • Use of scenarios in a simulated setting to help trainees practice skills

21. Further reading • ISCP Guidance notes on using the PBA https://www.iscp.ac.uk/static/public/cex_guidance.pdf • ISCP Tips for using PBA https://www.iscp.ac.uk/static/public/cex_tips.pdf • PBA Form https://www.iscp.ac.uk/static/public/cex_form.pdf

22. Acknowledgements The PBA was derived from OSATS by the Surgical Education Research Group, Department of Surgery, University of Toronto. The use of the PBA in the UK was pioneered in Orthopaedics by the Orthopaedic Curriculum and Assessment Project (OCAP) The OpComp Project in General Surgery led to further updates The surgical Specialty Advisory Committees (SACs) developed PBAs for the surgical specialties within for the Intercollegiate Surgical Curriculum Programme (ISCP) The ISCP PBA was validated by Professor Jonathan Beard http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0003/64749/FullReport-hta15010.pdf