anaesthesia specific appraisal for revalidation 2nd consensus development meeting

1. Anaesthesia Specific Appraisal for Revalidation www.rcoa.ac.uk/revalidation Prof Chris Dodds, Revalidation Lead Mr Don Liu, Revalidation Project Manager revalidation@rcoa.ac.uk

3. Anaesthesia Revalidation Appraisal Exercise Summary of Feedback Feedback from 18 individuals Based on 17 test appraisals and one review of documentation Test appraisals carried out between 25th November 2009 and 15 January 2010

4. Appraisal Process Preparation Appraiser: 2 to 3 hours Appraisee: 4 to 8 hours upwards to the equivalent of two weeks (70 hours) Collection of supporting information over several months / one year Appraisal Discussion 2 to 4 hours Structure of appraisal discussion Progress through attributes 1 to 12 sequentially On average around 10 to 15 minutes spent on each attribute 30 minutes or so - spent discussing Attribute 1 (Maintain your professional performance) and Attribute 2 (Apply knowledge and experience to practice)

5. Was the process overloaded? Did it try and cover too much at once? No We didn’t think the process was overloaded. We were aware that we had an experienced appraiser and an appraisee, who although sceptical at first, was bought-in to the process. It wasn’t hard work to keep things flowing. Inexperience on both sides might lead to a different outcome. Whilst being very thorough the process doesn’t go far enough, particularly where sub-speciality interests are concerned. This leads to the possibility of docs requiring more than one appraisal where they are dual accredited e.g. anaesthetics and ITU. Yes Lengthy but I can’t really see a way round that. It may take less time in subsequent years Yes Too much detail was sought at one time if this is meant to be a summative process. It would be difficult to omit an attribute but perhaps certain attributes could be concentrated on in set years selected by the appraiser and appraise. The process could definitely be more focused. Areas of repetition, overlap and duplication.

6. Elements that could be considered in outside of the appraisal No This is about individuals fulfilling these aspects of responsibility and so therefore is a review of individuals whole practice. This would render process incomplete if process became fragmented. Yes Organisational and departmental activities and responsibilities Considered in separate Trust, departmental and clinical governance reviews Yes and No Organisational and departmental activities – for the revalidation portfolio but not (substantial) discussion during appraisal Mandatory Trust training, e.g. fire safety and health and safety Risk assessment processes Trust quality issues Patient consent Compliance with guidelines Patient safety Incident reporting

7. Supporting Information Difficult or very difficult to collect – why Does not exist locally Has never been collected or not collected adequately by department Data is at an organisational or departmental level and cannot be easily attributed to individuals. Data not forthcoming from clinical audit or clinical governance departments Tools, e.g. MSF and patient surveys, not available or currently inadequate Problems mapping to the CPD Matrix

8. Multisource Feedback (1) Dependency MSF is obviously pivotal to success MSF seems to be important in providing evidence to many attributes. Can we formalise this process and make it compulsory before appraisal? It will be impossible to be appraised in future without an MSF in the last 3 years Availability The main problem is that there is no MSF or publicised/feedback patient survey information in our Trust which meant some of the [supporting] information required by the process was lacking.

9. Multisource Feedback (2) Benchmarking/standards 360 degree information needs to be specified nationally to allow results to compared against benchmarks. At present, the privately generated data is almost useless with no standard content, no standard of who and how many colleagues it should be sent to and thus no possibility of benchmarking. Confidentiality Under new appraisal the MSF will fundamentally change from a private discussion…describing individual areas of excellence and areas needing development…into a much more open document…that will go into the appraisal document for a wider group to see…will take away a great deal of confidentiality in the MSF…it cannot be good for professionals. It’s [MSF] prominence in revalidation makes it more of a public assessment tool and I’ll be bearing in that mind when I next do one and choose responders accordingly.

10. CPD Matrix Very useful My appraisee had already mapped the evidence to the matrix with a 5 year plan in mind Very useful, although it will mostly be used as a tool to guide future CPD Think this will be very useful prospectively for planning one’s CPD and retrospectively for “ticking” off subjects Problems/concerns Complicated on first sight Evidence to support CPD activity in accordance with college matrix difficult to obtain Currently very difficult to map CPD carried out up to now, to the new matrix – I haven’t kept a record of exactly what lecture on which course covered what exact topic. For this to work all courses from now on, and preferably retrospectively, will need to indicate exactly what domains they cover. The CPD matrix resulted in concerns related to excess use of limited time and financial resource use devoted to core topics by those in sub specialty practice Barely fit for purpose, much ‘excellence’ stuff not mappable, major omissions like vascular anaesthesia, too much detail on some topics

11. Other sources of supporting information Audit of notes written in medical records Audit of anaesthetic chart / drug chart Joint care statements by surgeons, ODPs, nurses, etc Log book data Thank you letters/emails/Christmas cards from patients Diary of reflective practice Departmental data Departmental data It became clear that a departmental portfolio was going to become necessary so the all Trust and departmental reports, surveys and audits as well as departmental attendance registers and minutes for meetings could be held electronically in one place. It is clear that the department is going to need to develop a range of policies to be able to support the appraisal process, as we saw on infection control, patient safety, quality management systems.

12. Quality of supporting information Reliability Information systems are going to need to be vastly improved so the Trust really has reliable data on activity and outcomes.  Verifiability  The supporting documentation has to be verified. Who is responsible for doing this prior to the appraisal? An appraiser (or another person) could spend a lot of time checking the information provided by colleagues. Is it better to say the supporting information should be verifiable. Benchmarks .... the quality of the objective information needs to be vastly improved and some formal benchmarks are required – average and ranges from a large database to support decisions that are otherwise entirely at the discretion of the appraiser. Data currently required for appraisals is not easily corroborated and is inadequate to use in judgment for revalidation.

13. Assessment / Judgement (1) Rating system and criteria Useful Pre-appraisal self-assessment Did help to open up / contribute to the appraisal discussion Did help to identify learning/development needs (in most cases) Self-assessment ratings matched final ratings made by appraiser (in most cases)

14. Assessment / Judgement (2) Difficulties in assessing the appraisee Easy / no problems encountered (majority of cases) Moderately difficult – very difficult (some cases) ...patchy evidence was presented for a lot of areas and an MSF was in the process of being completed [Lack of] guidance on minimum standards or data about the average performance of similar doctors This was harder as we haven`t had to do it before I obviously undertook this pilot with a volunteer colleague who was unlikely to have any major issues, when we face a multitude of more difficult/reluctant colleagues the process will become vastly more difficult and time consuming!!

15. Training and Guidance Training Training for both appraisers and appraisees How to successfully structure the interview to ensure it is consistent between appraisals and appraisees Guidance Supporting information requirements for each attribute Benchmarking and standards for supporting information Tools In mapping information to the CPD Matrix and 12 attributes In reducing time spent on documentation