The nMRCGP Assessment

1. The nMRCGP Assessment Theoretical overview The components Practicalities UKCEA Workshop 17 April 2007

2. Principles of test development • Define the purpose of the assessment • Define test content • Selecting test methods and formats • R&R, Test administration • Scoring issues…. standard setting • Reporting scores/ feedback • QC….. ongoing development

3. The nMRCGP • Is an outcomes-based assessment system • Relates to the entire training period • Is set at a standard expected of doctors being licensed to practise as independent general medical practitioners in the United Kingdom

4. nMRCGP blueprint • Assessment system is ‘matrix derived’ from a ‘blueprint’ • ….which is derived from the knowledge, skills, behaviours and attitudes (the intended learning outcomes ) that are defined by the RCGP training curriculum • And this overarching and triangulated blueprint is referenced to all of the areas of Good Medical Practice

5. The nMRCGP was designed • In accordance with best assessment and educational practice: specifically, each assessment tool was chosen on the basis of its strengths and weaknesses • Ensuring that decisions regarding scoring were based on rational arguments and referenced (where appropriate) to contemporary psychometric and educational evidence

6. The nMRCGP assessment system has been chosen on the basis of its • validity • reliability • feasibility and deliverability • cost-effectiveness • opportunities for feedback • impact on learning

7. Test length…reliability (alpha)

8. The following key features underpin the nMRCGP • The proposed assessment tools have been extensively piloted to establish their utility • The e-portfolio will provide evidence for revalidation and compliance with Good Medical Practice • There has been lay and expert input in the development of the assessment • Assessor recruitment is based on pre-defined criteria and their training is tailored to specific assessment tasks

9. The following key features underpin the nMRCGP • Standards will be set using recognised methods based on the test content and the judgements of competent assessors • Systematic data collection will support the routine reporting of the reliability and precision of the pass/fail decision • Adequate resources have been made available for development and implementation of assessment methods and training of assessors

10. Purpose of nMRCGP • The principal purpose is to confirm that a doctor has satisfactorily completed specialty training for general practice… and is sufficiently safe and proficient to enter independent general practice in the United Kingdom • Satisfactory completion of the nMRCGP will be a pre-requisite for the issue of a certificate of completion of training (CCT)

11. Purpose of nMRCGP • The nMRCGP will also provide feedback on progress during training for trainee GPs, their trainers and deaneries, particularly through workplace-based assessment • It will help to identify areas of weakness and support deaneries in planning remedial work where this is needed

12. MRCGP: an integratedassessment programme

13. The applied knowledge test (AKT) • Is a summative assessment of the knowledge base that underpins independent general practice within the UK • A rigorous multiple-choice test is an efficient and reliable means of assessing a trainee's ability to recall and apply relevant knowledge • Candidates who pass it will have demonstrated their competence in applying knowledge at a level which is sufficiently high for independent practice

14. The clinical skills assessment (CSA) • Is a summative assessment of a doctor's ability to integrate and apply clinical, professional, communication and practical skills appropriate for general practice • Simulates a UK GP surgery clinic

15. CSA • The CSA was designed following careful consideration of the utility of the current range of available test methods

16. CSA • Is an integrative skills assessment, testing a doctor’s abilities to gather information and apply learned understanding of disease processes and person-centred care appropriately in a standardised context, making evidence-based decisions, and communicating effectively with patients and colleagues

17. The CSA can…. • Assess a range of cases written for scenarios from primary care and played by role players • Assess cases relevant to most parts of the curriculum and can target aspects of clinical care and expertise • Sample cases according to pre-set stipulations from the blueprint • Present a standardised and pre-set level of challenge

18. Workplace based assessment • WPBA is the evaluation of a doctor’s progress over time in their performance in those areas of professional practice best tested in the workplace.

19. nMRCGP WPBA model • Consists of a competency-based Enhanced Training Record (ETR) feeding into which will be evidence gathered from • Locally delivered workplace assessments • Naturally occurring evidence • Supervisor structured reports • Overall WPBA judgements will be subject to deanery panel reviews and national quality control

20. WPBA • Provides an opportunity for gathering evidence of actual performance in the workplace • Allows assessment of aspects of professional behaviour that have proved difficult to assess in traditional examinations • Provides feedback on areas of a trainee's strengths and development needs

21. WPBA • Identifies trainees in difficulty • Drives learning in important areas of competence • Determines fitness to progress onto the next stage of the trainee’s career

22. PMETB ‘An independent statutory body’ • Establishing standards and requirements for postgraduate medical education and training • Making sure these standards and requirements are met • Developing and promoting postgraduate medical education and training across the UK

23. What is required? • Compliance with Principles 1 and 2 • Indication that work has begun to comply with Principle 5 • Plans for complying with all of the PMETB principles

24. PMETB Principles of Assessment Principle 1: The assessment system must be fit for a range of purposes • What range of assessment methods are proposed? • What is the purpose of each method? • At which stage is the examination done and why? • What does the trainee need to have achieved for progression?

25. Principle 2 • The content of the assessment will be based on curricula which are referenced to Good Medical Practice • A blueprint must be constructed which maps the test content against learning outcomes/ competencies

26. Principle 5 • Assessment must provide relevant feedback • What methods are in place to provide feedback to the trainee? • The process for providing feedback must be clearly stated

27. Practicalities: CSA • The CSA will be delivered in a single, purpose built, clinical skills assessment test centre. Eventually, this will be part of new College premises; in the meantime, an interim site has been procured, which will be fitted out for the purpose. Assessors will be appointed, trained and standardised by RCGP, and standard-setting procedures will be managed by RCGP.

28. Practicalities • The AKT will be delivered as a computer-based multiple-choice test in secure Pearson Vue test centres throughout the UK. Each form of the test will be drawn from an item bank administered by RCGP, and standard-setting procedures will be managed by RCGP.

29. Practicalities: WPBA • WPBA will be locally delivered by deaneries so that teaching, learning and assessment are closely linked. It will consist of a structured longitudinal assessment of a trainee’s progress over the entire three year training programme, mediated by regular locally staged and evidenced reviews

30. Practicalities: WPBA • Assessment will be based on a competency-based enhanced training record (ETR) which will be developed, maintained and reviewed by means of a web-based electronic portfolio of evidence for each trainee.

31. Practicalities: WPBA • Each WPBA review will be informed by core information including both naturally occurring evidence and that provided by specifically designed tools selected for local use in the workplace including: • Case based discussion (CbD) • A consultation observation tool (COT) • Other tools as appropriate in, for example, hospital settings: e.g. mini-clinical evaluation exercise (mini-CEX) or direct observation of practical/procedural skills (DOPS)