WA Health workplace-based assessment for international medical graduates

1. WA Health workplace-based assessmentfor international medical graduates

2. Australian Medical Council • Three pathways to registration: • Competent Authority • Specialist Pathway • Standard Pathway • Assesses the knowledge, clinical skills and professional attributes of international medical graduates (IMGs) on the standard pathway only seeking registration to practise medicine in Australia.

3. Australian Medical Council exams • English language exam • AMC1 MCQ 180 questions • AMC2 clinical exam MCAT/OSCE • Competency assessment • Role plays (pass = 12/16) • Pass rates in October 2016 were 35% (i.e. 106 of 305 IMGs passed)

4. AMC exam outcomes • After sitting the AMC clinical exam, IMGs historically demonstrate: • Performance issues • Poor communication skills • Inability to integrate well into a team • Poor understanding of cultural awareness • Poor clinical judgement was most significant concern • Outcome is less likelihood of retaining in WA Health

5. Miller’s Pyramid of Clinical Competence Further information: http://winbev.pbworks.com/f/Assessment.pdf

6. Workplace-based Assessment Standard Pathway • AMC examinations • AMC MCQ CAT and the AMC clinical • Assesses at ‘knows’ and ‘knows how’ levels • Workplace-based Assessment (WBA) • AMC MCQ CAT and a program of WBA of clinical skills and knowledge by an AMC accredited authority • Assesses at the ‘does’ level • Focuses on performance in the workplace setting • Provides opportunities for immediate constructive feedback

7. Workplace-based Assessment • Introduced under the 2007 COAG Nationally consistent assessment of IMGs initiative. • Established to assist with: • Workforce issues, particularly in rural and regional areas • The large number of IMGs waiting to sit the AMC clinical examination. • WBA has been adopted at 15 hospitals nationwide across 7 health providers. • WBA programs are accredited in accordance with AMC WBA Guidelines and Criteriawhile taking into consideration local circumstances.

8. WBA opportunities • Attraction and retention of clinical staff in the many WACHS sites continues to be a challenge. • Upsurge in Australian-trained medical graduates but continued reliance on IMGs for service delivery. • WBA has an significant impact on interest in employment at participating WACHS hospitals. • There are approximately 20 candidates every year. • Of those remaining in WA after obtaining general registration, on average, 70% continue to work in rural areas.

9. WBA benefits • Rigorous with a much higher pass rate than the AMC clinical. • Supports integration into the local community and assimilation into the Australian health workforce. • Assessment and feedback methods provide consistency of supervision and assessment techniques within the wider hospital community.

10. What is WBA • A form of authentic assessment, testing performance in everyday clinical practice involving the measurement of abilities and attitudes within highly complex areas of activity. • Must have established reliability and validity to ensure that defensible decisions are made through using: • A variety of assessment methods • Multiple observations in various clinical settings • Trained multiple assessors

11. What is WBA • Delivered in all states and well supported. • Involves observation, assessment and immediate feedback of candidates in the workplace. • Global ratings developed. • Longitudinal assessment over 12 months. • Can track development towards independence. • Improves clinical skills, communication and teamwork. • Identifies under-performance and has action plans to support improvement.

12. WBA in WA • The WA Department of Health is the AMC accredited provider for WBA. • Hospitals implementing WBA must be AMC accredited. • Currently accredited sites: Bunbury and Geraldton Hospitals and Kalgoorlie Health Campus • Accredited assessment plan spans over 9 months. • The length of the WA WBA program aligns with the Medical Board of Australia requirements for eligibility for general registration.

13. Entry to the program • Complete AMC MCQ, English exam and Electronic Portfolio of International Credentials (EPIC) verification • Must be eligible for limited registration on the standard pathway with the Medical Board of Australia • Employment with WACHS (contract to be provided)

14. Assessment? Assessment OF learning……… OR Assessment FOR learning……..

15. Assessment Assessment level “an Australian intern at the end of the PGY1 year” or “that which one would expect from a minimally or just competent medical officer at the end of PGY1” Further information Intern training – intern outcome statements Australian Curriculum Framework for Junior Doctors

16. Assessment goal • Asampling of a candidate’s performance. • Must be representative of the spectrum of that relevant to safe medical practice. • Assessors need to be sufficiently familiar with the patient to enable them to critically judge the performance being reviewed.

17. Assessment process • Part of everyday clinical practice. • Candidate is responsible for own learning. • 6 clinical areas and 6 assessment tools. • 12 month program. • Must pass a set number of tools (candidate blueprint). • Must have minimum 10 hours of exposure to a clinical area (i.e., ward rounds, clinics etc) before being assessed against clinical skills and dimensions.

18. What is an assessor? Clinicians who are: • Experienced, and possess expertise in the clinical tasks assessed. • Trained in WBA for target cohort (IMGs). • Available. • Have agreed to assess according to the principles of WBA.

19. What can an assessor do? • Select the patient/case and assure consent is received. • Ensure the candidate has undertaken adequate preparation. • Conduct direct or indirect assessment in a clinical area and provide immediate feedback. • Undertake a formative assessment and provide feedback if requested.

20. Direct assessment rules • General or specialist registrants who have successfully completed 4 years of experience in the Australian health care environment. • OR • Equivalent experience trained in a designated Competent Authority country. • DOPS may also be assessed by registered nurses with appropriate clinical assessment experience. • Other AMC candidates are not to be included as assessors or patients

21. Indirect assessment rules • The WBA provider should have clear statements of the expertise and experience required for the appointment of assessors. • All assessors must have: • Completed the WBA assessor and supervisor training package. • Signed the declaration that they have completed this training. • In the case of MSF, the candidate and/or WBA provider may choose to include other members of the health care team.

22. Conflict of interest • AMC and WA Health take this issue very seriously • The WA Health Employment Policy Framework applies to the all WA Health staff. • Specifies employment governance requirements that apply across the WA health system. Further information http://www.health.wa.gov.au/CircularsNew/Employment.cfm

23. Assessment areas • Must complete assessments across the 6 clinical areas covered in the AMC clinical exam • Child health and women’s health are not traditionally assessed at intern level. • Mental health has been identified as a clinical area that some candidates may not be familiar with.

24. Assessment blueprint • Assessment drives learning • Key is to assess ‘important things’ to learn and do. • A ‘blueprint’ should guide selection of encounters:

25. Blueprint - macro

26. Blueprint - micro

27. Assessment tools • 12 mini-clinical examinations (Mini-CEX) • 6 direct observation of procedures (DOPS) • 6 case based discussions (CBDs) • 3 in-training assessments (ITAs) • 1 multi-source feedback (3600) • 1 external assessment • Additional assessments • 1 Self-assessment and Learning Plan • 2 formative ITAs • 5 candidate self-assessments

28. WA Health approved assessment plan Direct assessment methods • 12 Mini-CEX – 2 in each clinical area (or more as needed) • 6 DOPS – 1 in each clinical area (or more as needed) • 1 external assessors report – direct observation (calibration tool) Indirect assessment methods • 6 CBD –1 in each clinical area (or more as needed) • 2 Formative supervisor’s reports. Between 18-20 weeks apart • 2 Summative supervisor’s Between 18-20 weeks apart. • MSF – 10 colleague evaluations usually undertaken mid-year. Collated into a report for the primary supervisor to discuss with and provide feedback to the candidate.

29. Mini-CEX • Process of directly observing a candidate in a focused patient encounter for purposes of assessment, followed by feedback on performance. • Observation is typically 10-15 minutes on a focused task. • 10-15 minute highly focussed feedback session. • Not all aspects of clinical encounter covered with every patient. • A recent study found that scores derived from as few as 10 mini-CEX encounters possessed a reliability coefficient exceeding 0.80.

30. Assessors role in Mini-CEX • Totally uninvolved in the encounter and as unobtrusive as possible, unless there are risks to patient safety. • Any issues identified should be followed up after the candidate has completed the encounter with the patient. • All questions on the Mini-CEX form should be completed with both effective and ineffective aspects of performance noted. • The Mini-CEX forms should be returned to the WBA administrative officer for data entry and record keeping.

31. Mini-CEXExamples of end of PGY1 characteristics

32. AMC feedback • All boxes/fields must be completed on the assessment forms, including comments in all areas. • Forms must be signed by candidate and assessor • When a task or requirement is not observed the assessor must provide a reason. • DOPS forms must indicate what is being assessed (i.e., procedure or information) and must include comments from the assessor. • The quality of assessments must be readable. • Suggested minimum observation and feedback times of 10-15 minutes for each.

33. Resources WBA Online http://wbaonline.amc.org.au/ The assessment of clinical skills/competence/performance http://winbev.pbworks.com/f/Assessment.pdf WA Health international medical graduate website http://ww2.health.wa.gov.au/Careers/International-applicants/International-medical-graduates-IMGs