Developments in the use of OSCEs to assess clinical competence

1. Developments in the use of OSCEs to assess clinical competence The Third International Conference on Medical Education in the Sudan Katharine Boursicot, BSc, MBBS, MRCOG, MAHPE Reader in Medical Education Deputy Head of the Centre for Medical and Healthcare Education St George’s, University of London

2. Introduction of OSCEs Poor reliability: too small sample (1 or 2 cases); selection of cases random; examiner variation, unstructured marking • Why was the OSCE developed? • Dissatisfaction with traditional forms of assessing clinical competence • Long cases • Short cases • Vivas • Reliability and validity issues • Burgeoning field of medical education and psychometrics Poor reliability : too much variation in cases selected; same examiners followed candidate for all cases, unstructured marking Poor validity: tests knowledge, not clinical skills. Poor reliability: random content; examiner variation, unstructured marking

3. Characteristics of assessment instruments • Utility = • Reliability • Validity • Educational impact • Acceptability • Feasibility Reference Van derVleuten, C. The assessment of professional competence: developments,research and practical implications Advances in Health Science Education 1996, Vol 1: 41-67

4. Test characteristics • Utility = • Reliability • Validity • Educational impact • Acceptability • Feasibility • Reliability of a test/ measure • reproducibility of scores • capability of differentiating consistently between good and poor students

5. Reliability • Competencies are highlydomain-specific • Broad sampling is required to obtain adequate reliability • Need to sample candidate’s ability over several tasks • Need to have multiple observers

6. Blueprinting • Enables test to map to the curriculum outcomes • Allows sufficient sampling and specification of content (i.e. not random cases)

7. Traditional ‘bluepriting’

8. Modern OSCE blueprint

9. Number of assessors/observers per station • Is one good enough? • Wouldn’t two be better than one? • Why not have the assessor follow the candidate around the stations?

10. Reliability of scores on an OSCE 2 new examiners for each case 1 new examiner for each case Reliability Same examiner for each case Hours of Testing Time

12. Behaviour~ skills/attitudes Does Knowshow Cognition~ knowledge Knows Showshow Model of competence Professional authenticity Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S67.

13. Performance assessment in vivo: mini-CEX, DOPS, Video Does Performance assessment in vitro: OSCE, OSLER Knows how (Clinical) Context based tests: SBA, EMQ, SAQ Knows Factual tests: SBA -MCQ Shows how Testing formats Does Shows how Knows how Knows

14. Utility = • Reliability • Validity • Educational impact • Acceptability • Feasibility Test characteristics: Educational impact Relationship between assessment and learning Curriculum Assessment Teacher Student OSCE: encourages students to learn to perform the required skills

15. Advantages of using OSCEs in clinical assessment • Careful specification of content = Validity • Observation of wider sample of activities =Reliability • Structured interaction between examiner and student =Reliability • Structured marking schedule =Reliability • Each student has to perform the same tasks =Acceptability(fairer test of candidate’s clinical abilities)

16. Disadvantages of using OSCEs in clinical assessment – practical issues • Cost • Feasibility issues • Major organisational task • Requires many examiners to be present • Requires suitable facilities

17. Disadvantages of using OSCEs in clinical assessment – academic issues • In-vitro situation: testing competence, not testing actual performance in vivo with patients • Deconstruction of doctor-patient consultation into component parts – not holistic

18. Summary of reasons for use of OSCEs • OSCEs test clinical and communication skills - clinical competence • Reliable • Fair test for all candidates • Organisational and resource implications • Should be used as part of a programme of assessment

19. OSCEs • OSCEs have become very widespread • Undergraduate • Postgraduate • Medicine • Other clinical professions

20. OSCE circuit

21. Clinical examination skills

22. Clinical examination skills

23. Communication skills

24. Suturing

25. IV cannulation

26. Ophthalmoscopy

27. Dentistry: History taking

28. Drilling

29. Laying up sterile equipment

30. Fitting a matrix

31. Veterinary medicine: setting up for X-Ray

32. Intubation of dog

33. Examination of foreleg of horse

34. Administration of injection

35. Trimming of sheep hoof

36. Sampling of milk/mastitis in cow

37. Infection control

38. Current issues • Problematic OSCEs • Authenticity • Standard setting • Compensation • SP recruitment, scripts and training • Examiner training

39. 1. Problematic OSCEs Suboptimal practices which undermine the usefulness of using the OSCE format • Too few stations (less than 12) – poor reliability • Poorly blueprinted OSCE – insufficient spread of skills being tested - poor reliability and validity • MCQs/ SAQs/vivas disguised as OSCEs – poor validity • Poorly constructed stations (lack of congruence) - poor validity and reliability • Inconsistency in conduct of station (site, examiner, SP, equipment, timing) - poor reliability

40. 2. Authenticity • Why should all stations last 5 minutes? • Not authentic • Some tasks require more time • Too little time = truncated ‘samples’ of clinical skills = NOT holistic • Match clinical situations as closely as possible • 5, 10, 15, 20 minutes

41. 3. Standard setting • Process by which decisions are made about pass/fail • Wide variety of processes • Arbitrary • Overall judgement of examiner at each station • Angoff for each station • Complicated schemes involving grades • Gold standard method for OSCEs.......

42. Performance-based standard setting methods • These are the GOLD Standard for OSCEs • Borderline group method • Contrasting group method • Regression based standard method

43. 1 2 3 4 5 Regression based standard Checklist X= passing score 1. Hs shjs sjnhss sjhs sjs sj 2. Ksks sksmsiqopql qlqmq q q qkl 3. Lalka kdm ddkk dlkl dlld 4. Keyw dd e r rrmt tmk 5. Jfjfk dd 6. Hskl;s skj sls ska ak akl ald 7. Hdhhddh shs ahhakk as TOTAL  Checklist Score  X    Overall rating 1 2 3 4 5 Clear Borderline Clear Excellent Outstanding fail pass 1 = Clear fail 2 = Borderline 3 = Clear pass 4 = Excellent 5 = Outstanding

44. Performance-based standard setting • Advantages • Utilises the expertise of the examiners • they are observing the performance of the students • they are in a position to make a (global) judgement about the performance • based on • their clinical expertise • expected standards for the level of the test • knowledge of the curriculum/teaching

45. Performance-based standard setting • Advantages • Large number of examiners set a collective standard while observing the candidates – not just an academic exercise • Reliable: cut-off score based on large sample of judgments • Credible and defensible: based on expert judgment in direct observation

46. Performance-based standard setting • Disadvantages • Requires large (ish) cohort of candidates to achieve enough numbers in the ‘borderline’ group • Passing score not known in advance • Judgments may not be independent of checklist scoring • Requires expert processing of marks immediately after the exam • Checking of results • Delay in producing results

47. 4. Compensation • A number of students achieve a totalscoreabove the overall pass markBUT • faila significant number of stations • Poor performance on a number of stations is compensated by good performance on other stations

48. Concerns • Students do not have an acceptable minimum standard across a range of skills • Students could proceed to graduation without having acceptable clinical examination or history taking skills

50. Criterion 2: Must pass ≥10/19 stations Two extra students failed Criterion 1: Pass mark 60.3 (BGM)