1 / 29

Trainee evaluation method

Trainee evaluation method. Ass. Prof. Dr. Abdul Sattar KHAN. MBBS, MPH, MCPS, MRCGP (UK), FRIPH (UK), FHAE (UK). Family & Community Medicine Department College of Medicine King Faisal University. Objectives. At the end of the session, participants are expected to know:

hyman
Download Presentation

Trainee evaluation method

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Trainee evaluation method Ass. Prof. Dr. Abdul Sattar KHAN MBBS, MPH, MCPS, MRCGP (UK), FRIPH (UK), FHAE (UK) Family & Community Medicine Department College of Medicine King Faisal University

  2. Objectives At the end of the session, participants are expected toknow: • What is an assessment process? • Who should assess the students? • Why assess the students? • What should be assessed? • How should the students be assessed? • When should the students be assessed? • Where should the students be assessed? ASK

  3. A curricular definition Learning - acquiring knowledge, skills, attitudes, and values and experiences. A cognitive definition Learning- a process of formulating new and more complex understandings of the world Learning - revising and refining mental constructs, i.e., the understandings that guide how we think, speak and behave What is Learning? http://dakota.fmpdata.net/PsychAI/PrintFiles/DefLrng.pdf ASK

  4. How do we know that learning is done? ASK

  5. Key Relationship in Learning Process Teaching Learning Assessment ASK

  6. What is an assessment process? The Oxford Dictionary1 defines assessment as ‘the action of assessing’. In the context of medical education, assessment could be defined as determining the competence of the product, that is, the Health Professional. It has three pillars2: Cognitive, Affective psychomotor 1. Oxford Dictionary of English, 2nd Edition, Oxford University Press; 2012 2. Newble D. Assessment. In: Jolly B, Rees L. (Editors) Medical education in the millennium, Oxford: Oxford University Press; 1998.p.131–42. ASK

  7. Who should assess the students? • Medical schools • International accrediting body • National accrediting body • Professional bodies • Licensure bodies • Department • Individual teacher • patients / community • Students themselves ASK

  8. Why assess the students? • “fit for purpose” • Ranking of student • Measurement of improvement in a student • To diagnose student difficulties • Evaluation of teaching method • Motivating student to study • Provision of feedback for the teacher ASK

  9. CanMedsroles Medical expert Communicator Collaborator Manager Health advocate Scholar Professional What should be assessed? Competencies / Outcomes • ACGME (Accreditation council for Graduate Medical Education) competencies • Medical knowledge • Patient care • Practice-based learning & improvement • Interpersonal and communication skills • Professionalism • Systems-based practice www.royalcollege.ca/common/documents/canmeds/ http://www.acgme.org/acgmeweb/ ASK

  10. Miller’s pyramid of competence Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. ASK

  11. How should the students be assessed? • Essay Type Questions • Long & Short • Modified Essay • Objective type Questions • Multiple Choice Question MCQ • Supply-item • True-False item • Extending matching item ASK

  12. Knows Assessing know Does Shows how Knows how Essay type, MEQ, MCQ etc Knows ASK

  13. How should the students be assessed? Relative characteristics of different types of examination ASK University of Dundee, 2004

  14. How should the students be assessed? • Assessment of skills • Patient Management Problem PMP. • Observational Assessment, OSCE, OSPE. MiniCEX, DOPS etc. A new approach: • Response Format (Open-ended & MCQ) • Stimulus Format (Context-free & Context-rich – EMQ & Key-feature) • Hybrid Format (Script Concordance Test- SCT) ASK

  15. Knows how Knows Assessing knowing how Does Shows how Written complex simulations (PMPs) Knows how Knows ASK

  16. Knows how Shows how Assessing showing how Does Performance assessment in vitro (OSCE) Shows how Knows how Knows ASK

  17. Does Shows how Assessing does Performance assessment in vivo by judging work samples (Mini-CEX, CBD, MSF, DOPS, Portfolio) Does Shows how Knows how Knows ASK

  18. Tomorrow’s Doctors Assessments will be fit for purpose – that is: • Valid • Reliable • Generalizable • Feasible • Fair http://www.gmc-uk.org/education/undergraduate/tomorrows_doctors.asp ASK

  19. anatomy physiology int medicine item analyses student comments surgery psychology review committee test administration item pool Info to users item bank Post-test review Pre-test review How be sure the process is appropriate? Maastricht Review Process ASK

  20. Reliabilities across methods Mini CEX6 0.73 0.84 0.92 0.96 Case- Based Short Essay2 0.68 0.73 0.84 0.82 Practice Video Assess- ment7 0.62 0.76 0.93 0.93 In- cognito SPs8 0.61 0.76 0.92 0.93 Testing Time in Hours 1 2 4 8 MCQ1 0.62 0.76 0.93 0.93 PMP1 0.36 0.53 0.69 0.82 Oral Exam3 0.50 0.69 0.82 0.90 Long Case4 0.60 0.75 0.86 0.90 OSCE5 0.47 0.64 0.78 0.88 1Norcini et al., 1985 2Stalenhoef-Halling et al., 1990 3Swanson, 1987 4Wass et al., 2001 5Petrusa, 2002 6Norcini et al., 1999 7Ram et al., 1999 8Gorter, 2002 ASK Cees van der Vleuten, 2010

  21. Factors influencing reliability • Environmental errors • Processing errors • Generalization errors • Bias errors a. Weighting b. Rater prejudice c. Halo effect d. Leniency and stringency ASK

  22. Factors influencing validity • Unclear instructions • Complicated vocabulary • Too easy/difficult items • Unintentional clues • Not matching with outcome • Inadequate time • Too few items • Unsuitable item arrangement • Detectable answer pattern ASK

  23. “Domain independent” skills Measuring the unmeasurable Assessment (mostly in vivo) heavily relying on expert judgment and qualitative information Does Shows how Knows how Knows “Domain specific” skills ASK Cees van der Vleuten, 2010

  24. Self assessment Peer assessment Co-assessment (combined self, peer, teacher assessment) Multisource feedback Log book/diary Learning process simulations/evaluations Product-evaluations Portfolio assessment Measuring the unmeasurable ASK Cees van der Vleuten, 2010

  25. When should the students be assessed? • At the beginning of the course • During the course (Formative - Diagnostic) • At the end of the course (Summative - Certifying) ASK

  26. Where should the students be assessed? • Examination hall • Class room • Hospital wards • Out patients department (OPD) • Work Places • Community ASK

  27. Testing motivates by threating If I threaten you will fail and then you will try harder Maximize anxiety to maximize learning Testing helps teachers make important instructional decisions Students are not assessment users The Myths of testing (Stiggins, 2004) ASK

  28. Important assessment decisions can be made once a year Investment of time, effort, and money into large-scale testing supports this belief Learning how to assess is not as important as learning how to teach Teachers teach and testing professionals test The Myths of testing (Stiggins, 2004) ASK

  29. Take Home Message When assessing students it is important to familiar with maximum methods. So, use as many as possible… ASK

More Related