Overview of Assessment ESME 2006

1. Overview of AssessmentESME 2006 Stewart Mennin, PhD

2. Objectives • Make more informed decisions about assessment • Apply information to your teaching • Describe the relationship between assessment & learning • Discuss criteria for effective assessment • Define & apply basic terminology ESME

3. Objectives • Describe selection, supply & performance methods • Link Miller’s pyramid to assessment methods • Match assessment methods to your outcomes • Draft assessment plan (blueprint)for your teaching ESME

4. GOALS OF ASSESSMENT • Enhance student learning • Guide faculty teaching efforts • Inform curricular development & revision • Fulfill responsibilities to students, institution & society • Contribute to research ESME

5. FEATURES OF ASSESSMENT • Assessment is more a curriculum design problem than a measurement problem • Part of learning & can not be separate from it • Assessment is about the present, past and future of learning • Assessment in not a collection of instruments • Assessment should be authentic, grounded in what is real ESME

6. EIGHT CRITERIA: EFFECTIVE ASSESSMENT • Articulation of a clear conception of intended learning outcomes • Use variety of assessment procedures/methods • Close agreement between intended outcomes of instruction, learning tasks, & the assessment procedures/methods • Adequate sampling of student performance Gronlund, 2000 ESME

7. EIGHT CRITERIA: EFFECTIVE ASSESSMENT • Procedures that are equitable for all participants • Explicit, specific criteria for use in judging successful performance • Timely feedback that emphasizes strengths of performance & focuses attention on specific areas in need of improvement • A grading & reporting system consistent with the rest of the learning & assessment process Gronlund, 2000 ESME

8. Key Terminology • Formative – Summative • Reliability & Validity • Criterion Referenced --- Norm referenced Progress test • Sequential test ESME

9. Purpose Timing Evaluator Use Assessment As Learning Formative Summative Improvement Decisions / Judgment Frequent & throughout Milestones & end Teacher & learner Teacher Feedback to student & learner Find misconceptions Identify strengths & weaknesses Change future teaching Develop learning plan Assign grades Judge competence Certify mastery Promotion ESME

10. Reliability • Consistency of results • Reliability is matter of sampling • Across content ESME

11. Test Sample Test Sample Domain of Interest   ESME

12. Competent Assessment of Clinical Competence is a Issue of Specificity ESME

13. Reliability • Practical suggestions: • Do not rely on short tests • Sample broadly (content, in time, examiners, patients) • Consider efficiency in: • selection of test format • construction test items ESME

14. Reliability • Adequate reliabilityrequires substantial sampling (therefore: resources, testing time, examiners, patients, etc.) • Be efficient ESME

15. Professional authenticity Validity Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. ESME

16. Professional authenticity Validity Performance or hands on assessment Does Shows how Written, Oral or Computer based assessment Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. ESME

17. Valid Inferences • Validity is a single concept with different forms of evidence • Content-related evidence • Criterion-related evidence • Construct-related evidence ESME

18. ASSESSMENT METHODS • All assessment methods have limitations and no one method can assess all types of knowledge & skill • Multiple methods are recommended ESME

19. Some Assessment Options: • Selection Item • Multiple Choice Question • One-Best Answer Questions • Extended-Matching Questions • Supply Items • Free Response Questions • Short Answer Questions • Essay Questions • Performance Examinations • Laboratory Practical Examinations (Identification and Concepts) • Clinical Skills (e.g., OSCE) • Other ESME

20. Does Knows how Knows Shows how Selection Methods • MCQ • Matching • Extended Matching • True False • Multiple True False ESME

21. Does Knows/Knows how Shows how Knows how Knows Knows • The stimulus format is more important than the response format ESME

22. How Well Written? • Stem • Poor • Good ESME

23. How Well Written? Stem a. b. c. d. a. Poor b. Good ESME

24. How Well Written? 30-year-old man has loss of pain and temperature sensation from the neck down on the right side of the body and on the left side of the face; partial paralysis of the soft palate, larynx, and pharynx on the left; and ataxia on the left. This syndrome is most likely to result from thrombosis of which of the following arteries? A Basilar B Right posterior inferior cerebellar C Left posterior inferior cerebellar D Right superior cerebellar E Left superior cerebellar ESME

25. Extended Matching For each quotation, select the most likely author. 1. I never teach my pupils. I only attempt to provide the conditions under which they can learn. 2. All genuine knowledge originates in direct experience. 3. To know one’s ignorance is the best part of knowledge. ESME

26. Extended Matching Answers Authors on Education: a. D. J. Boorsten e. Lao-Tzu b. Camus f. S. Scott Obenshain c. Albert Einstein g. John Pieper d. Thomas Harris h. Mao Tse-Tung 1. I never teach my pupils. I only attempt to provide the conditions under which they can learn. (c) 2. All genuine knowledge originates in direct experience. (h) 3. To know one’s ignorance is the best part of knowledge. (e) ESME

27. Does Knows how Knows Shows how Supply Methods • MEQ • Essay (long & short) • Practical • Short Answer • Written simulations ESME

28. Does Knows how Knows Shows how Performance Assessment • Procedures – skills lab • OSCE (with skills stations) • Mini CEX • Long Case • Logs • Portfolio • Clinical Notes • Observation • Other ESME

29. Interviewing & Communication Skills Communication Skills ESME

30. Essential Elements of Communication Skills • Open the Discussion • Build a Relationship • Gather Information • Understand the Patient’s Perspective • Share Information • Reach Agreement • Provide Closure Acad Med 2001;76(4):390-393 ESME

31. Clinical Problem

32. Testing time Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7 Test using key features Key Feature Approach Case 2 Case 1 Test using integral cases

33. Does Shows How: OSCE Shows how Knows how Knows • Make stations as clinically authentic as possible • Global judgments do well in OSCEs • Content specificity is the problem ESME

34. Checklists Dichotomous scores (yes/no) Relatively objective Record rather than interpret Station specific Rating Scales Ordinal level measures (poor, good, excellent) Relatively subjective Station independent Checklist/Global Rating Scale Comparison ESME

35. Does Shows how Knows how Knows Does: Practice Assessment • Indirect methods • Chart audit • Practice audits • Prescription data……… • Direct methods • Clinical (supervisor) ratings • Undercover standardized patients • Video assessment • Patient reports • Peer reports • Clinical work samples • Mini-CEX……… ESME

36. Does Shows how Knows how Knows Clinical Work Samples • Repeated direct observations of clinical performance in practice using (generic) evaluation forms, completed by any significant observer (clinician, nurse, peer…..) • Illustration ESME

37. Does Shows how Knows how Knows Mini Clinical Examination(Norcini, 1995) • Short observation during clinical patient contact (10-20 minutes) • Oral evaluation • Generic evaluation forms completed • Repeated at least 4 times by different examiners • (cf. http://www.abim.org/minicex/) ESME

38. Learning how to learn Self-assessment Humanization Leadership Team skills Skills of expression Professional behavior Reflection Permanent education NEW SKILLS EMPHASIZED ESME

39. Methods & Knowledge ESME

40. Task • Use the table from effective teaching and match assessment methods to your outcome competencies ESME

41. Faculty of Medicine Maastricht University Progress TestScores on one progress test ESME

42. Faculty of Medicine Maastricht University Progress TestMaastricht Test Results ‘77 - ‘88 Mean correct scores on the Progress Test from a number of Maastricht student cohorts (black lines) and reference group of family physicians in training (blue line) ESME

43. Planning: Blueprinting • Competencies • Time • Teaching methods • In class -- out of class ESME

44. Methods Supply Perform-1 Competent Perform-2 Capable Select Clinical Skills Critical Int Know Comm Human Self Assess Profess Competencies

45. Conclusions • Assessment is less of a psychometric problem, but rather an educational design problem(i.e. how to use assessment strategically for its educational effects) • Assessment requires careful planning and monitoring • To effective with the entire competence pyramid • a mix of methods that depends on the context; not a single method is best or bad Cees van der Vleuten ESME

46. Acknowledgements • Cees van der Vleuten - Maastrict • Deana Richter – New Mexico • Teresita McCarty - New Mexico • Nancy Sinclair - New Mexico • Regina Petroni Mennin – UNIFESP ESME

47. Presentations Available at: www.menninconsulting.com stewart@menninconsulting.com smennin@gmail.com ESME