1 / 31

Constructing a test

Constructing a test. Aims. To consider issues of: Writing assessments Blueprinting. Methods of assessment MCQs Multiple true/false Best of five Extended matching questions Short answer questions Modified essay questions Objective Structured Examination

mohawk
Download Presentation

Constructing a test

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Constructing a test

  2. Aims • To consider issues of: • Writing assessments • Blueprinting

  3. Methods of assessment MCQs Multiple true/false Best of five Extended matching questions Short answer questions Modified essay questions Objective Structured Examination Mini-sampling examinations Prolonged observation Summative at end of year Multiple throughout years Oral examination Non-structured vs. structured Assessment of professional behavioursUtility?

  4. The professional progress Knows Knows how Shows

  5. The professional progress Knows Knows how Shows Knowledge Can state how Demonstrates information is used competence MCQs MEQs MEQs EMQs EMQs EMQs Multi-station exam Multi-station exam OSCE Work–based test Multi-source data Professionalisms

  6. DANGER Subjects

  7. DANGER Subjects Many curricula Assessments

  8. DANGER Subjects Subjects Many curricula Assessments Learning

  9. DANGER Subjects Subjects Many curricula Assessments Learning Disconnected teaching leads to disconnected assessments Leads to disconnected learning

  10. Future directions in assessment Testing competencies by ability to handle complex tasks van der Vleuten and Schurworth 2005 Assessment in context of ‘practice’ – work-place based assessments Rethans et al, 2002 Multiple sampling by multiple assessors Petrusa, 2002 Peer review Do OSCEs do all that you need? Professional behaviours assessed longitudinally and at many times

  11. Methods of assessment MCQs Multiple true/false Best of five Extended matching questions Short answer questions Modified essay questions Objective Structured Examination Mini-sampling examinations Prolonged observation Summative at end of year Multiple throughout years Oral examination Non-structured vs. structured Assessment of professional behavioursUtility?

  12. Methods of assessment in use MCQs Multiple true/false Best of five Extended matching questions Short answer questions ‘Key feature case’ Modified essay questions OSCE Mini Clinical Examination (Mini-CEX) Long case Summative at end of year Multiple throughout years Oral examination Non-structured vs. structured Assessment of professional behaviours Multiple throughout course

  13. Methods of assessment in use in Sheffield • Extended matching questions • Modified essay questions • OSCE • Mini Clinical Examination (Mini-CEX) • Observed long cases - multiple throughout years • Assessment of professional behaviours -multiple throughout course • Work-place based assessments (from 2009)

  14. Methods of assessment in use in Sheffield • Extended matching questions • Modified essay questions • OSCE • Mini Clinical Examination (Mini-CEX) • Observed long cases - multiple throughout years • Assessment of professional behaviours -multiple throughout course • Work-place based assessments (from 2009) All need: Resources Staff who are trained and able Exams prepared months in advance Evidence that the exams are good Student understanding Good statistics support

  15. Validity: linking to the curriculum by blueprinting • Blueprinting • aims to ensure that the defined curriculum (and only that) is assessed • ensures that a reliable sample of the curriculum is selected for testing • enables the most appropriate test methods to be used for different aspects to be tested

  16. Assessment blueprinting Subjects/ modules taught Core problems

  17. Assessment blueprinting Subjects/ modules taught Core problems Components to be tested

  18. Assessment blueprinting Subjects/ modules taught Core problems Exam components and weighting Components to be tested

  19. Assessment blueprinting Subjects/ modules taught Core problems Exam components and weighting Components to be tested • Questions • Balanced sampling between subjects, • teaching components and exam • components • Marking appropriately weighted • Increases validity • Guides students and staff

  20. Validity: linking to the curriculum by blueprinting Core clinical problems etc RS GI ENDO CVS

  21. Validity: linking to the curriculum by blueprinting Core clinical problems etc RS GI ENDO CVS History Examination Case Management Communication skills Data interpretation

  22. FeasibilityCan the test be delivered? • Time management • Blueprinting / writing questions in time • External validation of the test • Educational • Can what you want to test be tested in the format proposed? • Practical • Sufficient space; examiners, patients

  23. Test securityKeeping it confidential • Who sees the examination paper? • Reuse of previous papers/questions

  24. CostsHuman & financial • Human resources • Academic and administration staff time • Simulated patients/actors • Financial costs • OSCEs • Manikins • consumables

  25. Information to candidates • Ensure candidates know the purpose of the assessment • Information about the test • Results and (remediation/resit)

  26. Feedback • All assessments must lead to feedback • All assessments are formative in part • Without feedback, the educational loop is left open

  27. Characteristics of good assessments Valid - test what they purport to test - link to the curriculum Reliable - reproducible Feasible - can be delivered Impact - preparation of students and feedback to them Leadership - with an overview

  28. Question writing • Link one question to one topic

  29. Question writing • Link one question to one topic • Decide what you wish to test • eg knowledge / reasoning / psychomotor skill / • communication / response to emergencies • Decide best format of question

  30. Question writing • Link one question to one topic • Decide what you wish to test • eg knowledge / reasoning / psychomotor skill / • communication / response to emergencies • Decide best format of question • Write: - accurately • - unambiguously and no negatives – single or double! • - without trying to confuse or mislead • - at the right level • - to produce an exam that links directly to the curriculum • - and show your work to colleagues / assessment group • - with non-specialists • - using the appropriate clinical context

  31. Assessment is enhanced if it is • Authentic • Related directly to practice • Seen by students to be fair, have a utility, • be related to the curriculum and be a source • of feedback • Is more than an exercise in factual recall

More Related