1 / 30

Question Banking – the UMAP experience

Question Banking – the UMAP experience. Emyr Wyn Benbow University of Manchester. So what was UMAP?. UMAP consortium five original partners (2003) Manchester Leeds Liverpool Newcastle Sheffield. So what is UMAP now?. UMAP consortium now Birmingham Cork Hull York Keele

nami
Download Presentation

Question Banking – the UMAP experience

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Question Banking – the UMAP experience Emyr Wyn Benbow University of Manchester

  2. So what was UMAP? UMAP consortium • five original partners (2003) • Manchester • Leeds • Liverpool • Newcastle • Sheffield

  3. So what is UMAP now? UMAP consortium now Birmingham Cork Hull York Keele Leeds Leicester Liverpool Manchester Newcastle Peninsula Sheffield Southampton UEA Warwick

  4. International Database for Enhanced Assessments & Learning The competition The I D E A L Consortium An International Collaboration of Medical Schools

  5. The competition • Australia (Sydney, Flinders, ANU, Monash, Queensland, Newcastle, Tasmania, Adelaide) • Canada (U Alberta, U British Columbia) • China (Hong Kong U, Chinese U Hong Kong) • Indonesia (U Padjadjaran) • Islamic Republic of Pakistan (Shifa College of Medicine, U Health Science) • Kingdom of Bahrain (Arabian Gulf U) • Kingdom of Saudia Arabia (King Saud Bin Abdulaziz U Health Sc) • Kuwait (Kuwait U) • Republic of South Africa (U Witwatersrand) • Sultanate of Oman (Sult.Qaboos U) • United Arab Emirates (United Arab Emirates U, U of Sharjah) • United Kingdom (Pennisula Medical Sch)

  6. The competition • summative assessment bank > 13,000 items • formative assessment bank > 6000 items • six types of question • best one of “n” choices (A) • extended matching (R) • multiple Response (Pick n) • short Answer (SAQ) • modified Essay (MEQ) • OSCE/OSPE

  7. The competition

  8. Review Workshops Question Bank Question Use Results Return Question Release UMAP Admin UMAP Admin UMAP process

  9. UMAP funding • originally HEFCE grant • Fund for the Development of Teaching and Learning Phase 4 (FDTL-4) • self-funding from January 2006 • each partner school pays an equitable proportion of costs (£7,500 in 2996-7)

  10. UMAP workshops • two workshops at each partner each year • each led by UMAP’s manager • one for MCQs, one for EMQs • use best practice • authors use question templates • authors review each others questions • clinical and academic participation, all levels

  11. UMAP workshops

  12. Typical UMAP MCQ A 52 year old man presents to his General Practitioner with pains in both hands. The symptoms began as a tingling sensation which have now developed into a pain which keeps him awake at night. The man describes the overall sensation as a feeling of ‘uselessness' in his fingers and he says though there is no swelling they feel swollen. What nerve has been compressed? A.     A   Axillary nerve BMedian nerve C Musculcutaneous nerve D Radial nerve E Ulnar nerve

  13. UMAP quality assurance teams • items are pre-reviewed for adherence to “house style” • question review teams concentrate on content • each partner site runs six sessions a year • specially designed database • QA begins with cover up test • each item to be accepted or rejected • no pressure to accept items

  14. Workshops Responsible: UMAP Manager, Best practice instruction, writing and review UMAP administration Responsible: UMAP Administrator Enter and check questions for common errors etc e.g. abbreviations, spelling, normal ranges Pre reviews Responsible: UMAP Manager Questions checked for overall concordance with UMAP ‘house style’ Review team meetings Responsible: team leads Network of teams working on UMAP reviewer databases Follow up Responsible: UMAP Administrator Reviewer decisions and comments acted on

  15. Quality assurance

  16. Quality assurance

  17. Progress with banking

  18. Progress with banking

  19. UMAP “rules” • all partners must • write items • provide quality assurance • only use items in secure summative assessments

  20. How it works: setting a paper Stage 1 - the planning form Stage 2 - question selection Stage 3 - question receipt Stage 4 - reporting problems Stage 5 - reporting your outcomes

  21. How it works: the planning form

  22. How it works: question selection

  23. How it works: question selection

  24. How it works: question receipt Two documents - Word document with all selected questions - UMAP ID number - theme - focus - question text - feedback

  25. How it works: question receipt Two documents - Excel spreadsheet - to report which questions you’ve chosen

  26. How it works: reporting problems and outcomes • any difficulties found at local question selection and quality control processes need to be reported back • detailed examination results are fed back, and performance data will be generated

  27. How it works: reporting outcomes

  28. The End - but this is just the end of the beginning!

More Related