1 / 29

Item Writing for CMAS Item Bank

Item Writing for CMAS Item Bank. April 23, 2004 American Association of Medical Audit Specialists 10 th National Conference Nashville, TN Marilyn Balcita RN CMAS CMAS Council. Do you remember your CMAS Exams?. ?. Four Parts Professional Standards Audit Process and Methodology

nicole
Download Presentation

Item Writing for CMAS Item Bank

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Item Writing for CMAS Item Bank April 23, 2004 American Association of Medical Audit Specialists 10th National Conference Nashville, TN Marilyn Balcita RN CMAS CMAS Council

  2. Do you remember your CMAS Exams? ? • Four Parts • Professional Standards • Audit Process and Methodology • Audit Skill • Medical Audit Environment • 200 mostly multiple choice questions • Few matching type questions • Collection of questions from members

  3. Before and After • Not “officially” linked to Job Practice Analysis Studies (Role delineation study) • Item Writing principles not formally followed • Need more contributions from Subject Matter Experts (SME) representing the broad spectrum of medical audit • For future CMAS Item Banking • To fulfill NOCA requirements

  4. How are questions created ????Benjamin Bloom's Taxonomy of the Cognitive Domain (1956)Classification of Educational Goals and Objectives Synthesis Evaluation Analysis Application Comprehension Knowledge *Adapted from: Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green.

  5. Taxonomy for Learning, Teaching, and Assessing*A Revision of Bloom's Taxonomy of the Cognitive Domain Remember Understand Apply Analyze Evaluate Create *Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Addison Wesley Longman.

  6. Remember Keywords: list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote, name, who, when, where Understand Keywords: summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend Bloom's Taxonomy of the Cognitive Domain

  7. Apply Keywords:apply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover Analyze Keywords:analyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer Bloom's Taxonomy of the Cognitive Domain

  8. Evaluate Bloom's Taxonomy of the Cognitive Domain • Create • Keywords: • appraise, compare, • conclude, contrast, • criticize, critique, • defend, describe, • discriminate, evaluate, • explain, interpret, • justify, relate, • summarize, support Keywords: categorize, combine, compile, compose, create, devise, design, explain, generate, modify, organize, plan, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write

  9. What do we need for the CMAS Exams??? Recall Have Enough.. Remember Understand Apply Analyze Evaluate Create More More More …

  10. CMAS Item Development Process Refine content Define Medical Audit Functions (JTA) Item Ready for Testing Randomly select items for testing Develop Med. Audit Test Blueprint Develop, collect and validate items Item Bank storing

  11. The Basis of the CMAS Exams • Job Practice Analysis (Role Delineation Study) Constructed from ….. • Job descriptions • AAMAS membership information – bulletins, informal communication, networking • Onsite and telephonic interviews • SMEs – Medical Auditors, Employers, Consultants, Academicians • Healthcare delivery reports • Official books and publications, journals, and other references

  12. Job Task Analysis Survey • Initial JTA instrument design developed and tested • Beta testing by AAMAS and non AAMAS volunteers • Focused on CMAS respondents • One state chapter volunteered • Tabulated and analyzed • Analysis used for the first time in the 2004 CMAS Examination • Demographic information included to gauge AAMAS general membership make-up

  13. Job Task Analysis Survey (Role Delineation Study) • Measures • “Criticality: how important or critical it is" to the overall performance of your job • Frequency: "its frequency" to the overall performance of your job • Ideally, should be measured every five years

  14. Job Testing Analysis SurveyInitial Result

  15. Investigate and verify charges against medical record documentation Audit Process, Work Flow, and Audit Findings Other Relevant Medical Audit Responsibilities Quality Improvement Activities, Education and Training Compliance and Special Investigations Contracts and Negotiations Denial and Appeals Management Health Information Management (Medical Records) Health Informatics and Technology Medical Audit Process and Methodology (40%) See Detailed List

  16. HIPAA Medicare/Medicaid Policies National Audit Guidelines OIG Compliance Guidance/Work Plan JCAHO Medicare Integrity Program GAAP/GAAS Principles CIA-AICPA Audit Rules NCQA/Hedis DHHS Audit Process US Sentencing Rules Health Insurance reimbursement methodology Medical Audit Environment (30%) Application of laws, guidance, standards, guidelines and other accrediting bodies

  17. Audit Skill (20%) • Interaction and Communication • Leadership and Managerial • Medical Audit Specific Knowledge and Skill Set Required • Clinical Judgment and Nursing Process • Accounting/Finance • Problem Solving • Statistics • Negotiation • Quantitative Analysis • Research • Project Management

  18. Professional Standards And Audit Behavior (10%) • Principles of objectivity • Principles of independence • Internal control • Goal setting, strategic planning, and mission/vision development • Code or standards of conduct • Conflict of Interest • Patient access and confidentiality • Enforcement, expectations and systems of accountability

  19. Why Multiple Choice?? • PROS • Quick and easy to score • Measures all levels of candidates’ ability • Allows objective scoring • Can be analyzed for effectiveness • Covers a broad range of knowledge in limited time • CONS • Challenge in measuring individual differences in skill • Hard to develop good items • Tendency to measure simple recall

  20. S T E M I T E M O P T I O N S National Hospital Billing Audit Guidelines require that: • Auditors must be certified • Hospitals must charge audit fees in excess of $275 • Auditors must conduct themselves in a professional manner, adhere to ethical standards, confidentiality requirements, and objectivity • All of the above What is the correct answer? What are the distractors? Is this a good item ? What’s wrong with it? Reference: National Health Care Billing Audit Guidelines. AAMAS 2003 Directory

  21. CMAS Item Writing Guiding Principles Ask yourself: Can a medical auditor with at least 2 years of basic medical audit experience answer this question correctly? Measure knowledge, skill, abilities (KSA) that is generally known in the medical audit field = JTA Present only ONE central idea, and ONE correct answer Provide 4 options to support the stem Avoid constructing items for the “street-smart” test takers Check for grammars all the time No questions with connotations of race, creed, religion, or ethnic/national background

  22. CMAS Item Writing Guiding PrinciplesThe Stem • Present stem in a complete sentence, even if it is in a question. Bad Example: A medical audit specialist is … • Ideally – no more than 50 words • Try to present stems positively. Avoid using: All (or which) of the following except, which of the following is NOT or Never • Avoid trick questions, or measure trivial things • Good item writers begin with: Who, What, Where, How, and When • “Which” should be followed by a noun • “What” should be followed by a verb • Use “should” instead of “would/could” • Avoid using always, all, and never • No double negatives – negative stem, and negative options

  23. CMAS Item Writing Guiding PrinciplesThe Options • No! No! --- All of the above, None of the above, No slang, No silly options • Correct answer must be 100% correct • Distracters MUST be the wrong answer • If numeric, arrange in logical sequence • Equally balance the options – long/short, or logical choice of options. Ex: 3 ICD codes, one CPT ---- don’t make it obvious … • Do not overuse Partial K type items – answers with (A) 1 and 2; (B) 3 and 4 etc.

  24. CMAS Item Writing Guiding PrinciplesReferences • Include author, title/edition, publisher, publication, date, (month for journals), and page numbers • Official, widely known and readily available references only • References must clearly support the correct answer • For practical type questions i.e., mathematical calculation - include the steps in calculation • DO NOT copy the exact wording from the reference source. Exception: Legal quotes, federal regs, contracts • Use official ONLINE reference sources ONLY – CMS, DHHS, OIG, other government sites. Cite the website in html if possible Examples: http://www.gao.govaud/d02870g.pdf/ http://www.cms.hhs.gov/manuals/10_hospital/ho200.asp#_1_16 Baker, Judith, (2000) Health Care Finance: Basic Tools for Nonfinancial Manager. Aspen Publishers Inc. pg 165-168 Abbey, Duane. “Designing and Maintaining an Effective Chargemaster." Healthcare Financial Management, (2001 March): 50-55.

  25. CMAS Subject Matter Expert Technical Advisory Group Team SME-TAG • Like that name???? • Who are we … • What is it for me ?? • Ability to share your knowledge • Prestige as an item developer • Earn points applicable for recertification and membership fees, discounts to AAMAS marketplace, etc. • Great learning experience • What do I need to do?? • ……………………. Care to join?????? The TAG Team

  26. CMAS SME TAG Team • CMAS certified fellows only • Well rounded experience in medical audit • Experience in test item development processes and review • Talent in writing and editing • Sound knowledge of psychometric analysis, validation, and statistics • Agree to sign CMAS certification security agreement • Committed and willing to learn • Have the time ….. • Have the “tools” to work from – PC at a secured place, appropriate software, compatible system C’mon down!!

  27. Now let’s take a look at this item one more time • National Hospital Billing Audit Guidelines • require that: • Auditors must be certified • Hospitals must charge audit fees in excess of $275 • Auditors must conduct themselves in a professional manner, adhere to ethical standards, confidentiality requirements, and objectivity • All of the above

  28. Ready to have fun???Let’s play …. • Find your group --- introduce yourself • Choose a team lead, recorder, reporter, and timekeeper • Choose one item to develop (15 minutes) • Identify what is being measured: Application vs. Analysis • Match with Test Blueprint (measure med. audit function) • Follow checklist, “guiding principles” • Validate with official resource • Each group: Be ready to present (5 min. ) and critique (5 min.) • Good Luck!!!

  29. Thanks for coming !!! Hope to hear from you soon !!! Mbalcitacmas@aol.com 240 – 686 -0180 (East Coast time)

More Related