1 / 46

Designing a Classroom Test

Designing a Classroom Test. Content. Understanding by Design (UbD) Purpose of classroom test Test blueprint & specifications Item writing Assembling the test Item analysis. How do we currently identify what we are going to teach in our classrooms?.

gaille
Download Presentation

Designing a Classroom Test

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Designing a Classroom Test Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  2. Content • Understanding by Design (UbD) • Purpose of classroom test • Test blueprint & specifications • Item writing • Assembling the test • Item analysis Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  3. How do we currently identify what we are going to teach in our classrooms? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  4. PDE’s Standards Aligned System and Understanding by Design (UbD) Framework Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  5. PDE’s Standards Aligned System Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  6. UbD Framework 3 parts: • What do you want students to know? • How are you going to assess them to determine they learned what you wanted them to learn? • How are you going to teach the content to ensure all students learn? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  7. UbD Templates • Sample Unit Template • http://wilkes-ed520.wikispaces.com/file/view/UbDOverview.pdf • Blank Unit Template • http://wilkes-ed520.wikispaces.com/file/view/UBD_Lesson_Template.doc Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  8. So what IS our test measuring? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  9. What is the purpose of a classroom test? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  10. Purpose of Classroom Test • Establish basis for assigning grades • Determine how well each student has achieved course objectives • Diagnose student problems • Identify areas where instruction needs improvement • Motivate students to study • Communicate what material is important Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  11. How do you currently design your tests? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  12. Test Blueprint • Sometimes called the “test specifications”, the goal is to ensure the test covers the content and/or objectives in the proper proportions. Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  13. Test Blueprint • To ensure the test assesses what you want to measure • Ensure the test assesses the level or depth of learning you want to measure • PSSA Test Blueprints: http://www.portal.state.pa.us/portal/server.pt/community/pennsylvania_system_of_school_assessment_(pssa)/8757/resource_materials/507610 Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  14. Setting up a test blueprint • Identify content to be assessed • Identify depth of learning (ex. Bloom) • Identify time spent on topic as compared to time taught in class • Emphasis/number of points for each target Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  15. http://www.teach-nology.com/worksheets/time_savers/bloom/ Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  16. Test Blueprint Example Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  17. Test Specifications Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  18. Let’s Evaluate our own tests Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  19. Preparing & Assembling the Test • Provide general directions • Time allowed (allow enough time to complete test) • How items are scored • How to record answers • How to record name /ID • Arrange items systematically • Provide adequate space for short answer and essay responses • Placement of easier & harder items Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  20. Item Writing – General Guidelines1 • Present a single clearly defined problem that is based on a significant concept rather then trivial or esoteric ideas • Use simple, precise & unambiguous wording • Exclude extraneous or irrelevant information • Eliminate any systematic pattern of answers that may allow guessing correctly Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  21. Item Writing – General Guidelines2 • Avoid cultural, racial, ethnic & sexual bias. • Avoid presupposed knowledge which favors one group over another (“fly ball” favors those that know baseball) • Refrain from providing unnecessary clues to the correct answer. • Avoid negatively phrased items (i.e., except, not) • Arrange answers in alphabetical / numerical order Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  22. Item Writing – General Guidelines3 • Avoid “None of the above” or “All of the above” type answers • Avoid “Both A & B” or “Neither A or B” type answers Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  23. Item Writing – Correct Answer is • Longer • More qualified or more general • Uses familiar phraseology • Is grammatically correct for item stem • Is 1 of the 2 similar statements • Is 1 of the 2 opposite statements Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  24. Item Writing – Wrong Answer is • Usually the first or last option • Contain extreme words (always, never, nonsense, etc.) • Contain unexpected language or technical terms • Contain flippant remarks or completely unreasonable statements Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  25. Item Writing – Grammatical Cues Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  26. Item Writing – Logical Cues Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  27. Item Writing – Absolute Terms Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  28. Item Writing – Word Repeats Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  29. Item Writing – Vague Terms Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  30. Item Writing • Effective test items match the desired depth of learning as directly as possible Applying & Analyzing • Applying: Implementing a procedure or process. • Analyzing: Breaking material into constituent parts, determining how the parts relate to one another and to an overall structure or purpose through differentiating, organizing, and attributing. • ITEM TYPES: MC, Short Answer, Problems, Essay Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  31. Comparison of MC & Essay1 Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  32. Comparison of MC & Essay2 Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  33. Item Writing - Application MC application of knowledge items tend to have long vignettes that require decisions. Case, et al. at the NBME investigated the impact of increasing levels of interpretation, analysis and synthesis required to answer a question on item performance. (Academic Medicine, 1996;71:528-530) Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  34. Item Writing - Application Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  35. Item Writing - Application Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  36. Item Writing - Application Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  37. Interpreting test scores Teachers High scores = good instruction Low scores = poor students Students High scores = smart, well-prepared Low scores = poor teaching, bad test Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  38. Interpreting test scores High scores too easy, only measured simple educational objectives, biased scoring, cheating, unintentional clues to right answers Low scores too hard, tricky questions, content not covered in class, grader bias, insufficient time to complete test Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  39. Item Analysis • Main purpose of item analysis is to improve the test • Analyze items to identify: • Potential mistakes in scoring • Ambiguous/tricky items • Alternatives that do not work well • Problems with time limits Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  40. Reliability • The reliability of a test refers to the extent to which a test is likely to produce consistent results. • Test-Retest • Split-Half • Internal consistency • Reliability coefficients range from 0 (no reliability) to 1 (perfect reliability) Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  41. Internal Consistency Reliability • High reliability means that the questions of the test tended to hang together. Students that answered a given question correctly were more likely to answer other questions correctly. • Low reliability means that the questions tended to be unrelated to each other in terms of who answered them correctly. Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  42. Reliability Coefficient Interpretation General guidelines for homogeneous tests • .80 and above – Very good reliability • .70 to .80 – Good reliability, a few test items may need to be improved • .50 to .70 – Somewhat low, several items will likely need improvement (unless short test 15 or fewer items) • .50 and below – Questionable reliability, test likely needs revision Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  43. Item difficulty1 • Proportion of students that got the item correct (ranges from 0% to 100%) • Helps evaluate if an item is suited to the level of examinee being tested. • Very easy or very hard items cannot adequately discriminate between student performance levels. • Spread of student scores is maximized with items of moderate difficulty. Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  44. Item discrimination1 • How well does the item separate those that know the material from those that do not. Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  45. Evaluation of Distractors • Distractors are designed to fool those that do not know the material. Those that do not know the answer, guess among the choices. • Distractors should be equally popular. (# expected = # answered item wrong / # of distractors) Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

  46. Thank you for your timeQuestions ??? Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

More Related