1 / 24

Developing Tests for Departmental Assessment

Developing Tests for Departmental Assessment. Deborah Moore, Assessment Specialist Institutional Research, Planning, & Effectiveness University of Kentucky. Session Overview. Steps in Test Development and Test Administration Process for Evaluating Test and Items Common Difficulties

kenny
Download Presentation

Developing Tests for Departmental Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Developing Tests for Departmental Assessment Deborah Moore, Assessment Specialist Institutional Research, Planning, & Effectiveness University of Kentucky

  2. Session Overview • Steps in Test Development and Test Administration • Process for Evaluating Test and Items • Common Difficulties • Strategies for Analysis and Reports

  3. Improve Student Learning

  4. Exercise 1 (10 minutes) Under What Conditions Do You Give a Test? • How do you decide you are going to give a test? • What do you do to develop a new test? • How do you decide what types of item formats to use? • How do you develop items? • How long does it take to put the test together?

  5. Using a Test for Classroom Assessment • Generally, instructors develop their own classroom tests, making all decisions about when and how to construct, administer, score, and report results of tests. • Construction is often done without formality or documentation. • The most frequent use of tests by instructors is to assign grades related to individual student learning.

  6. A Test Is a Good Choice When… • the student must demonstrate acquisition of knowledge or ability to process and use knowledge • the student's knowledge about a wide range of content is to be evaluated (e.g., survey and capstone courses) • multiple observations of the content-related knowledge are needed (e.g., math and foreign languages) • more resources are available for constructing the test than for scoring and reporting • a large group is being assessed

  7. Using a Test for Departmental Assessment • Planning, implementing, and using results become a group effort – a shared set of decisions and responsibilities. Consensus is emphasized. • Additional planning time, communication, and record keeping will be needed. • When used for program assessment, test performance is generally used along with other information to describe group achievement and is independent of grading.

  8. Advantages of Selected-Response Item Formats • Better content coverage • Higher reliability • Greater efficiency • Objectivity • Mechanical scoring

  9. Test Development Process • Define purpose/use • Outline curriculum & consult goals/objectives • Create test plan (aka blueprint or specifications) • Create a pool of items • Critique and revise items • Pilot/field test (e.g. item analysis, reliability/validity studies) • Set guidelines for test administration, scoring procedures, interpretation of scores (e.g. develop norm tables and standard setting procedures)

  10. Exercise 2 (10 minutes) Review a Sample Departmental Packet • What role does the exam play in the assessment plan for this department? • What type of item formats should be used on the comprehensive exam?

  11. Objectives 4 1 2 5 3 6

  12. Test Blueprint (Content X Objective)

  13. Test Plan (Items Within Cells)

  14. Administrative Approaches • Common exams • Course-embedded testing • Assessment Center/Assessment Days

  15. Research Curriculum Real life Professional development materials & textbooks Own final exam files Comprehensive exams from other universities Retired test/item banks Sources for Items

  16. Common Difficulties with Test Quality • Push for higher order thinking often weak • Insufficient consensus on test plan/blueprint • Discomfort with various item-writing approaches (writer’s block) • Insufficient editing • Little or no piloting of items

  17. Critiquing Items • Match with test plan • Item value • Item quality

  18. Typical Item Review Questions • Has each item received a quality appraisal? • Has each item’s content been verified? • Has each item been classified? • Has the key been identified? • Have you edited the items? • Have items been checked for bias/insensitivity? • Have you field tested the items?

  19. Planning A Departmental Test • Develop test • Plan administration • Analyses test data • Report and use results • Agree on leadership/support roles

  20. Exercise 3 (30 minutes) Developing a Planning Checklist • Review the headings of the checklist form • List additional headings needed • List activities that need to be completed under the various headings • Add a number to the activities to represent the order of completion

  21. Analyses • Examine quality of your assessment tools and procedures • Describe the test results • By test plan features • By meaningful subgroups • Combine with other datasets • Execute and summarize planned comparisons linked to research design

  22. Reports • Determine stakeholders/constituents • Identify specific interest • Create a timeline and share • Include a vetting procedure

  23. Testing in the Future • Computer based testing • Adaptive (CAT) • Self-Adaptive (SAT) • Computer assisted item generation • Automated essay scoring

  24. Thank you for your attention. • Deborah Moore, Assessment Specialist • 101B Alumni GymOffice of Institutional Research, Planning, & Effectiveness • dlmoor2@email.uky.edu • 859/257-7086 • http://www.uky.edu/LexCampus/

More Related