A Method for Cooperative Test Assembly for Large-Scale Assessment

1. A Method for Cooperative Test Assembly for Large-Scale Assessment Jon Brasfield Wonsuk Kim Matt Finkelman Louis Roussos

2. Automated Test Assembly (ATA) • Foundation: Birnbaum (1968) suggested using the additive relationship between Item Information and the TIF to assemble a test • General outline, not actual method • Computers weren’t powerful enough

3. ATA Literature • Lots of ATA literature • Doesn’t involve content experts • Need for a method that combines the benefits of ATA with the substantive contributions of content experts • More practical

4. Outline of Project • Use ATA to improve the feedback loop inherent in Form Pulling • Will automatically create forms that meet psychometric targets (TCC and TIF) and content constraints • ELA Test – Create form options • Math Test – Create initial form

5. ELA Test • Create detailed feedback to give to test developers • Based on initial form – constructed primarily on content concerns • Developers create initial form, denote “definite” and “maybe” items • Content Constraints: item type, passage association, standard addressed • Psychometric constraint: Target TCC tolerance

6. ELA test – Our process Item Pool 5,000 Times Test Form

7. ELA Test – Our process Item Pool 5,000 times Test Form

8. ELA Test – Our Process • Must meet tolerance on Target TCC

9. Initial Form

10. Information: Initial form

11. Results from random replacement

12. ELA Test – Best Result • Used Genetic Algorithm • Based on theories of evolution and natural selection • Originally discussed in Computer science (Holland, 1968) • Potential solutions “evolve” toward a target, or optimum solution • Require 3 components: Decision variable, objective function, and constraints

13. ELA Test – best result • Our objective function: minimize total distance from Target TIF • A set of items that meets the constraints is selected (a “parent”) • For each parent, each item is temporarily replaced (one at a time) with a random item from the pool • Process is repeated, selecting the fittest sets each time

14. Genetic Algorithm – “best” solution

15. Actual Form

16. Information: Actual form

17. Information comparison

18. Summed absolute distance comparison

19. Bottom Line: ELA Test • All forms found exhibited a lower summed absolute distance from the target than the initial form or the actual form Target TIF Improvement

20. Math Test • Initial form provided by psychometrics: Optimize statistics while staying within content constraints • Constraints include Item type, strand, standard, age of item • Only one form is needed, so GA is used

21. Math Results

22. Math Results

23. Math Test – Bottom Line TTIF Distance Reduction of 89.17% TTCC Distance Reduction of 91.68%

24. Summary • ELA Test – Content Experts can provide an initial form, psychometrics can give multiple statistically sound options while keeping important items • Math Test – Can easily construct a “first draft” form that is very close to psychometric targets and meets content constraints

25. Future research • Qualitative feedback from content experts, DOE • Use Genetic Algorithm to develop multiple forms