Pokpong Songmuang, Keizo Nagaoka Waseda University, Japan

1. Development of Design and Analysis Methods for Item Bank for Constructing Multiple Equivalent Test Forms Pokpong Songmuang, Keizo Nagaoka Waseda University, Japan

2. Outline • Introduction • Bees algorithm for constructing multiple equivalent test forms • Bees algorithm for designing an item bank for constructing multiple equivalent test forms • Bees algorithm for analyzing a given item bank for constructing multiple equivalent test forms • Experiments • Summary

3. Introduction • Educational assessments sometimes need “multiple test forms” in which each form consists of a different set of items but still has qualities that are equivalent (e.g., rate of correct answers, equivalent amounts of test information based on item response theory) to the others.

4. Automated test construction system Item bank Select items Multiple equivalent test forms Automated test construction system Constraints Test 1 Test 1 Test 1 Test N Number of items from each area Statistical information • Example： • Test information function • Average score 60scores • Response time 20minutes • Number of items from one field 10 items Test constraints setup interface

5. Previous methods for designing and analyzing an item bank • Stocking and Swanson (1998), Veldkamp and van der Linden (2000), and van der Linden, Veldkamp and Reese (2000) proposed methods using integer programming for designing an item bank for computerized adaptive testing instead of constructing METF. • To address this problem, Belov and Armstrong (2006) proposed methods based on a stochastic search for designing and analyzing an item bank for constructing METF

6. Problems in previous methods The problems of previous design and analysis methods for an item bank for constructing multiple equivalent test forms (METF) are as follows: • Only few methods were proposed for designing and analyzing an item bank for constructing METF, • The qualities of test forms constructed form the item bank designed by the previous methods were different.

7. Propose We apply Bees algorithm (BA) to design and analyze an item bank for constructing METF. The advantages of the proposed method are as follows: • The proposed methods design and analyze an item bank for constructing METF, • The proposed methods are based on BA that maximizes the equivalent among test forms.

8. Bees algorithm for constructing multiple equivalent test forms In the method of constructing test forms based on BA (Songmuang and Ueno, 2011), we divide the construction of test forms into two steps: • Step A • Goal: satisfying test constraints without taking into consideration the equivalence of test forms • Step B • Goal: searching for equivalent set of test forms

9. Bees algorithm for Designing an item bank for constructing multiple equivalent test forms • A blue print of item bank describes the required number of items in each type of items. • Since BA is an approximation method that constructs test forms to satisfy test constraints in several patterns, we can use this information to create a blue printof an item bank for constructing METF to satisfy the given test constraints. • This blue print is more flexible than one created by an exact method such as linear programming (e.g. Veldkamp and van der Linden, 2000; van der Linden, Veldkamp and Reese, 2000). • Moreover, the proposed method relaxes the problem of difference in fitting errors among test forms as described in Songmuang and Ueno (2011) so the designed item bank is appropriate for constructing METF.

10. Bees algorithm for Designing an item bank for constructing multiple equivalent test forms Test constraints Generated item bank Bees Algorithm Test 1 Test 1 Test 1 Test N 2. Construct METF using BA to minimize the difference of test information among test forms and to satisfy the given test constraints. Blue print 1. Generate all types of items but one item for each type. 3. Count the selection frequencies of each type of item

11. Bees algorithm for analyzing a given item bank for constructing multiple equivalent test forms Failed tests Test constraints Test 1 Test 1 A given item bank Success tests Bees Algorithm using Step A Test 1 Test N BA constructs METF using the given item bank to satisfy the given test constraints The details of failed test forms are used to indicate the required type of items not only content of items but also the statistical characteristics

12. Experiments • To show effectiveness of the proposed method to design the item bank, we compare the qualities of test forms constructed using the item banks that are designed by the proposed method and the design method based on stochastic search (SS) (Belov and Armstrong, 2006).

13. Result 1 • Average and Average of Standard deviations of fitting errors of test forms constructed using the item banks designed by the design method based on BA and SS

14. Experiment 2 • Before perform the experiment, we define test constraints and generate item banks by randomly select 100 items from the actual item bank. • Next, we analyze the generated item banks using the proposed method based on BA and the analyze method based on SS. • According to the results of the analysis, we have the types of required items to improve the generated item banks. • We improve the generated item banks by selecting the items according to the indicated types from the actual item bank and adding them to the generated item banks. • Next, we use BA to construct METF using the improved item bank to satisfy the given test constraints.

15. Result 2 • Average and Average of Standard deviations of fitting errors of constructed test forms from the item banks improved by the analysis method based on BA and SS

16. Summary • We proposed methods for designing and analyzing an item bank for constructing multiple equivalent test forms based on Bees algorithm. • Moreover, we performed some experiments to show the effectiveness of the proposed methods. • According to the results, the equivalences of test forms constructed from the item bank designed by the proposed method were better than the equivalence of test forms constructed from the item bank designed by the previous method. • Moreover, the results showed that the equivalence of the test forms, that were constructed using the item banks improved by the proposed analysis method, was better than the test forms constructed using the item banks improved by the previous analysis method

17. Thank you for your kind attention

18. Bees algorithm Item selection probability distribution Population Smallest 1 0 1 0 0 1 0 0 1 0 Fitting errors Training test form Largest Store thetest forms with smaller fitting errors New test form 0 1 1 0 0 1 1 1 1 0 0 0 0 0 1 0 1 1 0 0 0 0 1 0 0