1 / 23

Neural test theory model for graded response data

Neural test theory model for graded response data. SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp. Accuracy of tests. Academic test B 1 scores 73 points f T (B 1 )=73 f T (B 1 ) ≠ 74 ? f T (B 1 ) ≠ 72 ?. Weighing machine

braith
Download Presentation

Neural test theory model for graded response data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Neural test theory model for graded response data SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp

  2. Accuracy of tests • Academic test • B1 scores 73 points • fT(B1)=73 • fT(B1)≠74 ? • fT(B1)≠72 ? • Weighing machine • A1 weighs 73 kg • fW(A1)=73 • fW (A1)≠74 • fW (A1)≠72

  3. Discriminating ability of tests • Weighing machine • A1 weighs 73 kg • A2 weighs 75 kg • fW(A1)<fW (A2) • Academic test • B1 scores 73 points • B2 scores 75 points • fT(B1)<fT (B2) ?

  4. Resolving ability of tests • Weighing machine • A1 weighs 73 kg • A2 weighs 75 kg • A3 weighs ... • Academic test • B1 scores 73 points • B2 scores 75 points • B3 scores ... kg T

  5. Neural Test Theory (NTT) Academic tests are an important public tool Precise measurements are difficult 10% measurement error Tests are at best capable of classifying academic ability into 5–20 levels Neural test theory (NTT) Shojima, K. (2009) Neural test theory. K. Shigemasu et al. (Eds.) New Trends in Psychometrics, Universal Academy Press, Inc., pp. 417-426. Test theory that uses the mechanism of a self-organizing map (SOM; Kohonen, 1995) Latent scale isordinal 5

  6. Continuous academic ability evaluation scale based on IRT or CTT It is difficult to explain the relationship between scores and abilities because individual abilities also change continuously For Qualifying Tests Ordinal academic ability evaluation scale based on Neural Test Theory Because the individual abilities also change in stages, it is easy to explain the relationship between scores and abilities. This increases the test’s accountability. Graded evaluation ↓ Accountability ↓ Qualification test

  7. Statistical Learning Procedure in NTT ・For (t=1; t ≤ T; t = t + 1) ・U(t)←Randomly sort row vectors of U ・For (h=1; h ≤ N; h = h + 1) ・Obtain zh(t)from uh(t) ・Select winner rank for uh(t) ・Obtain V(t,h) by updating V(t,h−1) ・V(t,N)←V(t+1,0) Point 1 Point 2 7

  8. NTT Mechanism 1 0 1 1 1 1 0 1 0 0 1 0 Number of items 0 1 1 0 0 0 0 0 0 1 0 0 Response Point 1 Point 2 Point 2 Point 1 Latent rank scale 8

  9. Point 1: Winner Rank Selection The least squares method can also be used. Likelihood ML Bayes 9

  10. Point 2: Update the rank reference matrix The nodes of the ranks nearer to the winner are updated to become closer to the input data h: tension α: size of tension σ: region size of learning propagation 10

  11. Analysis Example A geography test 11

  12. Fit Indices Useful for determining thenumber of latent ranks ML, Q=10 ML, Q=5 12

  13. Item Reference Profiles 13 Monotonic increasing constraint can be imposed

  14. Test Reference Profile (TRP) Weakly ordinal alignment condition TRP increasesmonotonically, but not all IRPs increase monotonically Strongly ordinal alignment condition All IRPs increasemonotonically  TRP also increasesmonotonically For the latent scale to be anordinal scale, it must at least satisfy the weakly ordinal alignment condition (WOAC). • Weighted sum of IRPs • Expected value of eachlatent rank 14

  15. Rank Membership Profile (RMP) Posterior distribution of the latent rank to which each examinee belongs RMP 15

  16. Examples of RMP 16

  17. Extended Models Graded Neural Test Model (RN07-03) NTT model for ordinal polytomous data Nominal Neural Test Model (RN07-21) NTT model for nominal polytomous data Continuous Neural Test Model Multidimensional Neural Test Model 17

  18. Graded NTT ModelBoundary Category Reference Profiles

  19. Graded NTT ModelItem Category Reference Profile

  20. Nominal NTT ModelItem Category Reference Profile* Correct selection, x Combined categories selected less than 10％ of the time

  21. Website http://www.rd.dnc.ac.jp/~shojima/ntt/index.htm Software EasyNTT By Prof. Kumagai (Niigata Univ.) Neutet By Prof. Hashimoto (NCUEE) Exametrika By Shojima (NCUEE) 21

  22. Demonstration of Exametrika

  23. Can-Do Chart (Example) 23

More Related