1 / 22

Latent Rank Theory: Test Theory for Making Can-Do Chart

Latent Rank Theory: Test Theory for Making Can-Do Chart. SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp. Accuracy. Scale (Weighing machine) A 1 weighs 73 kg f W (A 1 )=73 f W (A 1 ) ≠ 74 f W (A 1 ) ≠ 72. Academic test

lareina-gordon
Download Presentation

Latent Rank Theory: Test Theory for Making Can-Do Chart

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Latent Rank Theory:Test Theory for Making Can-Do Chart SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp

  2. Accuracy • Scale (Weighing machine) • A1 weighs 73 kg • fW(A1)=73 • fW (A1)≠74 • fW (A1)≠72 • Academic test • B1 scores 73 points • fT(B1)=73 • fT(B1)≠74 ? • fT(B1)≠72 ?

  3. Discriminating Power • Scale (Weighing machine) • A1 weighs 73 kg • A2 weighs 75 kg • fW(A1)<fW (A2) • Academic test • B1 scores 73 points • B2 scores 75 points • fT(B1)<fT (B2) ?

  4. Resolution • Scale (Weighing machine) • A1 weighs 73 kg • A2 weighs 75 kg • A3 weighs ... • Academic test • B1 scores 73 points • B2 scores 75 points • B3 scores ... kg T

  5. Test Limitations Precise measurement is almost impossible CTT reliabilities: 10% measurement error A test is at best capable of classifying academic ability into 5–20 levels Why continuous scale? Classical Test Theory: Continuous Scale Item Response Theory: Continuous Scale Common European Framework of Reference for Languages (CEFR) 6 levels: A1, A2, B1, B2, C1, C2

  6. Continuous academic ability evaluation scale based on IRT or CTT It is difficult to explain the relationship between scores and abilities because individual abilities also change continuously For Qualifying Tests Ordinal academic ability evaluation scale based on Neural Test Theory Because the individual abilities also change in stages, it is easy to explain the relationship between scores and abilities. This increases the test’s accountability. Graded evaluation ↓ Accountability ↓ Qualification test

  7. Latent Rank Theory(or Neural Test Theory) A test theory Ordinal scale (not continuous scale) Self-organizing map (SOM) or generative topographic mapping (GTM) mechanism Shojima, K. (2009) Neural test theory. K. Shigemasu et al. (Eds.) New Trends in Psychometrics, Universal Academy Press, Inc., pp. 417-426. Shojima, K. (2011) Local dependence model in latent rank theory. Jpn J of Applied Statistics, 40, 141-156.

  8. Statistical Learning Framework in LRT ・For (t=1; t ≤ T; t = t + 1) ・U(t)←Randomly sort row vectors of U ・For (h=1; h ≤ N; h = h + 1) ・Obtain zh(t)from uh(t) ・Select winner rank for uh(t) ・Obtain V(t,h) by updating V(t,h−1) ・V(t,N)←V(t+1,0) Point 1 Point 2

  9. LRT Mechanism (SOM) 1 0 1 1 1 1 0 0 1 1 0 0 Number of Items 0 1 1 0 0 0 0 0 0 0 0 1 Input Point 1 Point 2 Point 2 Point 1 Latent Rank Scale

  10. Point 1: Winner Rank Selection Likelihood ML Bayes

  11. Point 2: Update the Reference Vectors The nodes of the ranks nearer to the winner are updated to become closer to the input data h: tension α: size of tension σ: region size of learning propagation

  12. Example A geography test of the NCT

  13. Fit Indices and N of Ranks N of ranks is 10 N of Ranks is 5

  14. Item Reference Profile (IRP) Monotonic increasing constraint can be imposed.

  15. Test Reference Profile (TRP) Strongly ordinal alignment condition (SOAC) All IRPs increasemonotonically  TRP also increasesmonotonically Weakly ordinal alignment condition (WOAC) TRP increasesmonotonically, but not all IRPs increase monotonically For the scale to be ordinal, at least the WOAC must be satisfied. • (Weighted) sum of IRPs • Expected score at eachlatent rank

  16. Rank Membership Profile (RMP) Posterior distribution of the latent rank to which each examinee belongs RMP

  17. Examples of RMP 17

  18. Can-Do Chart (Example)

  19. Extended Models Graded LRT Model (RN07-03) LRT model for ordinal polytomous data Nominal LRT Model (RN07-21) LRT model for nominal polytomous data Continuous LRT Model Multidimensional LRT Model

  20. Graded LRT ModelBoundary Category Reference Profiles

  21. Graded LRT ModelItem Category Reference Profile

  22. Nominal LRT ModelItem Category Reference Profile*Correct selection, x Combined categories selected less than 10% of the time

More Related