Han van der Maas Gunter Maris, Denny Borsboom University of Amsterdam, Cito Twente , 2009

1. q≥0 Cognitive psychology meets psychometric theory:On the relation between process models for decision making and latent variable models for individual differences Han van der Maas Gunter Maris, Denny Borsboom University of Amsterdam, Cito Twente, 2009

2. 2PL model in item response theory • Main model for educational and psychological measurement in Psychometrics • Justifications • Logistic equation does the job (Toolbox) • Derivation from ‘desirable’ statistical properties (Fischer) • Derivation from ‘desirable’ measurement properties (Roskam) • Derivation from psychological process model (Tuerlinckx & De Boeck)

3. Diffusion model • Stochastic accumulation of evidence stops when decision threshold is reached • Implements SPRT (optimal accuracy given RT) • Very influential and very well studied model • More complex biological realistic models reduce to this model • Explains the speed accuracy trade-off

4. Relation to IRT RT X=1 • Two main parameters: • Boundary separation (decision criterion): a • Drift rate (rate of evidence accumulation): v a if z=a/2 v z t if P(X=1) is large 0 Tuerlinckx & De Boeck (2005): Drift = ability – difficulty (v = q – b) Discrimination = boundary separation (a = a) X=0

5. Three qualitative predictions • Subjects are slowest when v=0, that is when q=b • When q<<b,responding is very fast • For negative v, i.e.θ<b, allowing moretime to think reducesthe probability correct. • By reducing time to thinkP(+) -> .5, irrespective of θ Special psychological meaning of q=b Long time limit Short time limit

6. Attitudes: “I stick to my decisions” (agree/not agree) • RT decreases with |q-b| • OK: extreme answers are fast • if q<band time limit increases P(agree)0 • OK: the longer I think about: I don’t stick to my decisions • If time to think (a) 0, P(agree) 1/2 • OK: I just say something

7. Ability: “24+79” (93/114/103/130 ) • RT decreases with |q-b| • NO, best subjects are fastest • if q<b and time limit increases P(agree)0 • NO, they guess because they know that they don’t know • If time to think (a) 0, P(+) 1/2 • NO: P(+) 0 for open questions and to 1/M for M multiple choice questions

8. Thus • For two choice attitude/personality tests the diffusion IRT model works well! • But not for multiple choice ability tests: • Problem of Multiple choice • Diffusion model is two choice model, IRT (2PL) is dichotomously scored not dichotomous choice! Guessing does normally not give P(+)=1/2 • Problem of Guessing • In ability testing subjects often guess and don’t score below chance level

9. Solving the firstproblem: Multiple choice

10. Correctionfor MC in 2PL • Assuming equal attractiveness of incorrect alternatives in Bock’s nominal response model • gives:

11. Solving the second problem: guessing • Subjects don’t score below chance level • Typical solution: 3PL • We propose a restricted 2PL model • Main idea: Ability can be zero but can not be negative, so v≥0 • subjects do not score below chance level • increasing time limit will not decrease P+

12. Ability ≥ 0 • In IRT bipolar traits such as attitudes and unipolar traits such as abilities are not distinguished • But ability is very special (think of walking) • No negative ability (no negative difficulty) • Real zero point (no ability) • With sufficient time and ability>0 any item can be solved • We need a diffusion IRT model with positive v

13. Idea: a and v both have an item and a person part drift v ability person vp (>0) difficulty item vi (>0) requirement v=f(vp, vi)>0 one sensible solution v=vp/vi boundary separation a response caution ap (>0) time pressure ai (>0) requirement a=g(ap, ai)>0 one sensible solution a=ap/ai

14. Quotient diffusion 2PLM (Q-diffusion model) All a and v are positive Response caution Time pressure Difficulty Ability

15. so if

16. Simple case • Ignore a,v distinction (RT’s required) • equal time limits for all item in test Easiness > 0 Ability > 0 Correction for multiple choice

17. ICC for dichotomous items

18. ICC for MC (4 choices)

19. ICC for MC (10 choices)

20. ICC ‘open’ questions (many choices)

21. Advantages • Process model for attitude and ability traits • ‘Mechanistic’ interpretations of IRT parameters • q is product of drift rate and boundary separation • Meaningful zero point (ratio properties) • Speed accuracy trade-off incorporated • a is easiness parameter, reciprocal of product of item drift rate and time pressure • b is intercept parameter, guessing probability • Model of accuracy, but also model of RT • Guessing explained by restricting the 2PL • Simple extension to MC

22. Relation to other IRT models • Ramsay’s Q model • Person fit • van der Linden’s IRT RT model • 2PL and multidimensional IRT • Guessing models

23. Ramsay’s quotient model • Ramsay (1989) investigates simple models for Skj • Difference model (q-b): Rasch model • Quotient model (q/b) • Fits better than Rasch model and Rasch model with guessing in 4 examples • if q = apvp, b = aivi and K=M-1 the Q-model and the Q-diffusion model are equivalent • Note: model in Cressie & Holland (’83) also equivalent:

24. Person fit Q-diffusion for ability D-diffusion for attitude is item discrimination (time pressure) is person discrimination (response caution) Person discrimination can be estimated by varying time pressure over items and by using RT data

25. IRT model for RT • General model: van der Linden’s Hierarchical model • Fundamental equation for RT modeling:

26. Translation of the item and person parameters of van der Linden’s model to diffusion model parameters • If speed and ability parameters at the second level in van der Linden’s model are positively correlated, then individual differences are primarily due to differences in drift rate (for an example, see van der Linden, 2007). • If these parameters correlate negatively, the individual differences in drift rate are probably similar across subjects, and differences are mainly due to differences in response caution (see example 2 of Klein Entink, Fox, & van der Linden, 2009).

27. 2PL • the Q-model and Q-diffusion model are restricted 2PL models • In the Q-diffusion model: • In the end all items will be passed, for all items, P(+|q>0,ai∞)=1, e.g. one Guttman item • If not (because the item also requires a jump) it measures another additional ability. The test is not unidimensional! • So any ability test with some items that I can solve and others that I will never solve (P+ does not increase with longer time limit), tests for more than one ability • requires Conjunctive Multidimensional Q-diffusion model b+ bx q

28. Guessing • Two process models • 3PL (p and g process) • 1PL-AG (p and ability dependent g process) • San Martin, del Pino, and De Boeck • DINA • One state models • Difficulty dependent guessing • Hessen 2005, d model • Q-diffusion model: one state ability based guessing

29. Fitting • Ramsay fit the QM, DM and DM-G (Rasch model withguessing) to four datasets and foundthat QM fitted best in all cases • For the full diffusion model several programs are available • Fast-DM(Voss & Voss, 2007) • DMAT (Vandekerckhove & Tuerlinckx, 2007; Voss & Voss, 2007) • Hierarchical IRT RT model of van der Linden • CIRT: Fox, J. P., Klein Entink, R. H., & van der Linden, W. J. (2007).

30. discussion • Diffusion model is simplistic model of simple perceptual discriminations. It is too simple for complex processes involved in typical ability items in IRT • But 2PL is simplistic too • Better a simple model than no model at all • We don’t need a process model • Then IRT is curve fitting without explanatory value • 3PL will fit better • 3PL is difficult to fit and uses many more parameters then the Q-diffusion IRT model • The point of the Q-diffusion IRT is not goodness of fit but theoretical clarity

31. thanks

32. Note: Capacities versus abilities • For Lumsden’s sticks for measuring length the Guttman model is OK • This also applies to STM • Digit span items

33. QM versus DM-G

34. ICC’s of new guessing model