Skills Diagnosis with Latent Variable Models

1. Skills Diagnosis with Latent Variable Models

2. Topic 1:A New Diagnostic Paradigm

3. Introduction • Assessments should aim to improve, and not merely ascertain the status of student learning • For test scores to facilitate learning, they need to be interpretative, diagnostic, highly informative, and potentially prescriptive • Most large-scale assessments are based on traditional unidimensional IRT models that only provide single overall scores • These scores are useful primarily for ordering students along a continuum

4. Alternative psychometric models that can provide inferences more relevant to instruction and learning currently exist • These models are called cognitive diagnosis models (CDMs) • Alternatively, they are referred to as diagnostic classification models (DCMs) • CDMs are multiplediscretelatent variable models • They are developed specifically for diagnosing the presence or absence of multiple fine-grained attributes (e.g. skills, cognitive processes or problem-solving strategies)

5. Fundamental difference between IRT and CDM: A fraction subtraction example • IRT: performance is based on a unidimensional continuous latent trait • Students with higher latent traits have higher probability of answering the question correctly

7. Fundamental difference between IRT and CDM: A fraction subtraction example • IRT: performance is based on a unidimensional continuous latent trait • Students with higher latent traits have higher probability of answering the question correctly • CDM: performance is based on binary latent attribute vector • Successful performance on the task requires a series of successful implementations of the attributes specified for the task

8. Required attributes: (1) Borrowing from whole (2) Basic fraction subtraction (3) Reducing • Other attributes: (4) Separating whole from fraction (5) Converting whole to fraction

9. Basic Elements and Notations of CDM • The response vector of examinee i will be denoted by , • The response vector contains J items, as in, • The attribute vector of examinee i will be denoted by • Each attribute vector or pattern defines a unique latent class • Thus, K attributes define latent classes

10. Example: When , the total number of latent classes is • Although arbitrary, we can associate the following attribute vectors with the following latent classes:

11. Basic CDM Input • Like IRT, CDM requires an binary response matrix as input • Unlike IRT, CDM in addition requires a binary matrix called the Q-matrix as input • The rows of the Q-matrix pertain to the items, whereas the columns the attributes • The 1s in the jth row of the Q-matrix identifies the attributes required for item j

12. Examples of Attribute Specification

13. Examples of Attribute Specification

14. Basic CDM Output • The goal of CDM is to make inference about the attribute vector • The basic CDM output gives the (posterior) probability the examinee has mastered each of the attributes • That is, we get • For example, , indicates that we are quite certain that examinee has already mastered attribute 1

15. Each examinee gets a vector of posterior probabilities • For reporting purposes, we may want to convert the probabilities into 0s and 1s • We can use different rules for this conversion • If ; Otherwise,

16. Example:

17. Each examinee gets a vector of posterior probabilities • For reporting purposes, we may want to convert the probabilities into 0s and 1s • We can use different rules for this conversion • If ; Otherwise, • If ; or If ; Otherwise,

18. Example: ? – means we do not have sufficient evidence to conclude one way or the other