Introduction to Classical Test Theory (CTT)

1. Introduction to Classical Test Theory (CTT)

2. X = T + e Meaning of X, T, and e. Basic assumptions Parallel tests Reliability Standard Error of Measurement p-value & point-biserial

3. X = T + e X = observed score (this is obvious) T = “true” score e = error T and e require some explanation

4. T, “true” score Take two forms of equal difficulty, you get two different scores. Suppose you take many such tests T is your mean score on all these tests T is an unobservable theoretical concept

5. e, error Does NOT refer to “error” as in baseball NOR to mistakes in testing or scoring. e is the difference between X and T. e is, thus,related to “standard error” If we have many samples: X isthe sample statistic T is the average of X over the samples standard error of X is SD of X over the samples, i.e., approximately the average of e

6. e, error Why does a student get different scores on two different tests of the same difficulty level? Short answer: Luck! Example: Spelling test, 1000-word pool. Suppose you know 90%. Imagine two tests, 10 words each, assembled to have sameaverage score for all students. On one test, by luck, you know all 10 words. On the other, by really bad luck, you know only 7!

7. Basic Assumptions of CTT No cheating. No copying between examinees Luck is completely random. Difficulty level of form does not affect luck How you did on another test does not affect luck If you take two forms, no “learning” occurred while taking the first

8. Parallel Tests According to CTT, two tests are parallel if: • All students have same “true” score on both tests. • SD of observed scores is the same on both tests. First condition relates to test difficulty. Second condition relates to reliability. So, we could simplify and say two tests are parallel if: • They have the same difficulty • They have the same reliability

9. Parallel Tests: Beyond CTT Of course, Parallel Tests must be much more than just statistically parallel: • Types of questions • Content • Time limit • Test-taking and administration directions • Legibility • Art work

10. Reliability A test is a measurement. Two parallel tests are two independent measurements. A student’s scores on two parallel tests are likely to be different. Roughly speaking, the degree to which such differences are minimized is Reliability. The greater the consistency of test scores between a test and its parallel form, the greater the reliability of the test. Definition: Reliability = correlation b/t scores on parallel forms.

11. Correlation Analysis of Data that come in pairs • Examples: • {Sodium/serving | Sugar/serving} in sample of 10 cereals • {Height | Weight} in a sample of 25 individuals • {Score on one item | sum score on rest of items} in a sample of 400 students (point-biserial) • {Score on test form | Score on parallel form} in a sample of 1000 students (reliability)

12. Interpreting Correlations • Correlations measure the degree to which one variable has a linear relationship with another. • 0 < Magnitude of Correlation < 1 0 means no linear relationship 1 means perfect linear relationship • Sign of correlation: • Positive increase in one gives increase in other • Negative  increase in one gives decrease in other

13. Interpreting Correlations (.87, .56, -.57, .19)

14. Interpreting Reliability as a Correlation • The degree to which scores on a test are linearly related to scores on a parallel test. • 0 < Reliability < 1 • Reliability is typically about 0.9 for standardized tests.

15. Methods for Estimating Reliability • Parallel forms (never have parallel forms) • Test-Retest (same test twice? Forget it!) • Split-half • Cronbach’s alpha

16. Estimating Reliability: Split-half • Here’s a great idea: • Split the test in half (two parallel halves) • Correlate scores on the two halves • Scale up to get the correlation that corresponds to two full-length tests • Hard to get two parallel halves.

17. Estimating Reliability: Cronbach’s a • Another great idea: • Do all possible split halves • Take average of all the scaled-up correlations • That’s Cronbach’s alpha! • Sounds computationally intensive

18. Estimating Reliability: Cronbach’s aN= No. of items on testSD =SD of scores on testpi = p-value for item i

19. Standard Error of Measurement SEM is an estimate of the average size of e in a population. Beginning with X = T + e we can derive the following formula for SEM: SEM = SD x SQRT(1 – Reliability) CTT assumes every examinee has the same SEM If everyone took many parallel forms, the SD of their scores would be the same for everyone.

20. CTT Item Parameters P-value: the mean of the scored responses for an item Used as an indicator of item difficulty Point-biserial: the correlation between score on an item and the sum-score on all the other items on the test. Used as an indicator of item discrimination power. Simple, yet informative, statistics: accurately measured with sample sizes as small as 400

21. Limitations of CTT Item parameters change (even their order of difficulty!) with student population, making them hard to interpret. True scores change across test forms. Hard to compare students who took different forms. Test level model, not item model. SEM is same for all examinees. Reliability changes with student population

22. Summary: Intro to CTT Basic equation is: X = T + e Parallel Test Forms Examinees have same T on both forms Observed Score SD is same for both forms. Reliability: Correlation of X across Parallel Tests. Estimated by Cronbach’s a (modified split-half approach)

23. Summary: Intro to CTT SEM=Standard Error of Measurement SD of X for an examinee over many parallel forms Related to reliability by simple formula CTT statistics have severe limitations item statistics change with student population student statistics change with test forms SEM is same for all student scores Reliability changes with student population

24. Introduction to CTT Thanks again for coming! And for the nice comments on my Basic Stats presentation. Hope to see you next time when Liz will unravel the complexities of setting standards and show how CTT statistics and human judgment are used to yield a logical step-by-step process that makes sense of this complex enterprise.

25. Famous Two-Number Data Summary:Mean & Standard Deviation • Mean: the ordinary average of all the data. • If you had to pick one number to typify your data. • p-value is a mean • Standard Deviation (SD): average deviation from mean • Obviously, not all the data equal the mean • SD tells, on average, how spread out the data are from the mean • Can be used to identify extreme values

26. Mean & SDExample using p-values Mean = 0.63 1 SD = 0.18 2 SD’s 50 25 0 0 0.2 0.4 0.6 0.8 1

27. Mean & SDExample using Heights Mean = 67 1 SD = 3.48 40 2 SD’s 32 24 16 8 0 55 60 65 70 75 80

28. The “Standard Error” • Data are sampled from a population. • Sample Mean is calculated. • A 2nd sample would have a different Mean.

29. The “Standard Error” • How much would the Sample Mean vary, on average, over many samples? • What would the SD of the Sample Mean be over many samples? • That’s the Standard Error (SE)! • It tells you how reliable your Sample Mean is.

30. Standard Error • Ok, this is very nice. But in real life we can’t take 10,000 samples! • In real life we get ONE sample!How can we possibly figure out the SE for the sample mean for a real data sample? • Magic of Statistics SE = (SAMPLE SD ) / SQRT(sample size)