Reliability & Validity

1. Reliability & Validity

2. Reliability Reliability – the degree to which a test consistently measures whatever it is measuring • Characteristics • Expressed as a coefficient ranging from 0 to 1 • A necessary but not sufficient characteristic of a test

3. Reliability Are the findings (or a specific measure) consistent if you were to do the study over again? A study can be reliable, but not valid. Furthermore, it cannot be valid unless it is reliable. Thus, reliability is absolutely required. Validity is equally important, but the degree of validity (such as external validity) may not be very high depending on the nature of the study.

4. Reliability: the problem of error • Error is the difference between the observed score and the ‘true’ score. • Random error occurs: • Due to observers… • Due to individual variation (age, mood, etc) • Due to inconsistent situations during data collection (i.e., survey on patriotism after 9/11)

5. Reliability Four reliability coefficients • Stability – consistency over time with the same instrument • Test – retest • Estimated by a correlation between the two administrations of the same test • Equivalence – consistency with two alternate or parallel tests administered at the same time • Alternate or Parallel forms • Estimated by a correlation between the tests

6. Reliability Four reliability coefficients (continued) • Internal consistency – artificially splitting the test into halves • Several coefficients – split halves, KR 20, KR 21, Cronbach alpha • All coefficients provide estimates ranging from 0 to 1

7. Reliability Four reliability coefficients Scorer/rater – consistency of observations between raters • Inter-judge – two observers • Intra-judge – one judge over two occasions • Estimated by percent agreement between observations

8. Reliability Standard error of measurement (SEM) • An estimate of how much difference there is between a person’s obtained score and his or her true score

9. Validity • Validity – extent to which interpretations made from a test score are appropriate • Characteristics • The most important technical characteristic • Situation specific • Does not refer to the instrument but to the interpretations of scores on the instrument

10. Validity Four types Content – to what extent does the test measure what it is supposed to measure • Item validity • Sampling validity • Determined by expert judgment

11. Validity Criterion-related • Predictive – to what extent does the test predict a future performance • Concurrent - to what extent does the test predict a performance measured at the same time • Estimated by correlations between two tests

12. Validity Construct – the extent to which a test measures the construct it represents • Underlying difficulty defining constructs • Estimated in many ways Consequential – to what extent are the consequences that occur from the test harmful • Estimated by empirical and expert judgment

13. Validity Factors affecting validity • Unclear test directions • Confusing and ambiguous test items • Vocabulary that is too difficult for test takers

14. Validity Factors affecting validity (continued) • Overly difficult and complex sentence structure • Inconsistent and subjective scoring • Untaught items • Failure to follow standardized administration procedures • Cheating by the participants or someone teaching to the test items