Technical Issues

1. Technical Issues • Two concerns • Validity • Reliability

2. Data Collection – Quiz 1 • Answer the five questions on Quiz 1.

3. Data Collection – Quiz 1 Answers • Score you paper using the following key • A • B • A • B • B

4. Data Collection – Quiz 1 • How well did you do? • Should I use this score as a part of your grade? • Does this score indicate your level as a graduate student? • “What we have here is a serious lack of communication!” • Most students object strongly to using this score as a part of their grade because it isn’t fair. • Most students object strongly to being labeled “bright” or “challenged” on the basis of their grade. • Their reasoning is the test isn’t fair – it doesn’t cover material relevant to this course. • Welcome to the technical world of instrumentation.

5. Technical Issues • Validity – extent to which interpretations made from a test score are appropriate • Characteristics • The most important technical characteristic • Situation specific • Does not refer to the instrument but to the interpretations of scores on the instrument • Best thought of in terms of degree

6. Technical Issues • Validity (continued) • Four types • Content – to what extent does the test measure what it is supposed to measure • Item validity • Sampling validity • Determined by expert judgment

7. Technical Issues • Validity (continued) • Construct – the extent to which a test measures the construct it represents • Underlying difficulty defining constructs • Estimated in many ways • Criterion-related • Predictive – to what extent does the test predict a future performance • Concurrent - to what extent does the test predict a performance measured at the same time • Estimated by correlations between two tests

8. Technical Issues • Validity (continued) • Consequential – to what extent are the consequences that occur from the test harmful • Estimated by empirical and expert judgment • Factors affecting validity • Unclear test directions • Confusing and ambiguous test items • Vocabulary that is too difficult for test takers

9. Technical Issues • Factors affecting validity (continued) • Overly difficult and complex sentence structure • Inconsistent and subjective scoring • Untaught items • Failure to follow standardized administration procedures • Cheating by the participants or someone teaching to the test items

10. Technical Issues • Reliability – the degree to which a test consistently measures whatever it is measuring • Characteristics • Expressed as a coefficient ranging from 0 to 1 • A necessary but not sufficient characteristic of a test

11. Technical Issues • Test reliability • Stability – consistency over time with the same instrument • Test – retest • Estimated by a correlation between the two administrations of the same test • Equivalence – consistency with two parallel tests administered at the same time • Parallel forms • Estimated by a correlation between the parallel tests

12. Technical Issues • Test reliability (continued) • Internal consistency – artificially splitting the test into halves • Several coefficients – split halves, KR 20, KR 21, Cronbach alpha • All coefficients provide estimates ranging from 0 to 1