Psychometric theory Tim Wildschut Office: 4009 Email: timw@soton.ac.uk

1. Psychometric theory Tim Wildschut Office: 4009 Email: timw@soton.ac.uk

2. Programme: February 18: Advantages of standardized measures, levels of measurement, and validity. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill. Chapters 1 and 3. February 25: Classical test theory and assessment of scale reliability. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill. Chapters 6 and 7. March 3: Test construction. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. New York: McGraw-Hill. Chapter 8.

3. Advantages of standardized measures Objectivity: Observations become independently verifiable. Quantification: Allows for finer detail and more powerful analyses. Communication: Statistical analyses of standardized measures are easily communicated. Economy: Standardized measures can be quickly and easily administered to large numbers of people. Scientific generalization: Facilitate the process of finding underlying order in particular events.

4. Levels of measurement

5. Validity Extent to which a scale measures what it purports to measure. Construct validity: Establishing relation of latent construct to manifest variables. Also known as “factorial validity.” Predictive validity: Predicting a criterion external to the measurement instrument. Also known as “criterion-related validity.” Content validity: Adequacy with which specified domain of content is sampled. Also known as “relevance” or “representativeness.”

6. Construct validity 1. Specifying domain of observables related to the construct. 2. Determining the extent to which observables measure the same thing (Campbell & Fiske, 1959). 3. Determine the extent to which supposed measures of the construct are consistent with theory and predictions about the construct (Cronbach & Meehl, 1955).

7. Extent to which observables measure the same thing: Multitrait-Multimethod matrix

8. Multitrait-Multimethod matrix Reliabilities

9. Multitrait-Multimethod matrix Heterotrait- monomethod

10. Multitrait-Multimethod matrix Monotrait- heteromethod

11. Multitrait-Multimethod matrix Heterotrait- heteromethod

12. Desiderata • Monotrait-heteromethod correlations should be sufficiently large to warrant further examination (convergent validity). • Monotrait-heteromethod correlations should be larger than heterotrait-heteromethod correlations in same column and row (discriminant validity). • For a given variable, monotrait-heteromethod correlations should be larger than heterotrait-monomethod correlations (discriminant validity). • Complicated—refer to article.

13. How well the measure fits into a network of expected relationships: Nomological net.

14. Predictive validity Important when tests are used to make decisions about people—for example, in educational selection. Orientation is empirical rather than theoretical—for example, the case of motorcycle accidents. Relation between predictor and criterion is called “validity coefficient.” Problems: • Confusion between criterion and predictor. • Selection of an appropriate criterion. • Bias in measurement of criterion

15. Content validity Relates to the adequacy of sampling from relevant domain—for example, school achievement tests. Face validity is one limited, post-hoc aspect of content validity.