Validity Lecture Overview

1. ValidityLecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity issues from the literature Discussion of validity issues with respect to student projects

2. Validity Descriptors �Hypothesis� validity �Construct� validity �Content� validity �Convergent� validity �Ecological� validity �Internal� validity �Statistical conclusion�validity �Concurrent� validity �External� validity �Predictive� validity �Criterion-related� validity �Discriminant� validity

3. Taxonomy of Validity Validity as it pertains to assessment Validity as it pertains to causal inference Validity as it pertains to generalization of findings to real-world phenomena

4. Validity Issues Surrounding Assessment/Measurement

5. Validity of an Assessment Tool Validity represents an overall judgment of the degree to which both empirical evidence and theoretical considerations support the interpretation of the score and the implications for action that this interpretation entails (Cronbach, 1971).

6. Validity of an Assessment Tool Score validation is an empirical evaluation of the meaning and consequences of measurement (Messick, 1989).

7. Validity Features Validity applies to all assessments, including performance/behavioral assessments Validity is not a property of the �test� per se, but rather of the meaning of the test score Validation is an ongoing process

8. Features Validity is not just a measurement principle, it is a social value that has powerful implications whenever evaluative judgments and decisions are made

9. Types of �Assessment� Validity Content validity Degree to which �Test� items adequately sample the universe of relevant items for a given domain

10. Types of �Assessment� Validity Criterion-related validity Degree to which a �Test� score relates to some relevant external criterion Concurrent validity Predictive validity

11. Types of �Assessment� Validity Construct validity* Ongoing, integrated summary of the evidence supporting the interpretation and utility of �Test� scores Combines information from content validity, criterion-related validity, and discriminant/convergent validity

12. Convergent/Discriminant Construct Validation Convergent validity Empirical evidence demonstrating communality between the test score and other indicators of the same construct Discriminant validity Empirical evidence demonstrating a lack of communality with the test score and indicators of a different construct In construct validation the test score is not equated with the construct it attempts to tap rather it is viewed as just one indicator of a larger set of indicators of the construct. Empirical evidence demonstrating communality between the test score and other indicators of the same construct is referred to as convergent validity. Likewise, empirical evidence demonstrating a lack of communality with the test score and indicators of a different construct is referred to as discriminant validity.In construct validation the test score is not equated with the construct it attempts to tap rather it is viewed as just one indicator of a larger set of indicators of the construct. Empirical evidence demonstrating communality between the test score and other indicators of the same construct is referred to as convergent validity. Likewise, empirical evidence demonstrating a lack of communality with the test score and indicators of a different construct is referred to as discriminant validity.

13. Threats to Assessment Validity Construct underrepresentation Exists when the assessment fails to include important facets of the construct (i.e., assessment is too narrow) Examples? Examples of Construct Underrepresentation: Behavioral assessment of childhood aggression Tim Rentz� dissertation: are Goldie and Luna sufficiently representative of the domain of dogs Examples of Construct-Irrelevant Variance: Achievement testing: Math performance test that includes word problems that require a reasonable high level of reading proficiency. Claustraphobia studies: Behavioral test is conducted in the dark and thus the test may erroneously score someone as claustrophobic when their distress is due to the irrelevant facet of darkness. Examples of Construct Underrepresentation: Behavioral assessment of childhood aggression Tim Rentz� dissertation: are Goldie and Luna sufficiently representative of the domain of dogs Examples of Construct-Irrelevant Variance: Achievement testing: Math performance test that includes word problems that require a reasonable high level of reading proficiency. Claustraphobia studies: Behavioral test is conducted in the dark and thus the test may erroneously score someone as claustrophobic when their distress is due to the irrelevant facet of darkness.

14. Threats to Assessment Validity Construct-irrelevant variance Exists when the assessment contains reliable variance associated with other distinct constructs (i.e., assessment is too broad) Examples? Examples of Construct Underrepresentation: Behavioral assessment of childhood aggression Tim Rentz� dissertation: are Goldie and Luna sufficiently representative of the domain of dogs Examples of Construct-Irrelevant Variance: Achievement testing: Math performance test that includes word problems that require a reasonable high level of reading proficiency. Claustraphobia studies: Behavioral test is conducted in the dark and thus the test may erroneously score someone as claustrophobic when their distress is due to the irrelevant facet of darkness. IMPORTANT NOTE: Mention that both threats are operative in all assessments to some extent. Hence, a primary concern is the extent to which the assessment might underrepresent the focal construct while simultaneously contaminating the scores with construct-irrelevant variance. Examples of Construct Underrepresentation: Behavioral assessment of childhood aggression Tim Rentz� dissertation: are Goldie and Luna sufficiently representative of the domain of dogs Examples of Construct-Irrelevant Variance: Achievement testing: Math performance test that includes word problems that require a reasonable high level of reading proficiency. Claustraphobia studies: Behavioral test is conducted in the dark and thus the test may erroneously score someone as claustrophobic when their distress is due to the irrelevant facet of darkness. IMPORTANT NOTE: Mention that both threats are operative in all assessments to some extent. Hence, a primary concern is the extent to which the assessment might underrepresent the focal construct while simultaneously contaminating the scores with construct-irrelevant variance.

15. Evidence for Assessment Validity Evidence of content relevance and representativeness The extent to which test scores are consistent with theoretical predictions Evidence examining the extent to which score properties and interpretations generalize to and across groups, settings, and tasks Important Note: Mention concept �Triangulation of a constructImportant Note: Mention concept �Triangulation of a construct

16. Evidence for Assessment Validity Evidence on the fidelity of the scoring structure to the structure of the construct being tapped Evidence from criterion-related studies including convergent and discriminant studies Evidence pertaining to the consequential aspect of test use and score interpretation, especially as it relates to issues of bias, and fairness

17. Strategies for Enhancing Assessment Validity Avoid sole reliance on measures that lack validation data (e.g., new author-constructed measures) Employ multiple indicators of the focal construct whenever possible Employ indicators from more than one assessment modality domain Discussion?

18. Drawing Valid Inferences about Causal Relationships

19. Types of Validity Pertinent to Drawing Causal Inferences Internal validity Degree to which causal inferences can be made between a measured or manipulated variable (i.e. independent variable) and another measured variable (dependent variable)

20. Types of Validity Pertinent to Drawing Causal Inferences Statistical conclusion validity Concerned with sources of random error and with the appropriate use of statistics and statistical tests (as opposed to systematic bias as in the case of internal validity)

21. Types of Validity Pertinent to Drawing Causal Inferences External validity Refers to the degree to which the observed causal relationship is generalizable across persons, settings, and occasions Important distinction between generalizing to a specified population (or setting or occasion) vs. generalizing across populations

22. Types of Validity Pertinent to Drawing Causal Inferences Construct validity* The degree to which causal inferences concerning one variable�s effect on another can be generalized to examplars of the constructs in question In every day practice, this form of validity deals with the issue of �confounds� Distinguishes between the operation (e.g., administering Prozac) and the purported construct hypothesized to cause the effect of the operation (e.g., change in depression). Also give CBT for panic prevention Distinguishes between the operation (e.g., administering Prozac) and the purported construct hypothesized to cause the effect of the operation (e.g., change in depression). Also give CBT for panic prevention

23. Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression

24. Threats to Internal Validity Selection Mortality Interactions with selection Ambiguity about the direction of causal influence

25. Threats to Internal Validity Diffusion of treatments Compensatory equalization of treatments Compensatory rivalry by respondents Resentful demoralization of respondents

26. Threats to Internal Validity Compensatory equalization of treatments Compensatory rivalry among participants Resentful demoralization Mortality

27. Threats to Construct Validity About Cause and Effect Construct underrepresentation Mono-operation bias Mono-method bias Confounding constructs and levels of constructs

28. Threats to Construct Validity About Cause and Effect Construct irrelevancies (i.e., confounds) Interaction of different treatments Hypothesis-guessing within experimental conditions Evaluation apprehension (demand characteristics) Experimenter expectancies Interaction of testing and treatment

29. Threats to External Validity Interaction of selection and treatment Interaction of setting and treatment Interaction of history and treatment

30. Strategies for Enhancing External Validity Employ random sampling to obtain a representative sample if time, resources, and feasibility permit Employ heterogeneous samples whenever possible Conduct analyses to determine whether the causal relationship holds across characteristics of subjects, settings, etc