Reliability and Validity

1. NTR 629 – Week 5 Reliability and Validity

2. Reliability (Precision) • The ability of a measuring instrument to give consistent results on repeated trials. • Repeated measurement reliability--the degree of consistency among repeated measurements of the same individual on more than one occasion. • There are degrees of reliability. • For example one can perform triceps fat-fold measures with great reliability in the incorrect place on the arm. This target illustrates good reliability with the darts hitting nearly the same place. Reliability has to do with the consistency or repeatability.

3. Types of Reliability (Tests) • Test-Retest • Measure of stability; consistency over time (r) • Parallel Forms • Measure of equivalence (r) • Inter-Rater • Measure of agreement (% agreement) • Internal Consistency • Measure of consistency across items within the same construct (e.g., Cronbach’s alpha)

4. Increase number of items (e.g., on test, survey) Eliminate unclear items Standardize conditions Moderate the easiness and difficulty of tests Minimize effects of external events Standardize instructions Maintain consistent scoring procedures (e.g., rubric for essay or performance) Ways to Increase Reliability

5. Validity (Accuracy) • The ability of a measuring instrument to give a true measure. • In other words, does it measure what it was intended to measure? • There are degrees of validity (not yes/no). • Can be evaluated only if an accepted and independent method for confirming the test measurement exists. Since there is good reliability (consistency), we have the potential for strong validity; must be reliable to be valid. If the center of the target represents the ‘truth’ (or true measure), this target illustrates high validity.

6. Types of Validity • Content • Content: How well do the questions represent an adequate sample of the potential items in the domain (comprehensive)? (Ask an expert) • OR Content validity--the degree to which a measure covers the range of meanings included within the concept. • Format is an aspect of content validation: • Is the language clear and appropriate? • Is the type size appropriate? • Does the format provide adequate work space (if needed), and clear directions? • Esp. for achievement tests

7. Types of Validity con’t • Criterion • Compare the scores obtained on the test (the one being validated) with the scores obtained using one or more other external instruments measuring the same variable (acts as the standard of comparison - criterion). Examine relationships. • Can be concurrent - How well does the test estimate a behavior or performance? • Can be predictive - how well does it predict a criterion? (e.g., Do RD scores predict future performance?) • Esp. for intelligence or scholastic aptitude tests

8. Types of Validity con’t • Construct • How well does the test assess an underlying theory or model? Is the evidence obtained consistent with theoretical explanations. • OR Construct Validity--degree to which the measurement agrees with the theoretical concept being investigated. • Esp. for development of a psychological or attitude inventory, or determination of items in a category. • Threats to construct validity: http://www.socialresearchmethods.net/kb/consthre.php • Face (provides only weak evidence) • Does the test appear to measure what it claims to measure? (“Ask your mom”)

9. Threats to Internal Validity • History • Maturation (biological or psychological forces) • Testing (e.g., equipment not calibrated) • Instrumentation • (e.g., assessment, interview technique, carry-over effects) • Statistical Regression • (i.e., movement towards the mean) • Selection (if no randomization) • Mortality • Imitation of Treatment http://www.suu.edu/faculty/white_l/research%20design/chapter%20notes/chapter%208.pdf (review for more details)

10. Careful planning and implementation of design. Develop operational definitions. Follow a standard study design Control group Randomization Match subjects if needed ANCOVA – analysis of covariance statistical technique Search for discrepant evidence and negative cases Triangulation by using multiple methods “Member checks” by soliciting feedback from participation about data and conclusions. Ways to Increase Validity If too tightly controlled (internal validity) – it may reduce generalizability (external validity) BUT must have internal validity (external not as essential)

11. External Validity • Generalizability • Population generalizability • Degree to which a sample represents the population of interest • Purposeful or convenience sampling can never guarantee representativeness on all relevant variables. • Loss of over 10% of original sample should qualify conclusions. • Ecological or environmental generalizability • Degree to which the results can be extended to other settings or conditions. • Temporal generalizability • Degree to which the results can apply to all times and not just the time period (e.g., season, year) of the study

12. Threats to External Validity • Pretest Sensitization • the pretesting informs people of what is to come • Interaction of Selection and Treatment • Study is too unique that findings are not generalizable • Multiple Treatment Interference • Subject attends support group plus education • Reactive Arrangements • Hawthorne Effect • Experimenter Effects • Different methods of presentation/procedure • Bias (Racial, Gender, Cultural)