Understanding Reliability and Validity in Measurement

1. Reliability & Validity Reliability “dependability” is the indicator consistent? same result every time? Does not necessary measure what you think it measures (ex. May consistently measure something other than the concept) Validity measurement validity - how well the conceptual and operational definitions mesh with each other does measurement tool measure what we think ?

2. Relationship between Measurement Reliability and Validity reliability necessary for validity but does not guarantee it (“necessary but not sufficient”) measure can be reliable but invalid Source: Neuman (2000: 171)

3. Examples of Types of Reliability stability over time representative across different subgroups of a population (ex. of problems: young people may exaggerate their ages, older people may reduce theirs) equivalence Comparable results from multiple indicators intercoder reliability

4. Improving Reliability clearly conceptualize constructs increase level of measurement use pretests, pilot studies e.g. use multiple indicators : A B Empirical Independent Variable Measure Dependent Variable Measure Association? a1 a2 a3 b1 b2 Specific Indicators Specific Indicators Measurement Using Multiple Indicators Neuman (2000: 167)

5. Selected Types of Validity: Face Validity judgement by the scientific community that indicator measures the concept Construct/concept ? Measure Scientific Community

6. Content Validity measure represents all the aspects of conceptual definition of construct. judgement concerning how adequately a test samples behavior representative of the universe of behavior the test was designed to sample. Beliefs & Values Love

7. Criterion The validity of indicator is verified by comparing it with another measure of the same construct Predictive Relies on occurrence of future event or behavior for verification of indicator Concurrent relies on pre-existing accepted measure to verify indicator Construct Validity A type of measurement validity that uses multiple indicators convergent and discriminate Some types of Measurement Validity

8. Other Dimensions of Validity Internal Validity no error of logic internal to research design External Validity results can be generalized Statistical validity correct statistical methodology chosen ? assumptions fully met

9. More terms & concepts for assessing validity (usually in experimental research) Threats to Internal Validity Selection bias History effects (something in context changes) Maturation Testing effect Instrumentation Mortality Statistical regression Contamination Compensatory behaviour Experimenter expectanc Threats to External Validity Realism Reactivity

10. Another Aspect of Measurement Design: Levels of Measurement • 1-Nominal (ex. Mother tongue) • different categories (names, labels, images) • not ranked • 2-Ordinal (county fair prizewinners ranked by first, second & third prize) • different categories • rank-ordered • attributes indicate relatively more or less of that variable • distance between the attributes of a variable is imprecise

11. Levels of Measurement (cont’d) • 3- Interval Measures (age measured by 5 year age groups) • different categories • ranked in order • Can tell amount of difference between categories • Usually no true zero • 4- Ratio Measures (age measured by date of birth) • different categories • ranked in order • amount of difference between categories • also possible to state proportion (have a true zero) • Relations between levels --can collapse from higher into lower, not vice versa

12. The Research Process Babbie (1995: 101)