Measurement

1. Measurement V. Measurement: (October 26, 28) • Frankfort-Nachmias & Nachmias (Chapter 7 – Measurement) • Carmines, Edward G. and Richard A. Zeller. 1979. Reliability and Validity Assessment. Newbury Park, CA.:Sage Publications. (pp. 9-48) • King, Keohane and Verba (Chapter 5, Introduction and Section 5.1) Applications • Zachary Elkins, “Gradations of Deocracy? Empirical Tests of Alternative Conceptualizations.” American Journal of Political Science, Vol. 44, No. 2. (Apr., 2000), pp. 293-300. • Edward N. Muller; Mitchell A. Seligson, “Inequality and Insurgency.” The American Political Science Review, Vol. 81, No. 2. (Jun., 1987), pp. 425-452. • Charles D. Brockett, “Measuring Political Violence and Land Inequality in Central America.”The American Political Science Review, Vol. 86, No. 1. (Mar., 1992), pp. 169-176.

2. Measurement • Definitions: • Assignment of “numerals…to empirical properties (variables) according to a prescribed set of rules” (FN-N) • The “process of linking abstract concepts to empirical indicants.” • Conceptual definition vs. operational definition • Concepts vs. indicators

3. Examples • Hypothesis: The level of democracy in a country is negatively related to the level of political violence • Operational Hypothesis: The turnout level in a country is negatively related to the number of riots experienced in that country

4. Operational Definitions • Turnout – The percentage of eligible citizens voting in the last national election • Riots – any act of spontaneous collective violence (rock throwing, vandalism, arson, sniping, beatings) involving 30 or more people that can be interpreted as politically motivated

5. Levels of Measurement • Orderable (Ordinal, Interval, Ratio) vs. Non-orderable Variables (Nominal) • Orderable Variable: Values of an orderable variable represent quantities of that variable, such that any value of a variable X must be either >, <, or = to any other value of X

6. Ordinal Variables • the quantity of X represented by |X1 - X2| is not • known to be equal to the quantity of X represented by |X2-X3| X X1 X2 X3 Example: Olympic Performance (Bronze, Silver, Gold)

7. Interval Variables • the quantity of X represented by |X1 - X2| IS • equal to the quantity of X represented by |X2-X3| X X1 X2 X3 Example: Temperature

8. Ratio Variables • Ratio: A variable measured at the ratio level is simply an interval level variable with a "true" zero (i.e. where a value of zero represents a complete absence of that variable) • Example: Prize money for a golf tournament (in thousands of $)

9. Nominal Variables • Nominal – Classification of observations into a set of categories that do not have direction (i.e. do not represent quantities of that variable) • Example: Race, where 1= White, 2=Black, 3=Asian, 4=Native American

10. Examples • Determine whether the following variables are measured at the nominal, ordinal, interval or ratio levels. • Percentage of state population that is Latino • Year of birth • Party of Governor (Republican, Democrat, Independent) • Weight of contestant • Size of shirt (small, medium, large, XL)

11. Measurement Quality • Measurement Error – Lack of correspondence between observed indicator and underlying concept

12. Validity • Validity - The extent to which a measurement procedure measures what it intends to measure • Running example – The GRE as a measure of graduate school potential

13. Content Validity • Face validity – does the measure appear valid? • Sampling validity – does the measure encompass the entire domain of the concept?

14. Criterion-Related Validity (Empirical Validity) • Is this measure related to other variables (external to the measurement instrument) that are known (or agreed) to be valid measures of the concept of interest? • Predictive • Concurrent

15. Construct Validity • The extent to which a particular measure relates to other measures consistent with theoretically derived hypotheses concerning the concepts that are being measured (C&Z, 23)

16. Examples • Content validity • Measuring “black insurgency”

17. Examples • Criterion-related Validity • Measuring party ideology • Industrialized democracies, Post-war period • Party manifesto data

18. Examples • Criterion-related Validity • Measuring party ideology

19. Examples • Criterion-related Validity • Measuring voter ideology • Using party ideology data and party electoral shares from elections

20. Examples • Criterion-related Validity • Measuring voter ideology

21. Examples • Construct Validity • Measuring voter ideology • How could we test for construct validity?

22. Examples • Construct Validity • Measuring voter ideology • Correlation with policy liberalism

23. Examples • Construct Validity • Measuring citizen ideology (Berry et al.) • Replications of other studies

24. Class Exercise • Measurement Example: An individual’s ideology (liberal-conservative) as measured by the following question: • “Do you support a woman’s right to have an abortion?” Is this measure valid? (Content, Criterion-related, Construct)

25. Class Exercise • Measurement Example: Political participation in a country measured as the percentage of eligible voters who voted in the most recent election. Is this measure valid? (Content, Criterion-related, Construct)

26. Class Exercise • Construct a better measure of political participation which maximizes content validity.

27. Strengths/Weaknesses • Content Validity • Criterion-related Validity • Construct Validity

28. Reliability • Reliability – the extent to which a measuring instrument consistently measures whatever it is that it is measuring

29. Validity vs. Reliability • The relationship between a concept (t) and an indicator (X) can be represented as: X = t + e where: X = indicator of concept t = true score for concept e = measurement error

30. Validity vs. Reliability • The relationship between a concept and an indicator (X) can be represented as: X = t + e X is perfectly valid if e = 0

31. Validity vs. Reliability • The relationship between a concept and an indicator (X) can be represented as: X = t + e X is perfectly reliable if var(t) / var(X) = 1 (and therefore if e is constant) Lesson #1: A perfectly valid measure is perfectly reliable. But a perfectly reliable measure NEED NOT be perfectly reliable.

32. Random vs. Nonrandom Measurement Error • Consider a variable measured with error (i.e. e ≠ 0) X = t + e X is measured with purelyrandom measurement error if e has a mean of 0 and e is a “random variable” (see C&Z, 30) This implies: E(X) = E(t) Lesson #2: Measurement error is relatively benign IF the error is random AND the magnitude of the error is not large.

33. Random vs. Nonrandom Measurement Error • Consider a variable measured with error (i.e. e ≠ 0) X = t + e X is measured with nonrandom measurement error if the mean of e is not equal to 0 This implies: E(X) ≠ E(t)

34. Random vs. Nonrandom Measurement Error • Consider a variable measured with error (i.e. e ≠ 0) X = t + e Two potential situations: • Nonrandom measurement with perfect reliability (e is constant) • Nonrandom measurement with imperfect reliability (e varies AND the E(e) ≠ 0) Lesson #3: The second situation is the most harmful.

35. Measurement Quality - Reliability • Determine if the following measures are likely to be reliable: • Individual ideology as measured on a seven point scale (survey) • Gender of an individual (based on interviewer observation in face-to-face survey) • Heightof an individual as a measure of ideology

36. Assessing Reliability through Parallel Measurements • Test – Retest Method • identical measure used twice on all units • Alternative Forms • Comparable (but not identical) indicator used on all units • Split-Halves (for a multi-item measure) • Half of the items used for one half of the sample, the other half of the items used for the other half of the sample

37. Improving Measurement Quality • Creating multi-item measures • Reduce errors in measurement that occur by using a single variable Types - Index - Scale

38. Original CPS Political Efficacy Index [agree-disagree] 1. People like me don't have any say about what the government does. 2. Sometimes politics and government seem so complicated that a person like me can't really understand what's gong on. 3. Voting is the only way that people like me can have any say about how the government runs things. 4. I don't think public officials care much what people like me think. 5. Generally speaking, those we elect to Congress in Washington lose touch with the people pretty quickly. 6. Parties are only interested in people's votes but not in their opinions.