Measurement

1. Measurement

2. What is measurement? “the assignment of a value on a variable to a unit of measurement in accordance with an operational definition” (Kleinnijenhuis 1999: 83 in Pennings et. al. 1999)

3. Components of measurement • A value: a categorization or number (e.g. whether or not a country is a democracy or not; or level of democratization) • A variable: a characteristic that can vary in value among observations • A unit of measurement: an observation/case (e.g. a country in a particular year) • An operational definition: a definition of a concept that facilitates the assignment of a value to an observation (e.g. what countries would be classified as democracies)

4. From concept to operational definition • Reduce abstraction to facilitate measurement • E.g. Political party positions / electoral appeals • Concept: the salience of themes to political parties • Operational definition: the percentage of a manifesto devoted to each of a set of defined themes (including “free enterprise”, “welfare state expansion”, “expansion of military capabilities”)

5. Levels of measurement • Nominal • Categories with no rankings (e.g. types of democracies / types of political parties / ethnic groups) - MECE • Ordinal • Some observations score higher/lower than others, but we do not know how much more or less (e.g. Left-Right position of parties / intensity of attachment to ethnic group) • Interval • The differences between the values of the observations are meaningful • Ratio • There is a meaningful zero point (e.g. numbers of decisions taken by referenda/ cross border trade)

6. Guidelines for constructing measures • Reveal assumptions implicit in operationalising a concept: often nothing intrinsic in a concept that implies a particular level of measurement • Maximise information: Wherever sensible, aim for as high a level of measurement as possible • Address trade-off between comparability and descriptive richness

7. Lijphart’s (1977) typology of democratic systems

8. Evaluating measurement • Validity and reliability • Valid: the measure does measure the concept it purports to measure? • Face validity • Correlational validity (internal validity) • Predictive validity (external validity)

9. An example of a correlational validity test • Compared key-informants judgements on “what were the controversial issues” in a particular decision situation with documentation • Key informants identified five controversial issues in decision making on a legislative proposal in the EU (the tobacco directive) • To what extent and in what way were these related to the controversial issues identified through analysis of documentation? For details see: http://www.dhv-speyer.de/tkoenig/DEU_manuscript_Oct2005.htm

10. E.g. of correlational validity test cont. 31 controversial issues on the basis of documentation 5 issues from the informants How many of the 31 were related to the 5? What distinguished document-sourced issues that were related to informants’ issues from those that were not?

11. Reliability • Reliable: repeated measures on the same cases give the same results • Intra observer reliability • Inter observer reliability

12. Another example of testing validity and reliability • The enactment of election pledges in a coalition system of government: The Netherlands (Thomson, R. 2001. The programme to policy linkage: The fulfilment of election pledges on socio-economic policy in the Netherlands, 1986-1998. European Journal of Political Research 40: 171-197)

14. Face validity • Do the election pledges appear to be substantively important? • Do the patterns of fulfilment match with qualitative descriptions of the main policy events in the period?

15. Face validiity

16. Reliability tests • Inter-coder reliability • Did other readers of the same election manifestos identify the same statements as pledges? • Inter-coder reliability • Did subject-area specialists identify the same pledges as unfilfilled or fulfilled?

17. An inter-coder reliability test

18. Measurement error • Aim at unbiased and efficient measures • Bias: systematic error • We either overestimate or underestimate the true value • Inefficiency: Non-systematic error • On average, we estimate the true value correctly, but with considerable variation

19. The ideal is unbiased and efficient estimates Measure/ estimate True value Unbiased, but inefficient Unbiased and efficient Biased

20. The impact of bias on inference • Bias / systematic error • Consistent over or underestimation of the true value • E.g. consistent overestimation of annual income • Leads to bias in description • If it affects all cases equally, does not lead to bias in estimating causal effects

21. The impact of inefficiency on inference • Hampers description, although descriptions are correct on average • Hampers explanation, but the impact differs depending on whether inefficiency pertains to the IV or DV • Inefficiently measured DV – estimates of causal effects will be difficult, but correct on average (unbiased) • Inefficiently measured IV – estimates of causal effects will be difficult and incorrect on average (i.e. biased)

22. Why inefficiently measured DVs do not bias estimates of causal effects DV: violence IV: unemployment

23. Why inefficiently measured IVs bias estimates of causal effects DV: violence IV: unemployment