Field experiments for assessing question validity

1. Field experiments for assessing question validity   Patrick Sturgis, Department of Sociology, University of Surrey, UK Paper presented at conference on ‘Survey Measurement: Assessing the Reliability and Validity of Contemporary Questionnaire Items’ The Royal Statistical Society,10 April 2008

2. Plan of Talk • Standard validity assessment for survey questions • Field experiments • Example 1 – Political knowledge • Example 2 – Social trust • Concluding remarks

3. Standard validity assessment • Nothing • Face/process validity • Correlation with criterion variables • Multi-trait-multi-method (MTMM) • Expert panels • Behaviour coding • Interviewer debrief • Thinkaloud protocols/cognitive interview

4. Limitations • Small n/purposive selection – do inferences generalize? • Do different techniques/researchers identify same ‘problems’ • Do modifications increase validity? • Paradoxical limitations for survey research!

5. Field Experiments • Large n with randomization of alternate forms • Clean and powerful inference • Lack of criterion reference can be problematic • But theory can help!

6. Example 1(with Nick Allum, Patten Smith) Measuring Political Knowledge: Guessing and partial knowledge

7. Standard approach • MCQ format: • “The Number of MPs in Parliament is about 100” • True • False • DK • DKs ‘encouraged’ • Two key problems (Mondak 2001; 2002): • Some say DK when they can answer correctly at p > 0.5 (partial knowledge) • Some provide a substantive answer when they cannot answer correctly at p >0.5 (guessing)

8. Personality Variance • Variation in knowledge scores reflects more than just knowledge • Men more likely to guess in absence of knowledge • Women more likely to say DK with partial knowledge • Thus, men ‘appear’ to know more about politics than women

9. The Solution? • Force all respondents to provide an answer even if they genuinely DK (Mondak 2001) • Randomly allocate residual DKs across substantive categories • Removes personality variance by omitting option of guessing • And of saying DK in presence of partial knowledge

10. Study 1 - Partial Knowledge • BMRB CATI omnibus (quota sample) • Interviewing 17-19 December 2004 • N = 1006 • Three true/false knowledge items: • Britain's electoral system is based on proportional representation • MPs from different parties are on parliamentary committees • The Conservatives are opposed to the ratification of a constitution for the European Union

11. Design • “For the next few questions, I am going to read out some statements, and for each one, please tell me if it is true or false. If you don't know, just say so and we will skip to the next one” • If respondent answers DK: • “You said earlier that you don't know whether the number of MPs is about 100. Could you please just give me your best guess?” • Partial knowledge in initial DK responses if % correct after probe > .5

12. Probed DK Responses

13. Results Binary logit predicting correct answer (0,1) • Model predicted probabilities of correct answers = • 71% for those giving an initial response • 53% for probed DKs • 50% for random allocation and No gender difference

14. Study 2 - Guessing • Ask standard format knowledge questions but where answer options are all wrong • Respondents choosing any substantive alternative are ‘guessing’: • Who is the Secretary of State for Trade and Industry? Is it, a. Geoff Hoon b. Peter Hain or c. Do you not know? (correct=Alan Johnson) • BMRB omnibus n=2011, 4-6 November 2005 and 9-11 December 2005

15. Who is the Secretary of State for Trade and Industry? Is it, a. Geoff Hoonb. Alan Johnson, orc. Do you not know?

16. Who is the Secretary of State for Trade and Industry? Is it, a. Geoff Hoonb. Peter Hain, orc. Do you not know?

17. Binary Logit Model Dependent variable guess=1, dk=0

18. Conclusions • No evidence that DKs in survey knowledge items conceal partial knowledge • Guessing, however, is common and differential (favouring men) • Guessing also related to political knowledge • Recommendation: use ‘standard’ format items • For marginal comparisons, randomly allocate DKs to substantive categories • For associational relationships use number right scoring (treat DK and incorrect as equivalent)

19. Example 2(with Patten Smith) Investigating Social Trust Using thinkalouds

20. Conceptions of Trust • Trust is a ‘good thing’ • Trusting citizens are good citizens (voting, volunteering, civic engagement) • Trusting societies are good societies (more democratic, egalitarian, > economic performance) • Trust ‘lubricates’ social and economic transactions • Reduces ‘monitoring costs’

21. ‘Thick’ Trust • Also ‘particularized’ or ‘strategic’ trust • Between people who know one another • Based on personal experience • Encapsulated interests; ‘your interests are my interests’ (Hardin) • I trust x to do y

22. ‘Thin’ trust • Also ‘social’ or ‘generalized’ trust • Trust between people not personally known to one another • More akin to a core social value or attitude • “an evaluation of the moral standards of the society in which we live” (Newton) • A ‘default position’ in transactions with unknown others

23. Does this matter? • Primary social and individual returns are to thin/social trust • Thick and thin trust may even be negatively correlated • The less we trust people in general, the more we retreat to the safety of those we know • So, empirically distinct measures are clearly essential

24. The standard trust question • Generally speaking, would you say that most people can be trusted, or that you can't be too careful in dealing with people? • Most people can be trusted • Can’t be too careful • Usually credited to Rosenberg (1959), the ‘Rosenberg Generalized Trust’ (RGT) item

25. The Local Area Trust item • How much do you trust people in your local area? • a lot • a fair amount • not very much • not at all • Reflects Putnam’s emphasis on trust being a property of local areas

26. Trust by Question type • These items are both used more or less interchangeably as measures of generalized trust • Yet, they yield very different estimates of trust at the national level. e.g.: • Social Capital Community Benchmark survey: 47% most people can be trusted; 83% trust people in local area ‘some’ or ‘a lot’ • UK Taking Part survey: 44% most people can be trusted; 74% trust ‘many’ or ‘some’ of the people in their local area • Why such a large discrepancy in generalized trust (trust in strangers)?

27. Research Design • Ipsos-MORI general population omnibus survey • Random selection of small areas, quota controlled selection of individuals • n=989 (fieldwork, November 2007) • Respondents randomly assigned to RGT or TLA item • In answering the last question, who came to mind when you were thinking about ‘most people’/ ‘people in your local area’?

28. Distributions for trust questions

30. Who comes to mind by RGT

31. Who comes to mind by TLA

32. Who came to mind – both questions

33. Explanatory Models 1

34. Explanatory Models 2

35. Concluding Remarks • Large-scale field experiments are a useful way of assessing validity of questions • Random sample + random manipulation yields strong inferential power • Under-utilized due to cost considerations • But are they really so costly? • A complement to rather than replacement for small n approaches

36. Papers • Sturgis, P. Allum, N. & Smith, P. (2008) The Measurement of Political Knowledge in Surveys Public Opinion Quarterly 72,90-102. • Sturgis, P. and Smith, P. (2008) Assessing the Validity of Generalized Trust Questions: What kind of trust are we measuring? Paper presented at the ‘Conference on Composite Scores’ ESADE, Barcelona, 14-15 February 2008. • Sturgis, P. and Smith, P. (2007) Fictitious Issues Revisited: political knowledge, interest, and the generation of nonattitudes. (under review). • Sturgis, P., Choo, M. & Smith, P. (2007) Response Order, Party Choice, and Evaluations of the National Economy: A Survey Experiment. Survey Research Methods (in press).