Questionnaire Design: Reading material

1. DTC Quantitative Research Methods Questionnaire Design and Scale ConstructionThursday 30th October 2014

2. Questionnaire Design: Reading material • In addition to the online course extract from Aldridge and Levine (2001), there are useful chapters in Oppenheim (1992) and De Vaus (2001). Remember • While the research instrument in ‘face-to-face’ surveys is arguably more correctly referred to as the ‘interview schedule’, the literature often incorporates such instruments under the heading ‘questionnaire design’.

3. The four stages of questionnaire design (i) “Mapping the semantic domain” This stage involves identifying for each of the concepts fundamental to the proposed research its components (i.e. sub-concepts), context (i.e. related and ‘similar but different’ concepts), and parameters (i.e. the conditions which must be present for the concept to apply). (ii) Formulating the question ideas This stage involves identifying the range of issues on which specific questions need to be asked, i.e. it identifies that a question has to be asked relating to a particular sub-concept: “We need to ask a question about...”. At this stage one is moving from concepts towards indicators. (iii) Writing the questions At this stage the specific wording of the questions is determined (iv) Assembling the questionnaire At this stage the questions are assembled into a coherent, structured questionnaire. Source: Halfpenny, P., Parthemore, J., Taylor, J. and Wilson, I. 1992. ‘A knowledge based system to provide intelligent support for writing questionnaires’. In Westlake, A. et al. (eds) Survey and Statistical Computing. London: North Holland.

4. Four aspects of a questionnaire’s contents De Vaus refers to (i) Measures of the dependent variables (ii) Measures of the independent variables (iii) Measures of “test” variables (i.e. variables that intervene between the independent and dependent variables, or which are temporally prior and possibly causally related to both) (iv) Background measures (i.e. ‘demographic’ variables)

5. Preliminaries • Unstructured discussions with potential respondents/‘key informants’. The idea of this qualitative work is to give the researcher a more complete picture of the topic, and the language and perspectives of respondents. • A search for existing questions on the topic, (e.g. via the ‘Question Bank’). However, inheriting measures from previous research carries a risk of reflecting the pre-conceptions of earlier researchers. • Piloting the draft questionnaire, to check for problems and to allow questions to be refined.

6. Some general points • “Avoid putting ideas into the respondent’s mind early in the interview if we need spontaneous responses later on” (Oppenheim). • Beware of asking questions that unwittingly reveal the researcher’s attitude to the topic. • “A question that strikes the respondent as rude or inconsiderate may affect not only his [sic] reply to that particular question but also his [sic] attitude to the next few questions and to the survey as a whole” (Oppenheim) • Make the questionnaire attractive to the respondent by making it interesting and of obvious relevance to its stated purpose.

7. A framework for question sequences Oppenheim outs forward the following, chronological framework (derived from a Gallup schema): • Respondent’s awareness of issue. • Respondent’s general feelings about issue. • Respondent’s views on specific aspects of issue. • Reasons for respondent’s views. • Strength of respondent’s views.

8. ‘Open’ or ‘Closed’ I? • ‘Open’ questions, where the respondent’s verbatim answer to the question is recorded in full, are easy to ask, less easy to answer and difficult to analyse. The emphasis is on the respondent’s perspective, but there is still the possibility that answers will reflect what is uppermost in respondents’ minds. • ‘Closed’ questions are easier to quantify, but result in a loss of spontaneity and expressiveness, and the ‘forced’ choice of answers may result in bias (and shift the balance towards the researcher’s perspective). •  A compromise is to ask an ‘Open’ question and then a similar ‘Closed’ question later in the questionnaire/interview. •  Oppenheim comments that “All closed questions should start their careers as open ones, except those where certain alternatives are the only ones possible”.

9. ‘Open’ or ‘closed’ II? • Awareness of issue  Closed, • General feelings on issue  Open, • Views on specific aspects of issue  Closed, • Reasons for views on issue  Open or Closed, • Strength of views on issue  Closed. Source: De Vaus (1986), adapted from Gallup.

10. Types of ‘forced choice’ answers • A Likert scale: e.g. a range of answers from ‘Strongly agree’ through to ‘Strongly disagree’. • A semantic differential scale: A range of positions between two extremes of a continuum (e.g. ‘Caring’ through to ‘Uncaring’). • A checklist: e.g. a list of leisure activities. • A ranking of items: e.g. placing the most important attributes of a potential partner in order. • A choice between statements: e.g. a choice of responses to the acquisition of the knowledge that one’s best friend’s partner is being unfaithful to them.

11. Some key issues Some ‘crunch’ questions that one might ask: • Does the respondent understand the question (in the same way the researcher does!), • Are they willing to answer it (accurately?), and • Are they able to answer it?

12. Scale construction • Researchers sometimes want to measure some latent characteristic of their respondents, e.g. whether they have a ‘traditional’ or a ‘modern’ viewpoint on couple relationships. • This is often done by asking a number of questions which each tap that characteristic and, when aggregated, collectively do so in more reliable way.

13. Some issues... • How does one know that a measure constructed by aggregating various items to give a scale is measuring the ‘right’ quantity, i.e. is a valid measure? • How does one ensure that what a measure is measuring is unidimensional, i.e. that it is not a composite of measures of two or more underlying concepts? • How does one assess which items need to be included to maximise the reliability of the measure? • And how does one assess the overall reliability of the scale?

14. Some answers... • For a discussion of assessing various forms of validity, see Oppenheim (1992). • Unidimensionality can be assessed using a technique called factor analysis, see DeVellis (2003). • Reliability can be assessed using a measure called Cronbach’s alpha (see De Vaus, 2001; DeVellis, 2003).

15. Factor analysis Factor analysis generates a set of underlying factors which successively maximise the amount of (remaining) variation in the items that they can explain. If a scale is working properly unidimensionally, then the first factor will explain a high proportion of the variation, and the subsequent factors similar, small amounts.

16. Cronbach’s alpha • According to DeVellis (2003: 95), “Alpha is an indication of the proportion of variance in the scale scores that is attributable to the true score”. • Items are chosen for inclusion so as to maximise that proportion, and if they have relatively high correlations with the rest of the items within the scale (viewed collectively). • De Vaus (2001) suggests that a value of at least 0.7 is preferable.

17. Some additional issues... • Should all the items in the scale be treated as of equal importance? Or should their values be added in such a way as to increase/ decrease the relative importance of some items? • Are the gaps between the values that a variable can take uniform in meaning?