General Issues in Survey Design

1. General Issues in Survey Design Dr Desmond Thomas, MA TESOL University of Essex

2. Useful References • Denscombe, M., 1998, The Good Research Guide for Small-scale Social Research Projects, Open U. Press • Foddy, W. 1993, Constructing Questions for Interviews and Questionnaires - Theory and Practice in Social Research, Cambridge: Cambridge University Press • Oppenheim, A.N., 1992, Questionnaire Design, Interviewing and Attitude Measurement, London: Pinter

3. The quantitative-qualitative divide • Q-Q Divide: Numbers vs words • Traditionally some methodologies (inc surveys) labelled ‘quantitative’ while others (eg ethnography) are considered qualitative • Methods are divided similarly eg structured questionnaires (quantitative) • This view of research is now challenged

4. Descriptive vs analytic • A more useful distinction for surveys • Descriptive surveys count – entire populations (as in a census) or representative samples • Analytical surveys explore relationships between different variables. These can be causal but are more likely to explore associations. Often less emphasis on representativeness. Linking ideas is the aim.

5. Questionnaire or interview? What’s the difference? • Not a great deal, when questionnaire items are read out and responses filled in by the researcher • But a world of difference between an anonymous structured survey questionnaire and an unstructured 1-to-1 interview • QUESTIONNAIRE OR INTERVIEW: WHICH SUITS YOUR RESEARCH? • And what kind of interview?

6. A research questionnaire should … • Be designed to collect information which can be used as data for analysis: this can be quantitative, qualitative or a mix • Gather information by asking people about issues directly related to a research project (demonstrating validity) • Consist of a written list of questions with each respondent reading an identical set and following identical procedures (reliability)

7. Questionnaires are useful when … • Used with large numbers of respondents in multiple locations • Information required is straightforward • Standardized data from identical questions is required

8. Questionnaire issues: Sampling • Is a sample representative of a total population really necessary? • Or can the sample in a qualitative survey represent itself? • If a representative sample is needed are there ready-made sampling frames? • Will sampling be random or in clusters? • How important is sampling size?

9. Questionnaire issues: Piloting • What should be piloted? Everything! • Questionnaire layout, length, question types, question wording, order of questions, rubrics all need to be tested • It is impossible to get things right at a first attempt; second attempts, in turn, will need to be piloted

10. Advantages of questionnaires • Supply a large quantity of data for a relatively low cost: not labour-intensive • Standardized pre-coded answers can enable speedy data collection, management and even analysis • Eliminate ‘interviewer bias’ – at least to a certain extent • Face validity

11. Questionnaire problem areas • Low response rates • Frustration for the respondent: box-ticking can deter respondents when no answer seems appropriate • Frustration for the researcher: no scope for clarification of answers • One chance only to ‘get it right’ – especially in terms of item wording

12. Creating web-based questionnaires • Go to: http://www.surveymonkey.com/ • Design and edit your survey • Send the link by email to potential respondents • Collect and analyse the data • Note: limitations of the free vs the paid version of this software

13. Types of 1-to-1 interviews • Structured: with tight control over question format and possible answers. Like a face-to-face questionnaire • Semi-structured: with a clearly-defined question schedule but some flexibility and more open-ended answers • Unstructured: a narrative prompted by one general question, perhaps

14. 1-to-1 Interviews are useful when … • Detailed information is needed from respondents • A smaller number of respondents is acceptable • Attitudes or feelings are investigated • Sensitive issues are explored (?) • ‘Key players’ are targeted • Ideas for a questionnaire need to be explored or ‘fine-tuned’

15. Advantages of 1-to-1 interviews • Depth of information • Insights eg gained from key informants • Flexibility of formats • Validity: direct contact means that data can be checked for accuracy and for relevance • High response rate • Opportunity for targeted individuals to make their voice heard – element of advocacy

16. Interview problem areas • The ‘interviewer effect’ and power relationships: face-to-face contact can directly influence answers • The ‘instant position’ effect: respondents feel obliged to supply answers of some sort • Investment of time and resources • Complexity of data analysis • Reliability issues for multiple interviews

17. Reliability issues • How can we know if respondents consistently understand the question? Or if they have misinterpreted it? • How can we know if respondents are consistently telling the truth? Or if they are misleading the interviewer deliberately or at a subconscious level?

18. Features of interview design • How many questions? • In what order? • How long/short? • Format? (open qqs, statements etc.) • Follow-up probes and/or prompts? • Wording of questions? • Mode of recording/analysing answers?

19. Conducting an interview • Establishing a relationship/trust • Explaining aims & procedures • Low key presentation of self and involvement • Active listening • Clarifications, probes and prompts • Recording, transcribing and analysing

20. Focus groups • Consist of a small group of people who are brought together by a trained ‘moderator’ (usually the researcher) to explore attitudes, feelings and ideas about a particular topic or set of issues

21. Main characteristics of FGs • A question schedule is prepared by the moderator to focus the discussion • Place value on interaction within the group as a means of eliciting information – a social experience • No requirement to reach consensus • Less pressure to hold a fixed point of view – in fact opinions can develop

22. FGs are useful when … • Topics or issues need to be explored in depth, or attitudes revealed • Typical groups of key informants are to be targeted eg language teachers

23. Advantages of FGs • A less directive interview format • ‘Interviewer bias’ can be greatly reduced • Provide a more natural social forum for the exchange of ideas. Interviewees are able to take the initiative – not just respond • The right to silence

24. FG problem areas • Less directive means less predictable • Huge amounts of data can be collected, much of which may be unusable • Perception that FGs are a market research tool (Tony Blair & New Labour)