1 / 34

Questionnaire & Interview Item-writing Part 1

Questionnaire & Interview Item-writing Part 1. Dr Desmond Thomas, MA TESOL University of Essex. Useful References. Denscombe, M., 1998, The Good Research Guide for Small-scale Social Research Projects, Open U. Press

Download Presentation

Questionnaire & Interview Item-writing Part 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Questionnaire & Interview Item-writing Part 1 Dr Desmond Thomas, MA TESOL University of Essex

  2. Useful References • Denscombe, M., 1998, The Good Research Guide for Small-scale Social Research Projects, Open U. Press • Foddy, W. 1993, Constructing Questions for Interviews and Questionnaires - Theory and Practice in Social Research, Cambridge: Cambridge University Press • Oppenheim, A.N., 1992, Questionnaire Design, Interviewing and Attitude Measurement, London: Pinter

  3. Questionnaire or interview? What’s the difference? • Not a great deal, when questionnaire items are read out and responses filled in by the researcher • But a world of difference between an anonymous structured survey questionnaire and an unstructured 1-to-1 interview • QUESTIONNAIRE OR INTERVIEW: WHICH SUITS YOUR RESEARCH? • And what kind of interview?

  4. A research questionnaire should … • Be designed to collect information which can be used as data for analysis: this can be quantitative, qualitative or a mix • Gather information by asking people about issues directly related to a research project (demonstrating validity) • Consist of a written list of questions with each respondent reading an identical set and following identical procedures (reliability)

  5. Questionnaires are useful when … • Used with large numbers of respondents in multiple locations • Information required is straightforward • Standardized data from identical questions is required

  6. Questionnaire issues: Sampling • Is a sample representative of a total population really necessary? • Or can the sample in a qualitative survey represent itself? • If a representative sample is needed are there ready-made sampling frames? • Will sampling be random or in clusters? • How important is sampling size?

  7. Quantitative Survey Issues: Sampling 1 • If the target population is : secondary school teachers of English in Bulgaria • How large a sample? 30 or more people? 5-10%? • Problem of self-selection in survey responses. How to solve this?

  8. Quantitative Survey Issues: Sampling 2 • Sample A: teachers belonging to five different schools (cluster sampling)? • Sample B: randomly selected? • Sample C: randomly selected within large clusters (eg the five schools)? • Sample D: Every 10th teacher on a national register (or other sampling frame)?

  9. Questionnaire issues: Piloting • What should be piloted? Everything! • Questionnaire layout, length, question types, question wording, order of questions, rubrics all need to be tested • It is impossible to get things right at a first attempt; second attempts, in turn, will need to be piloted

  10. Advantages of questionnaires • Supply a large quantity of data for a relatively low cost: not labour-intensive • Standardized pre-coded answers can enable speedy data collection, management and even analysis • Eliminate ‘interviewer bias’ – at least to a certain extent • Face validity

  11. Questionnaire problem areas • Low response rates • Frustration for the respondent: box-ticking can deter respondents when no answer seems appropriate • Frustration for the researcher: no scope for clarification of answers • One chance only to ‘get it right’ – especially in terms of item wording

  12. Creating web-based questionnaires • Go to: http://www.surveymonkey.com/ • Design and edit your survey • Send the link by email to potential respondents • Collect and analyse the data • Note: limitations of the free vs the paid version of this software

  13. Issues for all types of questionnaire • What makes a good questionnaire item? • Why is it so difficult to formulate the precise questions that we need to ask in order to obtain the required answers? • Why is it sometimes difficult to interpret the answers that we obtain?

  14. Fundamental Problem 1 • “It is almost impossible to ask a question without suggesting answers. The very fact that a question is asked implies that the researcher thinks the topic is of interest. Moreover, the way a question is asked inevitably reflects the researcher’s preconceptions. Unstated presuppositions always underlie a question. “ (Foddy 1993: 53-4)

  15. Fundamental Problem 2 Q: Which soft drink do you usually buy? (What is a ‘soft drink’ and what isn’t? How often is ‘usually’? What is understood by ‘buy’ and by ‘you’? What if you buy more than one?) A: Probably the first brand that comes to mind The question may not be understood as it is intended and results may therefore be invalid. (Starting point: what information does it seek to elicit?)

  16. Question questions such as … • How many journeys have you made on London Underground over the past month? • Do you enjoy going to coffee bars and restaurants in the evening? • Do you regularly do your shopping in large supermarkets? • The library facilities at Essex University are adequate for my needs. Strongly agree ( ) Agree ( ) Disagree ( ) Strongly disagree ( )

  17. And these as well … • Complete the following sentence. I enjoy studying at Essex because … 2. Do you think that the library should do more to improve its facilities for Masters students? 3. How important is exercise as part of your daily routine? Very important ( ) Important ( ) Not sure ( ) Unimportant ( )

  18. Framework for questionnaire design

  19. Design principles • Choose the right instrument! • Target information required first → afterwards the right questions to ask. Required information depends on aims. • Rigorously monitor all items • Pilot and re-pilot the questionnaire • Provide clear instructions and standard procedures (for reliability)

  20. Open vs closed questions OPEN: Answers have more depth but data processing is more complex. CLOSED: Easy to process but many useful insights are lost and respondents can become irritated by being put ‘in boxes’. (Oppenheim 1992, p.115)

  21. Some pitfalls to avoid • Respondents often answer questions when they don’t really know the answer. • Respondents can deliberately not answer or answer incorrectly (ie they lie) • Respondents misinterpret questions • Small changes in wording can produce major changes in distribution of responses. • Attitudes and opinions can be unstable; circumstances can change • Memory can be unreliable. • Relationship between what respondents say they do and what they actually do is not strong. • Cultural context has an impact on responses (eg attitude scale grades) • The format itself can affect responses • The order of questions and answers to earlier questions can affect responses.

  22. NEXT WEEK !!! Questionnaire Item-Writing Workshop: Coursebook Feedback Survey

  23. Types of 1-to-1 interviews • Structured: with tight control over question format and possible answers. Like a face-to-face questionnaire • Semi-structured: with a clearly-defined question schedule but some flexibility and more open-ended answers • Unstructured: a narrative prompted by one general question, perhaps

  24. 1-to-1 Interviews are useful when … • Detailed information is needed from respondents • A smaller number of respondents is acceptable • Attitudes or feelings are investigated • Sensitive issues are explored (?) • ‘Key players’ are targeted • Ideas for a questionnaire need to be explored or ‘fine-tuned’

  25. Advantages of 1-to-1 interviews • Depth of information • Insights eg gained from key informants • Flexibility of formats • Validity: direct contact means that data can be checked for accuracy and for relevance • High response rate • Opportunity for targeted individuals to make their voice heard – element of advocacy

  26. Interview problem areas • The ‘interviewer effect’ and power relationships: face-to-face contact can directly influence answers • The ‘instant position’ effect: respondents feel obliged to supply answers of some sort • Investment of time and resources • Complexity of data analysis • Reliability issues for multiple interviews

  27. Reliability issues • How can we know if respondents consistently understand the question? Or if they have misinterpreted it? • How can we know if respondents are consistently telling the truth? Or if they are misleading the interviewer deliberately or at a subconscious level?

  28. Features of interview design • How many questions? • In what order? • How long/short? • Format? (open qqs, statements etc.) • Follow-up probes and/or prompts? • Wording of questions? • Mode of recording/analysing answers?

  29. Conducting an interview • Establishing a relationship/trust • Explaining aims & procedures • Low key presentation of self and involvement • Active listening • Clarifications, probes and prompts • Recording, transcribing and analysing

  30. Focus groups • Consist of a small group of people who are brought together by a trained ‘moderator’ (usually the researcher) to explore attitudes, feelings and ideas about a particular topic or set of issues

  31. Main characteristics of FGs • A question schedule is prepared by the moderator to focus the discussion • Place value on interaction within the group as a means of eliciting information – a social experience • No requirement to reach consensus • Less pressure to hold a fixed point of view – in fact opinions can develop

  32. FGs are useful when … • Topics or issues need to be explored in depth, or attitudes revealed • Typical groups of key informants are to be targeted eg language teachers

  33. Advantages of FGs • A less directive interview format • ‘Interviewer bias’ can be greatly reduced • Provide a more natural social forum for the exchange of ideas. Interviewees are able to take the initiative – not just respond • The right to silence

  34. FG problem areas • Less directive means less predictable • Huge amounts of data can be collected, much of which may be unusable • Perception that FGs are a market research tool (Tony Blair & New Labour)

More Related