Lecture 11

1. Lecture 11 Questionnaire Research

2. Chapter Topics • The nature of questionnaire surveys • Merits of the questionnaire survey method • Interviewer-completion versus respondent-completion • Types of questionnaire survey • Questionnaire design • Conducting a pilot survey • Validity of questionnaire surveys • Questionnaire-based interviewing • Coding

3. The Nature of Questionnaire Surveys • Data collected via a printed questionnaire or interview schedule • Usually based on a sample selected from a population (see Chapter 11) • Reliance on information from individuals – self-reported data • Often large samples require computer-aided analysis

4. Merits of the Questionnaire Survey Method • Quantified data for decision making • Ideal for providing quantified information for organisations which need quantified information for decision-making. • Transparency • Provide a 'transparent' set of research procedures. • Procedures clear for all to see. • Data can be re-analysed for alternative interpretation. • Succinct presentation • Quantification can provide complex information in a succinct form. • Comparability • Longitudinal and annual surveys enable the study of change over time. • Capturing complexity • An effective means of gathering a wide range of complex information on individuals or organisations.

5. Interviewer-completion vs Respondent-completion • Interviewer-completion: • Interviewer reads out the questions and writes down answers • Respondent-completion • Respondent reads the questionnaire and writes down the answers

6. Interviewer-completion vs Respondent-completion: Advantages & Disadvantages

7. Types of questionnaire survey • Household surveys – respondents selected on the basis of where they live and interviewed in their home; • Street/quota/intercept surveys - respondents selected by stopping in street, shopping malls, etc; • Telephone surveys - interviews are conducted by telephone; • Mail surveys - questionnaires sent and returned by mail; • E-surveys - questionnaires sent and returned by email/Internet; • Customer/visitor on-site intercept surveys – users/customers surveyed on site; • Captive group surveys - members of groups are surveyed • Organisation surveys - members of an organisation/ organisations are surveyed

8. Types of questionnaire survey: characteristics

9. Household questionnaire survey • Face-to-face interview – interviewer completion • Variations: • Drop-off and collect (eg. national census) • Partial interview and partial drop-off & collect • Representative of a defined community – eg. suburb, local govt area, city • Interview can be quite lengthy – eg. 30 minutes + • Relatively expensive • Omnibus surveys: • Different clients buy different questions in a composite questionnaire

10. Street/quota intercept surveys • Respondents intercepted in shopping streets/malls etc. • Face-to-face interview – interviewer completion • Interview short • Aim: a sample representative of community, but: • Some groups under-represented • ‘Quota sampling’ = specified number of interviews in particular gender/age groups (based on census) • Failure to achieve quotas can be corrected by weighting (see Ch. 11) • Relatively cheap

11. Telephone survey • Common for political/opinion polls • Sampling based on ‘White Pages’ • Excludes non-telephone subscribers • Emerging problem of mobile phones • Problem of non-response • Automatic Computer-aided Telephone Interviewing (CATI) • Automatic sampling and dialling • Answers keyed directly into computer (quick, cost-effective) • Interviews typically short – 15-20 mins max. • Respondents cannot be shown lists • Not being face-to-face is an advantage and a disadvantage • Relatively cheap

12. Mail surveys • Questionnaire is mailed to respondents • Sometimes the only practical method – eg. geographically scattered sample • Relatively cheap • Respondent-completion • Problem of response rates: • 70% non-response often reported

13. Mail survey response rates • Response rate affected by: • Interest of the respondent in the survey topic • Length of the questionnaire • Questionnaire design, presentation, perceived complexity • Style, content and authorship of accompanying letter • Provision of a reply-paid envelope • Rewards for responding • Number/timing of reminders/follow-ups – see Fig. 8.3

14. Mail survey response pattern

15. E-Surveys • A mail survey sent via email • Variations: • Download a questionnaire for completion and return • Direct completion and submission via the internet • Automatic analysis software • Structured filters etc. (see ‘Questionnaire Design’)

16. Customer/visitor on-site intercept survey • Various names: • Customer survey; Visitor survey; On-site survey; User survey; Intercept survey • Interviews take place on-site • Mostly interviewer-completed but respondent-completion possible • Relatively cheap • By definition, excludes non-customers

17. Captive group survey • Survey in an organised setting – participants may have little choice • But research ethics requires voluntary participation … • Respondent completion • Very cheap and quick

18. Organisation surveys • The organisation or organisational unit is the unit of analysis • Any of the above survey formats may be used • Issue of target informants: CEO or more specialised, lower level employees?

19. Questionnaire design • Topics: • Research problems and information requirements • Types of information • Open-ended and pre-coded questions • Wording of questions • Measuring attitudes and opinions • Ordering of questions • Layout • Filters • Introductory remarks

20. Research problems & information requirements • Questionnaire design/content should arise from the research problem and its information needs – see Figure 8.4

21. Questionnaire design process

22. Type of information • Respondent characteristics Who? • Activities and behaviour What? • Attitudes and motivations Why?

23. Open-ended and pre-coded questions

24. Wording of questions • avoid jargon • simplify wherever possible • avoid ambiguity • avoid leading questions • ask only one question at a time (avoid multi-purpose questions)

25. Wording of questions: good and bad practice

26. Measuring attitudes and opinions • Formats • Open-ended or direct questions • Checklist • Ranking • Likert scales • Attitude statements • Semantic differential

27. Fig. 8.7a/b:Attitude/opinion question formats a. Open-ended/direct: What attracted you to apply for this course? ________________________________ b. Checklist: Of the items on the card, which was the most important to you in applying for this training course? A. Good reputation B. Easy access C. Curriculum D. Management pays fees E. Easy parking

28. Fig. 8.7c: Attitude/opinion question formats c. Ranking: Please rank the items on the card in terms of their importance to you in choosing a course. Please rank them 1 for the most important to 5 for the least important. Rank A. Good reputation ___ B. Easy access ___ C. Curriculum ___ D. Management pays fees ___ E. Easy parking ___

29. Fig.8.7d:Attitude/opinion question formats • d. Likert scales: How important each of the following items in your • decision to choose this training course? • Very Quite Not very Not at all • important important important important • Good reputation □1□2□3□4 • Easy Access □1□2□3 □4 • Curriculum □1□2 □3□4 • Level of fees □1 □2□3□4 • Easy parking □1□2□3 □4

30. Fig. 8.7e:Attitude/opinion question formats • e.Attitude Statements: Please read the statements below and • indicate your level of agreement or • disagreement with them by ticking the • appropriate box. • Agree Agree No Disagree Disagree • Strongly opinion strongly • The learning experience □1 □2 □3 □4 □5 • is more important than • thequalification in education • Graduate course fees are □1 □2 □3 □4 □5 • too high

31. Fig. 8.7f:Attitude/opinion question formats • e. Semantic differential: Please look at the list below and tick • the line to indicate where you think • this course falls in relation to each • factor listed. • Difficult |_______|________|________|________| Easy • Irrelevant |_______|________|________|________| Relevant • Professional |_______|________|________|________| Unprofessional • Dull |_______|________|________|________| Interesting

32. Ordering of questions • Start with easy questions • Start with 'relevant' questions • Leave sensitive questions until later

33. Questionnaire layout • Be aware of reader/user – interviewer or respondent? • Special care with mail survey questionnaires • Compactness (eg. single page) = ease of handling • Two-column layout often helps

34. Filtering

35. Introductory remarks • Eg. ‘Hallo, my name is _____ : we are conducting a survey of _______: would you mind answering a few questions? • Different content/formats for different survey types • Interviewer-completion: include in interviewer instructions – additional information available if required • Respondent-completion – printed on questionnaire • Mail survey: this is dealt with in covering letter

36. Conducting a pilot survey • Purposes of a pilot survey: • Testing questionnaire wording • Testing question sequencing • Testing questionnaire layout • Gaining familiarity with respondents • Testing fieldwork arrangements (if required) • Training and testing fieldworkers (if required) • Estimating response rate • Estimating interview or questionnaire completion time • Testing analysis procedures

37. Validity of questionnaire surveys • Validity: the extent to which the data truly reflect what they are thought to reflect • Validity may be affected by: • Exaggeration or under-statement (to give good impression) • Inaccuracy of recall • Desire to please the interviewer • Some checks: • Include ‘dummy’ categories in some questions– eg. books that do not exist, events that have not happened • As the same question twice – in different way • Repeat interviews – some time later

38. Questionnaire-based interviewing • Ideally interviewer should stick strictly to the wording on the questionnaire • … especially opinion/attitude questions

39. Coding of questionnaire responses • Pre-coded vs Open-ended questions • See Figure 8.5 • Pre-coded – codes already exist • Open-ended – coding system must be devised • See Figure 8.9

40. Open-ended and pre-coded questions

41. Coding open-ended questions

42. (Fig. 8.10) Recording coded information – 1 Management Training Survey 2003 | Office Use | # 001 QUESTIONNAIRE NUMBER | qno | 1. What training course are you attending? | | People Skills 1 | Global Business 2 | 2 crse Strategic Management 3 | Other _______________ 4 | | ONLY ONE ANSWER POSSIBLE – ONE CODE – ONE VARIABLE (crse)

43. Recording coded information - 2 2. What staff development services have you used in the last six months? | | Career Planning 1 | 1 cp Mentoring clinic 1 | 1 ment Computer training 1 | 0 comp Performance Appraisal 1 | 0 pa | UP TO FOUR POSITIVE ANSWERS POSSIBLE – FOUR VARIABLES

44. Recording coded information - 3 • 3. Please rank the items below in terms of their importance to | • you in choosing a training course, from 1 for the most | • important down to 5 for the least important. | • Rank | • A. Good reputation 1 | 1 rep • B. Easy access 4 | 4 access • C. Curriculum 2 | 2 curr • D. Management pays fees 3 | 3 fees • E. Easy parking 5 | 5 park • FIVE RANKS REQUIRED – FIVE VARIABLES

45. Recording coded information – 4/5 4. How much have you spent on books for the training course? | | ANSWER RECORDED DIRECTLY – NO CODING $ 100 | 100 | cost | 5. Please indicate the importance of the following to you in studying. | | Very Important Not at all | important Important | Good textbook 321 | 3 text Knowledgeable lecturer 321 | 3 lect Easy assignments 321 | 1 assgn THREE ANSWERS REQUIRED – THREE VARIABLES

46. Recording coded information - 6 • Do you have any suggestions for improving the training course? | • | • | 1 sug1 • ________Less theory_________________________________ | __ sug2 • | __ sug3 • _________________________________________________ | • OPEN-ENDED (CODING SEE Fig. 8.9) – UP TO THREE ANSWERS RECORDED = THREE VARIABLES

47. Data from completed questionnaires Data in Figure 8.11: ready for computer analysis – see Ch. 13

48. Summary • Questionnaire surveys concern the gathering of information from individuals using a formally designed schedule of questions called a questionnaire or interview schedule. • Surveys are useful when: • the research questions indicate the need for relatively structured data • data are required from samples representative of a defined wider population. • Questionnaire surveys may be: • interviewer-completed or respondent-completed. • household surveys; street/quota surveys; telephone surveys; mail surveys; e-surveys; customer surveys; captive group surveys, organisation surveys, • Questionnaire design: • starting point is the project’s management/theoretical research questions • A pilot survey be conducted to: • test questionnaire wording, sequencing and layout • test fieldwork arrangements, including training fileldworkers • estimate response rate and interview or questionnaire completion time. • Validity of questionnaire surveys can be tested by inclusion of dummy categories, repeat questions and repeated interviews • Coding is necessary for analysis by computer.