MHE3: RM 5 – Questionnaire development

MHE3: RM 5 – Questionnaire development A McCaw-Binns Section of Community Health UWI, Mona

Learning Objective: Steps in questionnaire construction • Definition of information being sought • Identification of variables for inclusion • Drafting the questions • Constructing the questionnaire • Pilot testing the questionnaire • Redrafting the questionnaire

1. Definition of the required information • Enquiry: thinking, discussion, reading, inspiration • Development of goals and objectives • Study design and method of data collection • Interviewer administered • Self administered • Group setting • Postal • Use of secondary data

Questionnaire format • Select the most appropriate format for the purpose of the study • Interview schedule • Guide for an interviewer who asks the questions • Self-administered questionnaire • Respondent reads the questions and fills in the answers • Term questionnaire used to indicate a list of questions prepared for either of these purposes

Mode of administrationInterviews • Face to face • Phone • Computer assisted • Choice of a method depends on: • Feasibility e.g. prevalence of phones • Cost • Literacy level of respondent • Nature and purpose of the study

Types of Interviews • Less structured interview • Useful in a preliminary survey to obtain information to help in subsequent planning of a study • Structured interview • Preferred in situations requiring standardized technique • Wording and ordering of questions are decided in advance

Mode of administrationInterviewer administered Face to face interview: Advantages • Stimulate and maintain a respondent’s interest • Create rapport conducive to answering questions • Can explain questions which are not understood • Optional follow-up or probing can occur if previous responses are inconclusive or inconsistent • Observations can be made • Interviewer can use visual aids • e.g. cups and portion sizes in a dietary survey

Mode of administrationPhone interviews • Sometimes preferred to: • Mailed questionnaires • Face to face interviews • Some questions easier to ask on the phone, others face to face • e.g. questions on sensitive issues easier for both interviewer and respondent over the phone

Mode of administrationComputer assisted interviews • Questions posted on a computer screen • Keyboard or mouse used to enter answers • Skipping/branching patterns can be built into the programme → screen automatically displays appropriate question • Checks may be built in to immediately identify invalid responses • Response which lies outside an acceptable range • response inconsistent with previous replies • Responses entered directly on to the computer record, avoiding subsequent coding or data entry

Mode of administrationSelf administered questionnaire • Decision made on practical considerations • Self administered questionnaires (SAQ) • Simpler and cheaper • May be administered to many people simultaneously • e.g. a class of kids • However: • Respondent must have a certain level of education/skill • People of low SES less likely to respond to mailed or other SAQ • Restrict to simple questions with simple instructions

Mode of administration Self administered questionnaires • Better for: • Reports of disability, pain and emotional disturbance • Reporting socially undesirable details • e.g. history of STDs • number of sexual partners • Disadvantage: • More likely to have missing information • Exception: respondents who have special motivation • e.g. follow up of a patient’s progress by a physician

Variable selection –Types of variables: Universal • Variables which are so often relevant, that their inclusion should always be considered • Defines the characteristics of your study population e.g. • Socio-demographic: age, gender, union status, religion • Socio-economic: education, social class, occupation • Health status: height, weight, BP

Types of variables: Dependent • Your outcome(s) of interest • The manifestations whose variation you want to explain or account for • e.g. BMI, reason for coming to medical school • Value is dependent on the effect of other variables – the independent variables

Types of variables: Independent • Factors you suspect of being associated with your outcome • The independent variable is not influenced by the event, but may cause or contribute to variation in the event • e.g. diet, exercise, parent’s occupation • Selection of these variables are important if you are undertaking an analytical study

Types of variables: Confounding • Factors which may obscure the relationship between your outcome and other independent variables • Need to collect data on them to account for their effects in your data analysis • Best identified from literature review • May be picked up during data analysis when you observe associations which do not make sense

Complex Variables • Some variables are too complex to measure as single entities and you have to: • 1. Break into component parts • 2. Measure each part separately • Must ensure that all component parts are in your list of variables/questions to be asked • e.g. Gestational age = date of delivery – date last menstrual period (LMP)

Proxy Variables • Some variables cannot be measured directly • Persons unwilling to disclose information • Information not readily available • Have to use indirect or substitute methods or proxy variables e.g. • Income, occupation or education →social class • Toilet facilities as a measure of SES • A. Type of toilet facilities (flush type, pit latrine, none) • B. Whether toilet facilities are shared • Attitude to corporal punishment: • A. Do you beat your child? • B. If your child hit another child, would you: (a) spank (b) punish (c) reprimand (d) ignore (e) smile

Defining the Variable • Conceptual definition • Outlines the variable as we conceive it • e.g Obesity=excessive fatness, being overweight • Operational definition • Working definition of the characteristic as it will be measured • e.g. BMI > 25 kg/cm2 • Weigh person in their underclothes, without shoes • Record weight in kg to nearest 1/10 of kg. • Measure height in cms. using a stadiometer

Measurement Scales I • Variables take on different values for different persons • e.g. religion – Catholic, Jew, Muslim, Protestant • Variables therefore have a scale or range of possible responses or values over which they can be measured • Qualitative (e.g. religion) • Quantitative (e.g. height or education)

Measurement Scales II • Nominal • Classification into unordered qualitative categories • No inherent order to the categories • e.g. religion, parish of residence • Dichotomous • Arranges items in either of two mutually exclusive categories

Measurement Scales III • Ordinal • Classification into ordered qualitative categories • Values have a distinct order • Categories are qualitative • No natural numerical distance between possible values • e.g. Social class, education

Measurement Scales IV • Interval (Discrete) • Defined as a whole number only (can’t have 1.5 babies) • Equal interval – values have natural distance between them • e.g. years of completed education, parity, date of birth • Ratio (Continuous) • Interval scale with a true zero • Ratios between values meaningfully defined • Can speak of one value as being so many times greater or lesser than another • Value may fall anywhere on a number line • e.g. temperature, weight, height, income

Characteristics of a satisfactory scale • Clearly defined components • Operational definitions for variable and its components • e.g. union status (married, common-law etc) • Sufficient categories (comprehensive) • Allow all groups to be classified • Mutually exclusive • Non-overlapping – each case should only fit into one category • e.g. 20-30, 30-40 versus 20-29, 30-39

3. Drafting of questions • Issues: • Phrasing/design of the questions • Design of the questionnaire (layout) • Ordering of questions • Question format • Open ended – free response questions • No predetermined response • Subjects answer in their own words • Closed ended – fixed response questions • Respondents given predetermined list of response options • Answer by choosing from a number of fixed alternatives

Question format:Open ended • Advantages • More detailed answers elicited • Important in exploratory surveys • Indicate range of likely replies • Guides formulation of alternative responses for closed ended questions • If followed by ‘probe’ questions, are valuable in the study of complicated opinions or attitudes • May be used for case illustrations

Question format:Open ended • Disadvantages: • Less structured responses • Difficult to code and analyse using powerful statistical methods • More time consuming to complete

Question format:Closed-ended questions Advantages: • Greater uniformity in data collection • Less time taken to collect responses • Simplifies coding and analysis • Dichotomous • yes-no • agree-disagree • Multiple options • Behaviour: Never, seldom, occasionally, frequently, very often • Education: Primary, secondary, tertiary

Question format:Closed-ended questions Disadvantages: • Limits the variety and detail of responses • Less ‘depth’ in answers • May frustrate respondents if their answer choice is not among the possible selections •  non-response/missing information • Should include: • “Other (specify) ………………….” category • Insurance against oversight in choice of categories

Guidelines for Success:Closed ended questions • Include ‘not known’ or ‘not applicable’ option • Reduces non response rate • Allows one to differentiate between persons who • A. Did not have the information requested • B. Failed to or refused to answer • C. Information requested was not applicable to their situation • Avoid long successions of yes-no or other identical responses • Respondents (and interviewers) may fall into a rut and give the same response unthinkingly (question fatigue)

Seven-or-five point Likert scale Strongly agree Agree Undecided Disagree Strongly disagree Forced choiceresponse Strongly agree Agree Disagree Strongly disagree Question format:Likert and forced-choice responses Attitudinal questions: two possible response formats may be chosen What is the difference between the two question types?

Question format:Likert and Forced choice • Forced choice – does not allow/include a ‘middle of the road’ or ‘neutral’ response • Guards against an ‘acquiescent response mode’ • i.e. respondents who give neutral responses all the time • You have to decide whether you need to force a choice or not

Good questions:Face validity • Questions should be asked only if necessary • Questions should accurately reflect what the investigator wants to know • Process of question development may require rethinking conceptual definitions

Good questions:Know the answer • Respondent must be able to answer the question • If person cannot answer the question directly, indirect methods may be used e.g. • Mental health status may be inferred by asking about a series of signs and symptoms • Attitude re discipline may be determined by asking how a parent would react in specific situations

Good Questions: User friendly–Level of wording • Wording of questions must be tailored to the intended respondents • Avoid jargon, especially professional jargon • e.g. Trisomy 21 versus Downs syndrome • Avoid double negatives • Keep questions simple and concise • Avoid intricate and demanding questions requiring complicated alternatives • Short questions preferable to long ones, except where length increases clarity

Good Questions: Clear and unambiguous • Questions may mean different things to different people • Phrase questions in language that: • Respondents will understand • All respondents understand in the same way • e.g. “single” may mean: • ‘never married’, or • ‘currently alone’ such as separated, widowed or divorced • e.g. ‘old people’ may mean: • Everyone over 30 years to a teenager • Everyone 60 or 70 years to adults of different ages

User friendly:Double barreled questions • Avoid questions which have more than one thought (double barreled) • e.g. do you eat apples and oranges? • May be difficult to answer • Split them into two questions • Questions requiring a yes answer to indicate agreement with a negative statement may confuse respondents • e.g. should a 50 year old woman not have regular breast X-rays?

Fair questions:Biased and leading questions • Questions should not be phrased in a way that suggests a specific answer, e.g. • How often do you go to church? • May lead respondent to give an answer which is not entirely truthful • Alternative: • Do you go to church? • If yes, how often do you go?

Fair questions:Biased/leading response formats • Response categories in closed ended questions should not be phrased in a way that excludes other options, e.g. • How often do you go to church? • What would be the impact of omitting ‘never’ from the option list?

Sensitive issues – Not offend/embarrass • If sensitive questions need to be asked, care needs to be taken in wording and sequencing • Self administered questionnaires usually elicit a greater number of socially undesirable responses • For interviews: May put possible responses on cards so that the respondent can simply choose the option or point without having to use words

Sensitive issues –Not offend/embarrass • Long introductions (explaining why you need to know) and open ended questions increase response rates for sensitive issues • Presence of low frequency categories often made people less willing to admit higher frequencies • Use words familiar to respondents • e.g. Cannabis use – give respondent option to select the term they prefer • e.g. – do you call cannabis ganja, herb, weed or marijuana? • Use their preferred term in subsequent questions

4. Questionnaire construction • Before you begin to construct the questionnaire, list the variables you wish to measure • These will be selected based on: • Objectives of the study • May be identified while reviewing the literature • One’s intuitive understanding of the problem under investigation

Question selection • Once variables are selected, formulate questions that have face validity • To enhance validity, may ask multiple questions on some topics • To enhance comparability with other studies, questions may be borrowed from other sources • Borrowed/standard questions have been already tested and found to be serviceable • Caveat: The same questions may differ in validity in different population or different circumstances

Putting it together:Introduction • Introductory explanation needed; state: • The purpose of the study • That failure to participate will not affect access to care • Include statement about confidentiality • Does not mean anonymity unless you can guarantee that there is no way to trace which questionnaire belongs to whom • Generally not advisable - may have to go back to clarify missing data • Systems should however be in place to safeguard trust

Putting it togetherInstructions • SAQ • Introduction should include clear instructions for completion, with examples • Include instructions for disposal/return • Mail in • Hold for collection, etc • Interviews • Instructions and guidelines may be placed in separate interviewers manuals

Putting it togetherData collection I • First questions should be: • Easy to answer • Of relevance to the topic • If possible, interesting • General questions should precede specific questions • Sensitive questions should be left until later (even age, education or income may come later) • Sequence should flow naturally and logically

Putting it togetherData collection II • If questionnaire is long, may need to switch topics or alter the format to prevent boredom • Avoid long successions of questions requiring identical responses • e.g. yes/no; often/seldom/rarely/never • With proper sequencing, may bypass irrelevant or not applicable questions using skipping instructions – however do not overuse • e.g. if no, go to question 10 • Examine sequencing carefully, to ensure that you are not skipping over important questions

5. Questionnaire pilot/pre-test • Test the draft questionnaire on: • Small group of intended respondents • Clinical or research colleagues • Ask for feedback from pilot respondents • Questionnaires should be: • Pilot tested • Redrafted • Retested • If you cannot pilot your questionnaire, do not do the study!

Pretest I • Interview 10-30 persons • Chose haphazardly, but judiciously to include, possible ‘difficult’ customers to highlight flaws • e.g. high/low education • Should not be potential members of the study population but be similar in characteristics to your study population • Record reactions such as boredom, irritation or even interest, as well as responses (use margins) • Critique sequence, skip patterns, content

Pretest II • Pretests reveal: • the need for changes in the questions (identifies questions lacking face validity) • Sequencing problems • How the questionnaire may be shortened • Questions lacking resolution (e.g. 99% response to one option) may be omitted

6. Redrafting of questionnaire • Address problems identified in the pilot • Last chance to check for gaps in information being sought • At second pretest, time test • If takes more than an hour, try to prune

MHE3: RM 5 – Questionnaire development