Toward developing your questionnaire instrument

1. Toward developing your questionnaire instrument Facilitating quantitative measurement Greg Yates, January 2010. (See me for a listing of books and other valuable resources about questionnaire design, and also examples of questionnaires used as ‘models’ in this seminar. GY)

2. Purposes of Qs: Allow the quick and dirty • to be able to obtain data from a group of individuals, who we have the opportunity to target. • at around the same point in time. • to treat individuals similarly. • to enable individuals to be compared, and/or classified as to group membership. • to enable variance in responding to emerge. • to regard the individual case as representing one of a sample from a “population”. • To allow statistical analysis via the principles of data accumulation. (Note: we often hear some remarkable misunderstandings and myths about ‘samples’ and ‘populations’, so not get hung up about this).

3. You need an instrument • Questionnaire. • Mostly paper and pencil format. • Can be online (Many tools now, including UNISA’s own procedures, TELLUS-2). • Well, is it not simple? Ask a question, record the answer. It is only Q &R, anyway! • But, developing a good instrument is remarkably difficult: Many mistakes made, as people rarely have appropriate skills. No-one was born knowing this stuff.

4. Think about your goals: Are they • to get people to disclose or “open up”. • to get people to provide you with specific information which is not available to you by observation alone. • to tap a person’s knowledge. • to solicit the individual’s perceptions, i.e. attitudes, beliefs, or predispositions. • to enable you to classify the individual.

5. But questionnaires just cannot 1 • Obtain information that is not available to the individual. • “What where you doing on 12 March 2008, at 8 pm”. • That is, asking for information that is inconsistent with encoding or memory processing principles. Studies show, for example, that people cannot correctly answer “How many times did you visit the doctor in the past year”, since this is not how your mind actually works. • Similarly, people typically cannot report on the type of drugs, medications, dosages, they have taken, etc. • (We will guess, yes, but our accuracy is horrendous, and we have no way of knowing how wrong we are)

6. But questionnaires just cannot 2 • Qs cannot substitutes for genuine behavioural indices. E.g. “Do you prefer to learn from pictures or words?” Such learning style preferences are verbal preferences but possess no validity when it comes to actual behavioural measurements within genuine learning contexts. • Qs are self-report, by definition, and limited by this facet. Hence, our interpretations of results are inevitably constrained by such awareness. • Our research reports need to reflect this, rather than “pretend” data refer to behavioural indices.

7. Other big constraints • Assumption that we all share language such that meanings are clear without being defined within the Q. (Bad vibes: Do terms such as ‘abuse’ and ‘spank’ or ‘punish’ have agreed referents?). • Assumption that people are truthful. • Verbal reports of change have no genuine validity. Eg. (a) Have you changed in your attitude towards the homeless? (b) Last year, how did you feel about people on welfare? (c) If your football team won, for how long would you feel happy? (this last example is called affective forecasting, at which humans are remarkably poor, but do not know they are poor at it).

8. Devising items: How to ask questions • “Obvious” advice: Simplicity, brevity, clear language, and use of direct propositions. • But before writing items, think very hard about response formats that are possible. These will substantially dictate the type of questions you will devise. • Following are some common formats.

9. Response formats 1 • Yes / No format. Allows a binary response. Could be a checklist, where person may be asked to tick only positives. Very fast method, but can lose shades of meanings. • Line scale method. Response between two anchors (or poles),so a visual analogue is created. On paper, often a 10 cm line. In computer, this can be expressed as a slider to grab with mouse. • During the exam I experienced : • No anxiety l---------------------l Extreme anxiety

10. Response formats 2 • Variations on line scales can involve an upright slider, in form of a ‘thermometer” • Or use of open-ended percentage items. Examples: What percentage of the time are teachers working under high pressure?______% • What percentage of your career do you want to serve as a country teacher?___% • What percentage of the time are you happy?___% (A highly valid index, the Fordyce scale)

11. Response formats 3 • Variations on line scales can involve placing gradations along the line (example) • But when gradations are specified, it is more often called an intensity scale. This means specifying a dimension that you are asking the person to accept, and then apply. E.g. volume of exam anxiety. • During the exam I experienced

12. Response formats 4 • Intensity scales may also be called adjectival scales since they ask people to rate how they apply adjectives to events or themselves. • Very common practice in area of self-research is to use intensity scale principles getting people to assay how much a statement applies to them, on a 5-point scale (see the Self-control scale of Baumeister). IC

13. Response formats 5 • Perhaps the most ecumenical format is one used by Rene Likert (say “lick-urt”), 1930s • Level of agreement with a definitive statement, ranging from negative to positive, typically with a midpoint, possibly defined as ‘neutral’. • Teachers have a easy life

14. Likert issues 1 • Meaningfully, suited only for positive statements. Negation of a negative is not consistent with human cognition, and confuses people. • Do you think that teachers are not under pressure at work? • Note: The definition (or even existence of) the midpoint is a sensitive matter. Does neutral ‘unsure’ mean no opinion? Or unwilling to say? Or lack of knowledge? Not applicable? Or that that the individual simply wants to fence sit.

15. Likert issues 2 • Sometimes it is gauche to use Likert, especially when the negative is not feasible • I love my mother, or • Employment is an important goal of education • Note how this feels just wrong, and even increasing the options to 7 may make the problem worse. Some researchers have used the unbalanced Likert style, and it generally works well:

16. Likert issues 3 • Note how the middle point can still become problematic even if we use this unbalanced style • I love my mother • If a person says ‘no opinion’, perhaps Mum is dead, and so he cannot “agree”: So the neutral point is more like ‘not applicable’ to that person. • There is no overall resolution about the midpoint problem, so it hinges more upon sensibly working through possibilities that could arise.

17. Issue of number of options • Should you use 5, 6, 7, 8, 9, or even 10? Contrary to common sense, this is not a big issue, and has almost no serious impact upon variance qualities. • For children, they are fully able to cope with 4, from about 7.5 years. (Avoid using line scales with children, but smiley faces are great) • For adults, anything above 7 creates confusion. People cannot discriminate more than 7 levels (In any situation at all: Information processing limits). • For most applications, 5 points are ideal, and going above this yields no real advantages, only problems. So regard 5 as the best overall default workhorse.

18. Things not to use 1 • Ensure that any specified response dimension really will fit the question stem. Example (poor): • Are you a studious person? • Initially seems sensible: But you are asking the person to agree with a question, not a statement. • As much as possible, remove qualifiers from question stems, as they belong in response options. • Example: You are a very studious person. If you disagree, could be (a) because you are not studious at all. Or (b) you are quite studious, but VERY is just not the word you would apply, so you ‘disagree’

19. Things not to use 2 • Avoid rankings like plague. • Which for you is the most important goal in education? Employment, life fulfilment, achievement, or citizenship? • There is not much one can do with ranked data, and the researcher loses richness and nuances of meaning. Any ranking based on more that 3 items is faulty on basis of human information processing characteristics such as ordering. (Yet I have seen people try to get people to rank 20 items… recipe for nonsense).

20. Response options can influence responses • When words are unclear, the available options may cue people as to how to interpret the question. Notably, when time is being used • How often do you feel annoyed? • Alternatively: • People tend to use end points as psychological anchors, and then navigate between the two anchors. But naturally will swing as close to centre as they can.

21. Avoid subjective indices • How much do you read to your child? • How often do you set formal type tests in your classroom? • Such questions are fine, but response options have no meaning and need reality anchoring.

22. Watch for directionality • May go either way. HOWEVER, good practice is to go from low to high, and not vary this: Teachers are paid For teachers, work pressures demands are The worst offenders are when numbers are used, so that 1 might be “low” for one item, but on the next one, 1 is said to be “high”. Total mental meltdown.

23. Guide to writing questions 1 • Aim for simple, direct, short, and positive items. Avoid jargon or technical terms. • Use reverse statements, OK, but not reversed response options. It is NOT necessary to reverse many items. • Eliminate vague or imprecise terms, such as many, often, rarely, sometimes, usually. • Watch out for negated intensifiers such as “not everybody”. Is it true that not all people enjoy life?

24. Guide to writing questions 3 • Be alert to questions presuming values (a) Do you think teachers have to cope with matters that are trivial? (b) Would you endorse policies which, other things equal, save money? (c) Should we stop parents from being overly zealous? (Improper) • Or convey ‘no-brainers’, or non sequiturs. (a) Does punishing children do them harm? (b) Would you apply for a better job? (c) Should we remove cancer-causing agents from our schools? (d) Should we stop climate change?

25. A computer package • QUAID: Question Understanding Aid • http://mnemosyne.csl.psyc.memphis.edu/quaid/quaidindex.html This scans your questionnaire and provides feedback on its grammar, vocabulary, sentence structure, syntax, ambiguities, and use of vague terms. Bl__dy marvellous. BEST ADVICE EVER: HAVE OTHERS READ IT

26. Construct formation • How many items are needed? • Pragmatics enters the picture, but basically 5 items are often seen as basic to forming a trait. • But, there are many more complex rules about construct formation, and that is not today’s topic. • END