Effects of Scale Lengths and Layout on Response Option Order

1. Response option order effects. Scale lengths and horizontal or vertical layout. Johan Martinsson University of Gothenburg www.lore.gu.se

2. Overview • 2 related but separate studies • The general focus on effects of different ways to arrange response options on a screen (eg. ”primacy effects”, tendency to pick the first response option) • Three factors to consider: • scale length • layout direction (horizontal or vertical) • response device (PC, tablet, smartphone)

3. Why do we care? • You want a neutral measurement instrument • An arbitrary design choice on your part should not influence results • An arbitrary choice of device of respondents should not influence results • => robust estimates

4. Some background • More and more respondents use smartphone to answer surveys (though clearly a minority) • Smartphone compatibility • no classic grids • shorter scales • vertical layout • Screen sizes differ

5. Some background • More automatic adjustment to device by survey softwares (responsive design) • Different respondents get different design • Robustness of measurement instruments more important to examine

6. Examples

7. Reversing response option orders

8. Horizontal or vertical layout

9. Example of automatic mobile adjustment by survey software

10. Study 1 • Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014 • Field work: July to September 2014 • Total n approximately 20,000 • Randomized experimental design 2x2x2: • ”Right direction” or ”wrong direction” first • 7 or 3 scale points • Horizontal or vertical layout

11. Study 1 • We had a few hypotheses... : • primacy effects exist • stronger effects with vertical layouts than horizontal • stronger effects for smartphone users • (especially with horizontal layouts)

12. Results

13. Results

18. Kind of boring results, no? • Luckily, we discovered something more interesting by mistake!

21. Study 2 • Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014 • Field work: July to September 2014 • Total n approximately 34,000 • Randomized experimental design 5x2(x2): • Horizontal or vertical layout of the questions • 10, 7, 5, 3, or 2 scale points (response options)

22. Results

23. Results

24. Results

30. Conclusions and recommendations • We observe weak but fairly robust primacy effects (as usual) • These do not seem to depend on device or scale length much (more surprisingly)

31. First, some useless advice... • Ban smartphones in your surveys! • Only use horizontal scales (not even sure it is better)

32. Some less useless advice • Use really short scales • Randomize when you can (nominal scales) • Split half reversed order? • Use long questions wordings and make sure respondents are aware of all available options before allowing them to answer (but increases burden and time)

33. Read more atwww.lore.gu.seLaboratory of Opinion Research (LORE)