Questionnaires

1. Questionnaires Topic Presentation Julie Stromer October 20, 2004

2. Contents • What is a questionnaire? • Who invented the questionnaire? • When to use questionnaires • Information types • Question types • Likert Scales • Advantages / Disadvantages of Questionnaires • Interviews – the alternative to Questionnaires • The debate: how hard it is to write your own questionnaire • Guidelines for questionnaire writing • Questionnaire writing – by example

3. What is a Questionnaire? • a set of questions on a topic that can be asked of a potentially large group of respondents. • Answers can be analyzed and conclusions drawn from the responses • Specifically for HCI studies, questionnaires are meant to collect information from study participants regarding: • Backgrounds • Opinions about different aspects of tasks • General usability of the product (e.g. interface)

4. Who Invented the Questionnaire? • Francis Galton (1822-1911) • Was the first cousin of Charles Darwin. • Explorer, Meteorologist, Geneticist, Psychologist, Statistician, Anthropologist, Criminologist, Eugenicist.

5. When to use Questionnaires • Resources and money are limited • Participant privacy is necessary • To complement information obtained by other methods (e.g. experimental methods) • Early definition phase of computer system design • What should the new system be able to do? (but NOT the design of it) • During usability study of interface prototype • May indicate eventual acceptance of system • After system delivery • How well system works, what needs to be changed

6. General Layout Guidelines • Set context for respondent • situation, topic, or set of circumstances • Only ask for information you will directly use • Keep the questionnaire as short as possible • Don’t bore or frustrate the respondent • Have an underlying reason for every question • Group topic areas together to keep the respondent focused

7. Information Types • Factual • E.g. age group, computer experience • Opinions • Respondent’s thoughts and ideas • Generally never wrong • Attitudes • How do they feel about the system?

8. Question Types • Closed-ended items • Multiple choice • Rating Scale • Open-ended items A B C D E 1 2 3 4 5

9. Closed-ended Questions • Very particular finite answer expected • Easier to tabulate and analyze • Saves time and money • Two types: • Multiple Choice • Rating scales

10. Rating Scales • Questions or statements with a numerical value assigned to the alternatives • Note: the numerical value is internal, and is usually not presented to the respondent • Added advantage: traditional statistics used to analyze the responses • Typically use a Likert Scale

11. Likert Scales • Characterized by an odd number of choices • Odd to provide a clear number of choices • Two types: X X • Continuous better suited to most statistics • Distribution assumptions

12. Open-ended Items • Collected information is unrestricted • Much like an interview • Typically used at the questionnaire end • Problems: • Answers cannot be easily used in statistical analyses • Answers have to be hand-read • Many readers means the answers are subject to the reader’s interpretations • Answers require more thought and time on the part of the respondent • e.g. dishonesty favoring brevity

13. Advantages of Questionnaires • Anonymity elicits more honest responses • Less expensive than other means • a questionnaire can be administered to many people at once • Feedback is from the respondent’s point of view • BUT: If conducted orally, as in an interview, these advantages are limited.

14. Disadvantages of Questionnaires • Additional information and opinions cannot be discovered if questions are on paper • People will write as little as possible, even if they have more to say. • Typically designed to fit a wide range of applicable situations • Specifics may be missed • Often used by themselves • Cannot gather all the information required • Are quite subjective • Best used in combination with objective methods (e.g. laboratory experiment, field study, etc.).

15. Interviews – more flexible? Questionnaires • A printed form of questions is fixed! • No additional information retrievable Interviews • More personal and flexible • Novel opinions may emerge • Interviewer can ask for clarification or elaboration on respondent’s statements

16. Questionnaire Writing:Not for the Novice… • Pre-made questionnaires have undergone rigorous testing • Questionnaire writing requires a deep understanding of: • Psychological measurement • Statistics • Experience administering / interpreting questionnaires • Other sources offer “easy-to-follow” guides for creating questionnaires • So which is it?!?

17. Existing Questionnaires • Pre-made questionnaires for HCI studies: • Refer to www.acm.org/~perlman/question.html for more information

18. Purdue Usability Testing Questionnaire

19. Composing Questions • Clarity • avoid colloquialisms, unambiguous • be precise • Leading Questions • Don’t put words in the respondent’s mouth • Phrasing • Child-like vs. childish • Prestige Bias • Over or under-estimating own abilities

20. Steps to Writing a Questionnaire • Defining the objectives of the survey • Determining the sampling group • Writing the questionnaire • Administering the Questionnaire • Interpreting the results

21. Our Example This year’s GSA (Graduate Students Association) freely distributed day timer.

22. Step 1 : Defining Our Objective • How satisfied are University of Calgary graduate students with this year’s graduate day timer? • Is it useful for them? • How might its design be improved upon in future editions of this publication?

23. Step 2 : Sample Group • This comes directly from the research question • University of Calgary graduate students

24. Step 3 : Writing the Questions • What do we want to know? • What information will we use? • Is this item better asked in an open-ended or closed-ended question? • How should we phrase it?

25. Try it Yourself! • Demographic questions to establish which group this respondent fits into. • General time scheduling habits, establishing the requirements of a day timer for a graduate student at the University of Calgary. • Day timer use in general • GSA day timer

26. What’s Next • After the Questionnaire is written, it is evaluated, tested, and revised • This cycle may be repeated more than once • Next the Questionnaire is distributed to the target group • Finally, the responses are analyzed

27. Thank you!

28. References and Resources • Eberts, R.E. (1994). “Chapter 4: Experimental methodology” in User Interface Design, Prentice Hall. (A basic reference, as provided in class, about questionnaires.) • Kirakowski, J. (?). Questionnaires in Usability Engineering – A list of Frequently Asked Questions. Human Factors Research Group, Cork, Ireland. Accessed September 27, 2004. http://www.ucc.ie/hfrg/resources/qfaq1.html (A list of answers to questions typically asked by questionnaire designers. Quite useful for basic information, and rationale for question design choices.) • Wikipedia – the free encyclopedia. “Francis Galton – inventor of the questionnaire” Accessed October 4, 2004. http://en.wikipedia.org/wiki/Francis_Galton (A simple reference identifying the inventor of the questionnaire.) • Root, R.W., Draper, S. (1983). Questionnaires as a Software Evaluation Tool. CHI’83 Proceedings, December, 83-87. (A study of the strengths and weaknesses of questionnaires.) • O’Brien, D. (1997). Questionnaire Design. Accessed September 20, 2004. http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/ (A list of tips and suggestions for designing questionnaire questions.) • Mantei, M.M. (1991). Questionnaire Design Studio. Tutorial Notes CHI’91, New Orleans, April 28, 1991. (A tutorial on creating effective questionnaires, written in plain English with many good examples and indications of what to do / what not to do when creating questionnaires.) • Perlman, G. (1998). Web-based User Interface Evaluation with Questionnaires. Accessed September 20, 2004. http://www.acm.org/~perlman/question.html (A web-resource of pre-made usability study questionnaires and information regarding available online cgi scripts and web-based material.)