Running a User Study

1. Running a User Study Alfred Kobsa University of California, Irvine

2. Pilot Test • A pilot test is extremely valuable for detecting problems in the planned user test. • These problems may be harmless, but they may also render test results unusable. • 2-5 pilot subjects are sufficient. • Some of them should be people who are very much willing to criticize the experiment (e.g., good friends, colleagues). • At least one of them should be a real test subject. • Pilot test should be run two days before the actual study at the latest (and there are many good reasons to do them far earlier). • Pilot test should include all procedures.

3. Problems that are frequentlyuncovered in pilot tests • Subjects take considerably longer/shorter than expected. • Task instructions are not well understood by subjects • Subjects cannot carry out a task since some necessary functionality has not been explained to them • Subjects use procedures for solving a task that are different from the one that the experimenters wanted to test • Subjects overwrite each others’ data files • Experimenters overwrite data of previous experiments • Software crashes • Experimenters forget to do certain things • Some materials are missing • Users have troubles finding parking, the building entrance, the usability lab, etc.

4. Welcome, Briefing,Instruction and Training • Welcome • Make participants feel comfortable / reduce anxiety • Bridge time until everyone has arrived • Let them show their ID (“for security/tax purposes”), and announce this beforehand • Ask them to switch their cell phones off. • Briefing • Inform about purpose of the experiment (as far as this is possible) • Emphasize that it will help develop a better product • Encourage criticism and praise • Emphasize that the product is being tested and not the user • Emphasize that the people they will see are not the developers • Show them the lab (or movies or pictures of it) • Have them sign legal forms • Let them fill in a pre-questionnaire (e.g., demographics, pre-test) • Instruction and Training • Instruction through video, or through instructor who follows a written script • Subsequent training tasks allow users to practice what they learned and to understand it better. • Guided training tasks: task description contains (partial) instructions on how to carry out a task, which subjects are asked to follow

5. Conducting the Test • Setup • (Bring subjects to test computers) • Verify initial settings and materials, start recording devices. • Ask them to begin (and to let you know when they are done) • Enter users in logbook/timesheet (e.g., who used which computer) • During the test • Inform users about timeouts • Answer questions generically; avoid biasing participants • If applicable, remind them to think aloud if they become silent • Watch test users and take notes, or monitor users from a distance • Record unusual occurrences in logbook/timesheet • If applicable: remind users occasionally to think aloud • If applicable: make sure users don’t influence each other • After the test • Post-questionnaire (e.g., opinions on software, test procedures; post-test) • Payment, escort • Prepare test station for new test subjects

6. At any time Be organized! Follow checklists!

7. Unexpected events • A participant does not arrive in time • A participant forgot(?) his or her ID • A participant refuses to sign the informed consent form or the non-disclosure agreement • A participant refuses to be videotaped • A participant does not want to switch off his/her cell phone • A participant is called away in the middle of the test • A participant’s cell phone rings continuously • A participant does not have the required qualifications • A participant becomes confrontational with other participants or with the experimenters • A participant takes over the group • Software freezes, computer breaks down, etc. • Outside interference (construction noise, vacuums, …)