Selecting, recruiting and informing users

1. Selecting, recruiting and informing users Alfred Kobsa University of California, Irvine

2. Recruiting users • How many? • Using 5-8 users for a set of tasks will already reveal most major UI problems related to those tasks • Use 3-5 users for each subgroup (☛ potentially exponential growth of subject size) • Considerably more if statistically significant differences are at issue • Inhouse or external? • Inhouse: confidentiality preserved, good motivation, inexpensive(?), fewer logistics, can be rushed • External: users often more representative, neutral • Recruitment channels: • announcements (posters, flyers, [online] bulletin boards, [online] classified ads, Internet forums, email, …), word-of-mouth, crowdsourcing, recruiting agencies, sales / distribution channels (?)

3. Selecting and scheduling users • “Screener”: set of criteria to help filter participants • Positive criteria: specific demographics, skills • Go for facts, avoid self-assessment • Negative criteria: contacts to competitors, “odd people”, “professional test users” • Screen potential participants (preferably via phone or in person, 10-15 mins) • Send out invitation in writing, and an email reminder on the day before (with directions, cellphone number for last-minute changes) • Over-recruit (?), or better have backup • Create an inhouse database of potential subjects (also for tax purposes)

4. Participant incentives • Prime motivation to participate should be • being interested in your product (but not too much) • being curious about user studies • feeling flattered • wanting to help develop a better product • wanting to help you and others • Incentives can also be a motivator, but generally should not be the only one • “Freebies”, gift coupons, cash, charitable donation, … • Thoughtful selection can be very important • Incentive should not be too high • Give everyone the same amount, or make sure that partici-pants are not aware of discrimination

5. Compliance with policies and regulations • Company policies often require subjects to sign liability waivers, non-disclosure agreements, video release forms … • Federal regulations require federal agencies and colleges receiving federal funding to follow procedures to protect human subjects in scientific research (whose purpose is to enhance human knowledge, and not “only” to improve a product). • Subjects must give informed consent • Codes of conducts of professional societies (such as the American Psychological Association) may regulate how their members have to carry out human subjects studies.

6. Informed consent • Purpose of study In general terms; deception/omission permitted, specifically if full disclosure is given at the end of experiment. • Study procedures • Risks of study • Benefits, Incentive/Compensation • Confidentiality • Right to withdraw • [Consent to being video/audiotaped] • Retention and “secondary use” of data • Point of contact