Laboratory studies

1. Laboratory studies Lorrie Faith Cranor September 2011

2. Laboratory study process • Design study • Submit IRB application and wait for approval • Recruit participants • Screen participants • Schedule participants • Prepare for study • Conduct study • Instructions to participants • Analyze data

3. Design study • See lecture on designing experiments • Some additional issues for lab studies • How to provide realistic setting and tasks in a lab environment • Whether or not to think aloud • How to observe, record • How much to pay • How long

4. IRB application • Most usable security lab studies will be minimal risk and qualify for expedited review • However, use of deception may require more review • Use of deception must be justified by you and approved by IRB • Requires debriefing

5. Recruit participants • To recruit students • Flyers around campus • Posts to campus email lists and bboards (misc.market) • CMU CBDR http://www.cbdr.cmu.edu/ • To recruit non-students • Flyers on bus stops, grocery stores, libraries, any local business that has a bulletin board that will let you post • Posts to Pittsburgh Craigslist • Asking your friends and neighbors • Distribute handouts or make announcements at local religious organizations, school parents’ organizations, athletic organizations, etc. • We’ve recruited families by handing out flyers at Schenley Oval during kids’ soccer games • We’ve recruited senior citizens by visiting senior citizen lunch at Squirrel Hill JCC • Many local organizations offer inexpensive ads in their newsletters (for example Colfax PTOhttp://colfaxpto.org/newsletters/ • Some organizations may be willing to recruit members to participate as a fund raiser

6. Screen participants • Decide what your criteria are to participate and develop screening survey to screen out those who don’t qualify • You may also want to use screening survey to help you balance demographics

7. Schedule participants • Contact participants by email or phone to schedule their visit to the lab • CBDR includes an online scheduling system • Give participants clear directions for getting to the lab • Give them a way to contact you if they have to cancel or get lost finding the lab

8. Prepare for study • Make sure you have copies of study materials, consent form, payment record, etc. • Make sure all equipment is setup, working, has batteries, etc. Test it! • Develop a checklist of everything that needs to be setup before a participant comes into the lab so you can make sure you haven’t forgotten anything • Bring sufficient cash (or other payment) with you • Make sure you have exact change! • You can get reimbursed for human subjects payments, but if you plan ahead you can get a cash advance so you don’t have to layout money • Post signs so people can find the lab • Weekends and evenings you may have to meet people outside locked door • In some buildings posted signs routinely get taken down, plan to repost daily (or more frequently!) • If at all possible, have at least 2 people run study together • Useful to have someone greet participants and bring them to lab while someone else runs study • For some studies its useful to have someone ask questions and interact with participant (facilitator) while someone else takes notes • Practice, practice, practice!

9. Instructions to participants • Describe the purpose of the evaluation • “I’m testing the product; I’m not testing you” • Tell them they can quit at any time • Demonstrate any equipment • Explain how to think aloud • tell us what they are trying to do • tell us questions that arise as they work • tell us things they read • Explain that you will not provide help • Explain the basic concept of the UI, but not too much • Can be done in writing or via video for consistency • Describe the task • give written instructions, one task at a time