Facilitator Induce Errors: The Effects of Social Facilitation and Electronic Monitoring on Usability Testing

1. Facilitator Induce Errors: The Effects of Social Facilitation and Electronic Monitoring on Usability Testing Erin Harris & Jerry Weinberg Department of Computer Science Sue Thomas & Danica Geaslin Department of Psychology

2. St. Louis Area Usability Labs

3. ElectronicPerformance Monitoring • Increases levels of stress • Decreases both the levels of productivity and quality of work done.

4. EPM Examples • A study involving 762 telecommunications workers found that those who were monitored reported higher levels of stress and anxiety. • Other studies have found that participants working on tasks requiring much thought and skill performed significantly worse under EPM while participants working on simple repetitive tasks performed better

5. Social Facilitation • Social psychology concept that the behavior of an animal, including humans, is affected by the presence of another. • An individual’s dominant response is enhanced while the non-dominate response is impaired.

6. Two Social Facilitation Theories • Mere Presence Theory • This theory states that the presence of other people alone increases one’s arousal • Evaluation-Apprehension Theory • This theory goes further to state that it is not the presence of another individual alone, but also the anticipation that the individual is assessing your performance that increases one’s anxiety.

7. Social Facilitation Examples • Cockroaches completing a maze • Simple mazes were completed much faster in the presence of other cockroaches. • Complex mazes were much more difficult to complete in the presence of other cockroaches. • Pool players’ shot-making ability • Good players increased their shot-making ability • 71% to 80% • Less talented players dropped their percentage • 36% to 25%

8. Research Hypothesis • The presence of a facilitator will increase the effects of social facilitation • These effects will be magnified by viewing the observation lab and meeting the design team due to the increased salience of the evaluation • These effects will also be magnified by being constantly reminded of the observation

9. SIUE’s Usability Lab

10. SIUE’s Usability Lab

11. Procedures • Each subject was informed that they were going to be videotaped and observed over a closed circuit camera • Each subject was given a series of 16 Microsoft Word tasks; 8 simple and 8 complex • Each task had a time limit • The number of errors per task was recorded

12. Example Tasks • Simple Tasks • Center and underline the title of the document. • Insert a page break at the end of the document • Complex Tasks • Add a center tab stop at the 3-inch mark from the left side of the screen. • When you add a header to Microsoft Word, the header is added to all pages of the document. Setup Word so that the header of the first page of the document is different from the rest of the pages.

13. The Experimental Set-up • Facilitator Group • Served as control group of the study • No Facilitator Group

14. The Experimental Set-up • Facilitator + Reminding Group • Frequent communication with the software design team • Facilitator + Design Team Group • Touring the lab and meeting the design team prior to the test

15. 62.3 56.8 48.7 45.0 15.1 9.4 7.2 7.2 Results – Mean Errors

16. Results – Mean Errors 35.98 35.82 27.98 26.10

17. Results – ANOVA for Errors • Main effect for type of task • Significantly fewer errors were made on simple tasks (M = 9.74) than on the complex tasks (M = 53.2) • F(1, 96) = 367.34, p = .0001. • Main effect for group • The No Facilitator and Facilitator + Constant Reminding groups committed significantly fewer errors than the Facilitator and Facilitator + Design Team groups. • F(3, 96) = 3.50, p = .018. • The interaction between group and the type of task did not reach statistical significance • F(3, 96) = 2.10, p = .106

18. Results • Social facilitation is present in usability testing • Simple tasks were not really simple • Simple tasks needed to be so well learned that they were automatic. • Constant reminding • Our constant reminding caused the users to explain their actions. • Providing these explanations caused the users to think harder about what they were doing, improving performance • Wright & Converse (1992) had similar findings when instructing users to provide explanations for their actions

19. Putting it into Practice • Minimize interjection into user’s awareness reduces the errors • Will this provide a clearer picture of usability? • Is it preferable to induce more errors? • Subjects’ Remarks • “More aware of the observers when I didn’t know what I was doing.” • “Am I going to flunk?”

20. Putting it into Practice • Evaluation apprehension is subconscious • Regardless of how much you try to put the user at ease, the feeling of evaluation is still present subconsciously • How should we interpret results when “White Coat Response” effects performance? • Subjects’ Remarks • “Out of sight, out of mind” • Didn’t mind having the facilitator there, because she had met me, but the other observers made her nervous

21. Putting it into Practice • The social-psychological context of the usability test can have a significant impact on a user’s performance. • Other psychological factors that may impact usability results • Different Lab Set-ups • Gender differences between the facilitator and the user • Different introductions into the process • Our study only measured user performance • Measure the user’s state anxiety