1 / 39

Statistical User Research IP w

Statistical User Research IP w. Jenny Huang, PraisePop. Outline. Introduction Statistical Analysis for A/B Testing Standardized Usability Questionnaires Usability Testing – Plan Usability Testing – Data Collection Usability Testing – Graphical Display Usability Case Studies.

sjim
Download Presentation

Statistical User Research IP w

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Statistical User Research IPw Jenny Huang, PraisePop

  2. Outline • Introduction • Statistical Analysis for A/B Testing • Standardized Usability Questionnaires • Usability Testing – Plan • Usability Testing – Data Collection • Usability Testing – Graphical Display • Usability Case Studies

  3. Introduction to User Research Week 1

  4. Quantifying User research • “User research is the systematic study of goals, needs, and capabilities of users so as to specify design, construction, or improvement of tools to benefit how users work and live.” – Schumacher, Ph.D., User Researcher

  5. Usability Testing • Usability – “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”  • Things to Measure • Completion Rates • Task Time • Errors • Satisfaction Ratings

  6. SampLing • Sample Size • Though it is common belief that we need large samples to use statistics to interpret data, in user research, a small sample can still inform  design decisions. • Stratified Sampling • It is more important to choose a small sample representative of the target market than a large sample that does not represent it. 

  7. A/B Testing  • Method for comparing alternate designs • Users randomly work with one of two deployed design alternatives and the data differences are analyzed to tell whether they are statistically significant • Report button designs • Goal: To find a button design that makes reporting clear, encourages users to filter negative posts, and does not reduce enjoyability of the app • Random Name Generator button designs • Goal: To find a random name generator that encourages users to click it and also use the name it gives • Different menu designs • Goal: To find a menu design that is easily navigable and engaging for users • “Was this post sweet or salty?” (pop-up vs. no pop-up) • Goal: To determine whether the benefit of reducing negativity outweighs the annoyance users experience from the pop-up

  8. Is There a Statistical Difference Between Designs? Week 2

  9. Chapter 5 of Quantifying the User Experience • “Is the difference between designs greater than what we’d expect from chance?” •  A/B Testing • Overview • t-test (comparing means), chi-square test of independence (comparing proportions) • Find p-value • Determine if statistically significant

  10. Standardized Usability Questionnaires Week 4

  11. Chapter 8 of Quantifying the User Experience • Standardized Questionnaire – form designed to assess participants’ satisfaction with the perceived usability of products • SUS • developed in the mid 1980s to become a popular questionnaire • 10 items, five scale steps, odd numbered items have a positive tone while the even numbered items have a negative tone • shown to be most sensitive post-task questionnaire

  12. Planning Usability Tests Week 5

  13. Essential Elements • Scope • Purpose • Schedule & Location • Sessions • Equipment • Participants • Scenarios • Metrics • Quantitative Metrics (Planning guidelines fromusability.gov.)

  14. Scope • Indicate what you are testing: Give the name of the Web site, Web application, or other product. Specify how much of the product the test will cover. • We are testing PraisePop v1.1 (updated June 2015). The test will cover navigation and usability.

  15. PurPose Identify the concerns, questions, and goals for this test. These can be quite broad or quite specific. In each round of testing, you will probably have several general and several specific concerns to focus on. Your concerns should drive the scenarios you choose for the usability. • Can users easily scroll through the feed and read the posts? • Can users easily find how to write a post? • Can users easily write a post to a specific person? • Can users easily send a post to the feed? • Can users easily pop the kernel? • Can users easily report a negative post? • Can users easily find how to delete posts about themselves or one they wrote? • Can users easily find the time and date a post was written? • Can users easily respond with the pre-set messages on the post? • Can users easily export posts to social media, email, and messages? • Can users easily find the settings? • Can users easily sign up? • Can users easily confirm their emails? • Can users easily sign in? • Can users easily log out? • Do users see that they can swipe right to see stats about the community feed? • Do users know how to user the random name banner? • Do users know how to use the random name generator button?

  16. Schedule & Location • Indicate when and where you will do the test. If you have the schedule set, you may want to be specific about how many sessions you will hold in a day and exactly what times the sessions will be. • Library (5/5): 5-5:30pm, 5:40-6:10pm, 6:20-6:50pm • Day Hall Dorm (5/5): 9:30-10:00pm, 10:10-10:40pm

  17. Sessions • You will want to describe the sessions, the length of the sessions. • Sessions will occur at the specified times, asking users to fill out a background questionnaire, do the tasks, and afterwards, fill out the SUS.

  18. Equipment • Indicate the type of equipment you will be using in the test; desktop, laptop, mobile/Smartphone. • questionnaires and pencil • Phone with PraisePop app • Laptop to record data

  19. Participants • Indicate the number and types of participants to be tested you will be recruiting. Describe how these participants were or will be recruited. • 5 participants • Stratified Sampling • at least one participant from each grade • at least 3 males and 3 females • Convenience Sampling • In the library and dorm, I will just find people there who are there and willing to participate 

  20. Scenarios • Indicate the number and types of tasks included in testing. Typically, you should end up with approximately 8 (+/- 2) scenarios for a mobile/smartphone test. You may want to include more in the test plan so the team can choose the appropriate tasks. • Write a post to John Palfrey complimenting him on his ASM on Wednesday. • Delete the post you wrote. • Upvote a post. • Report a post. • Find the time and date a post was written. • Respond to a post in the feed with a pre-set message. • Export a post to iMessage. • Find your and the community’s stats in the feed. • Find the rules of PraisePop. • Find two ways to choose a random name of a person to write about. • Log out of the account.

  21. Subjective Metrics • Include the questions you are going to ask the participants prior to the sessions (e.g., background questionnaire), after each task scenario is completed (ease and satisfaction questions about the task), and overall ease, satisfaction and likelihood to use/recommend questions when the sessions is completed. • Background Questionnaire • Name? • Grade? • Gender? • Experience using PraisePop? • iPhone or Android phone?

  22. Quantitative Metrics • Indicate the quantitative data you will be measuring in your test (e.g., successful completion rates, error rates, time on task). • I measure: • Task time • Error rates (error = 1, no error = 0) • Participant writes down: • Completion rates (success = 1, fail = 0) • Satisfaction ratings • “It was easy to complete this task.” (1-5) • At the end:  • SUS questionnaire

  23. SUS

  24. My Directions • Explain the test: “Hi, my name is Jenny and I am part of the PraisePop team. Today, you will be participating in a usability test that will help us understand how to make PraisePop better. You will follow certain tasks that I ask you to do and rate your satisfaction. While you are completing tasks, I will be collecting data like task times and error rates. To begin, please fill out this questionnaire.” • Give participant questionnaire (page 1 of Questionnaires) • “Are you ready for your first task?” • Give tasks to complete and time them and count the errors • After each task, have them rate their satisfaction on the scale (page 2) • At the end, have them fill out the SUS (page 3) • Thank them for their time!

  25. Usability Tests Data Collection Week 6

  26. Results • Five participants (2 males, 3 females; 4 seniors, 1 upper) • Four of the users had never used PraisePopbefore • The five SUS scores obtained were: 72.5, 95, 87.5, 97.5, 97.5. • These give a sample mean of 90. • This score actually ranks in the A+ range of the Suaro/Lewis SUS grading curve. Yay PraisePop!! • Bias • Convenience sampling • Sample not representative of population

  27. Measurements

  28. Data collection with unfamiliar subjects Week 7

  29. Usability case Studies Week 8

  30. THANK YOU FOR LISTENING!

More Related