1 / 34

Evaluating Your Prototype with Users

Evaluating Your Prototype with Users. Class 30. Agenda. 3:00-3:05 Announcements 3:05-3:50 Evaluation with Users. Overview of Evaluation Methods. With tasks. Without tasks. Without users. With users. Testing with Users. Task-specific think alouds wizard of oz Task-independent

nova
Download Presentation

Evaluating Your Prototype with Users

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating Your Prototype with Users Class 30

  2. Agenda • 3:00-3:05 Announcements • 3:05-3:50 Evaluation with Users

  3. Overview of Evaluation Methods With tasks Without tasks Without users With users

  4. Testing with Users • Task-specific • think alouds • wizard of oz • Task-independent • alpha and beta testing • Bugs vs usability

  5. Conducting a User Study • Planning • Preparing • Doing • Analyzing

  6. The Challenge • Within 2 miles there are no playgrounds, safe areas to run around, or supermarkets. But there are fried food joints. • Within 5 miles there is a supermarket, but no playgrounds or YMCAs. • The schools in the immediate area got rid of gym, music, and art to get the resources needed to fulfill the No Child Left Behind requirements. Recently, they cut down on recess time for more classroom time! • Children under 5 have 3 of the 10 factors that go into heart disease risk (obesity, high blood pressure, and tobacco use - second hand). The single parents have 7 of the 10 factors! 5 mi. 2 mi.

  7. How can we solve this challenge?

  8. How can we solve this challenge? Exercise Nutrition Education Obesity High Blood Pressure Tobacco Use Low Tech: Pedometer, information cards, pamphlets High Tech: Pedometer or Mobile Phone Walking…in regular life Informed decisions about what you eat Just in time messages instead of lecture, nag, zzzz…

  9. Possible Ideas… My food phone Bones in Motion

  10. Planning: Developing a Task • Describes what the user wants to accomplish, not how • Describes a “complete” job • Describes who the user is • Very specific

  11. Example Task for our Challenge • Tammy wants to find out how many miles she walked today and see if she could increase her step count for tomorrow. She wants to see a map of where she walked and see if there are safe alternatives to increase her step count. • What? • Complete Job? • Who? • Specific?

  12. Example Task for our Challenge • To calculate how far you walked today, turn on your phone and walk throughout the day. To see how far you walked in total, go to the menu screen, select BMI, and view your results on the main BMI screen. • What? • Complete Job? • Who? • Specific?

  13. Develop a task for My Food Phone • What? • Complete Job? • Who? • Specific?

  14. Planning: 80% of usability problems are uncovered with 5 participants Proportion of Problems Uncovered Number of Participants in Study Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting, 291-294. Santa Monica, CA: HFES. Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 4, 457-468.

  15. Be mindful… • Environmental Factors • Know the rules & regulations • Visit the site • Be mindful of safety • Developing Relationships • Trusting relationship • Dress appropriately • Participant’s Ability • Be aware of comfort • Take sensitivities into account • Time Management • Design adaptable studies • Multitask Lessons Learned Conducting User Studies in a Dialysis Ward. Katie A. Siek and Kay H. Connelly. In Extended Abstracts of CHI 2006: Workshops - Reality Testing.

  16. How could this affect your data?

  17. How could this affect your data? How much useful data will you draw out from a hospital visit? I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older Adult

  18. How could this affect your data? How much useful data will you draw out from a hospital visit? I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older Adult Our entire healthcare system is based on the idea assuming that everyone receiving care in a hospital unit are 100% coherent and understanding of their condition…even if this may not be the case. - Doctor

  19. How could this affect your data? How much useful data will you draw out from a hospital visit? I was just in the hospital for 2 months and it was not a good time for me - I was disoriented and confused most of the time. [Hospitals] are not a good place for people to reflect. - Older Adult Our entire healthcare system is based on the idea assuming that everyone receiving care in a hospital unit are 100% coherent and understanding of their condition…even if this may not be the case. - Doctor Compromise: For this particular study, we visit the patients while they are in the hospital to see how care can be improved and 2-4 weeks after their hospital visit to give them time to reflect.

  20. How should information be given?

  21. Did you say 4th grade reading? SMOG (< 30 sentences): Count the total number of sentences in the material. Count the number of words with 3 or more syllables. Find the total number of sentences and the corresponding conversion number in SMOG Conversion Table II. Multiply the total number of words with 3 or more syllables by the conversion number. Use this number as the word count to find the correct grade level from Table I. • Harvard Health Literacy Studies: http://www.hsph.harvard.edu/healthliteracy/materials.html • MedLine Plus “How to Write Easy to Read Health Materials” http://www.nlm.nih.gov/medlineplus/etr.html • Doak and Doah http://www.hsph.harvard.edu/healthliteracy/doak.html

  22. Did you say 4th grade reading? SMOG (< 30 sentences): Count the total number of sentences in the material. Count the number of words with 3 or more syllables. Find the total number of sentences and the corresponding conversion number in SMOG Conversion Table II. Multiply the total number of words with 3 or more syllables by the conversion number. Use this number as the word count to find the correct grade level from Table I. Hello. My name is …, and I am a computer science Master's student. I am currently conducting a study for a Medical Informatics class project on smoking factors and what would motivate people to quit. I would like to thank you for participating in this study and would like to express my gratitude with this chocolate. For the following hour, I would like to ask you some questions about your smoking history and experience. After the question answer session, we will be doing a small 5-minute exercise. Please feel free to stop me at any time you wish to. There are no correct or incorrect answers to the questions, so please do not worry and express everything you want. If you have any questions, please do not hesitate to ask. You are helping me a lot by contributing your experience to my study. Any information you give me will only be shared between my teacher and I, and all data will be anonymized.

  23. Did you say 4th grade reading? SMOG (< 30 sentences): Count the total number of sentences in the material. Count the number of words with 3 or more syllables. Find the total number of sentences and the corresponding conversion number in SMOG Conversion Table II. Multiply the total number of words with 3 or more syllables by the conversion number. Use this number as the word count to find the correct grade level from Table I.

  24. Planning: What to Observe Qualitative Quantitative

  25. Planning: What to Observe • Process Data (Qualitative) • How humans go about completing tasks • Why a design does/does not work • Bottom Line (Quantitative) • Quantitative measures • Task completion times • Error rates • How well/poorly a design performs

  26. Preparing: What tools will you need?

  27. Preparing: Practice, Practice, Practice • Schedule participants and rooms • Prepare forms • NDA - Non-Disclosure Agreement • Voluntary informed consent • Plan the procedure • Pre-session survey • Tasks • which ones and in what order • adapt your tasks • Post-session: exit survey and debriefing

  28. Doing: Think Aloud Process • Reduce participant stress level • Your key behaviors • No excuses • No justifications for the software • Record observations • Note-taking • Audio/Video • Instrumentation • De-briefing • Ask questions you think of along the way • Give compensation (if possible)

  29. What would you do a think aloud on?

  30. What would you do a think aloud on? Low Fi High Fi

  31. Doing: Wizard of Oz What is it? When do we use it?

  32. Examples shown in Class or in Papers?

  33. How could we address our challenge with WoO method? …to give automatic feedback on food intake/calories… …to give just in time educational messages about tobacco use

  34. Looking Forward • Week 12 • EMRs (Chris) • Healthcare for Underserved • Presenting Data • Week 13 • MI Standards (Danish) • Guest Speaker Sheana Bull • NO CLASS – Users Studies • Week 14 – Fall Break • Week 15 • NO CLASS – User Studies • Examine User Study Data • What’s Next

More Related