Needs Analysis: Design through Discovery

1. Needs Analysis:Design through Discovery Philip Kortum, Ph.D. Robert Bushey, Ph.D. SBC Laboratories Human Factors

2. Today’s Agenda • Who we are (5 min) • Basic needs analysis techniques (60 min) • Virtual Lab tour (10 Min) • Break (10 min) • Needs analysis in high-data environments (70 min) • Conclusions (10 min)

3. Who is SBC? • Short Answer: • We’re the phone company • Long Answer: • 2nd largest telecommunications providers the US • Wireless • Long Distance • Data • Residential/Business phone service • Over 60 million access lines • $41 Billion in Revenue (# 27 on the Fortune 500)

4. Technology Focus Areas Who is SBC Laboratories? SBC Laboratories is the Applied Research and Development arm of SBC. About 200 Advanced-degreed Scientists and Engineers work at SBC’s laboratories in Austin and the Bay area in California. Broadband Architecture, Infrastructure, & Services Intelligent Networks Internet Wireless Systems Information Technology

5. SBC Laboratories Human Factors Engineering

6. Where today’s talk fits in this class From Mayhew, 1999

7. What is Needs Analysis? Needs analysis is the method of uncovering user requirements through direct or indirect interaction with that user

8. Why do we perform needs analysis? • We are not the user • Users may not know what they really need • User is unable to articulate the need • True needs are often masked • Good match between needs and implementation leads to superior efficiency and usability

9. How is Needs Analysis different than Task Analysis? • Needs Analysis – trying to uncover the underlying motivations • Task Analysis – trying to uncover the procedural steps

10. Task Analysis Setting up a VCR Programming a VCR Needs Analysis The ways a person uses a VCR and the reasons behind that use What they record What they do with what they record Time-shifting Remote control use use How Is Needs Analysis Different Than Task Analysis?

11. How Is Needs Analysis Different Than Marketing? • They both deal with the user • They both try to determine what the user ‘needs’ • Some of the techniques are similar • Some of the information collected is the same

12. Marketing Demographics Where people live, shop How much they make Physical attributes Purchasing behavior patterns What people will buy Price points Needs Analysis User behavior patterns What people do What users know Users environment Users mental models How Is Needs Analysis Different Than Marketing?

13. Marketing How much money do the users have to spend? What other similar devices do they have? What kinds of technology do they own? How much do they use these technologies Needs Analysis Why would they want to use an image recorder? How do they use current devices? When do they use such devices? What other technology interactions are there? Needs vs. Marketing ExampleA device that records off of the television

14. A Simple (if unfair) Characterization: Marketing deals with wants Needs deals with needs

15. A word about Semantics…. • Many techniques are similar and are called different things by those who use them (even within the usability community) • Good practitioners almost always use a mix of techniques, and often call these mixtures by a new name.

16. Needs Analysis Can be very focused and limited in scope Doesn’t have to ‘big sky’

17. Basic Types of Needs Analysis • Questionnaires • Interviews • Observational Research • Hybrid methodologies

18. Questionnaires “A set of questions for obtaining statistically useful or personal information from users”Webster

19. Questionnaire Advantages • Easy to administer • Inexpensive • Can be administered by untrained staff • Can be administered remotely • Can have a relatively large number of questions and data samples • Can more easily target highly specific users • Can branch to reliably capture pertinent information • Data can be simple to code

20. Questionnaire Disadvantages • Difficult to construct a good test instrument • Must know a priori exactly what you want to ask • No follow-up questions • Question intent may be open to interpretation • Can result in unknown bad results • Open ended responses highly variable • Open ended questions are difficult to code • Validation/verification difficult if study is not closely controlled

21. Questionnaire Disadvantages: Fixes • Difficult to construct a good test instrument • Follow scientifically designed questionnaire guides (Kirakowsi, Gillham, USARI) • Test for validity, reliability, repeatability • Must know a priori exactly what you want to ask • Conducted pilot tests to try out the questions • Conduct interviews to help build a reasonable first pass • Use domain experts to help capture detail questions

22. Questionnaire Disadvantages: Fixes • Question intent may be open to interpretation • Conduct pilot tests to check for weaknesses • Use experts and novices to examine interpretation differences • Use unambiguous, quantifiable, anchored language

23. Ambiguous Question Example How bad was your last car accident? Really bad bad not really bad not bad at all 2 respondents, who both dented the front bumper slightly, no injury Respondent 1: ‘Really Bad’ Respondent 2: ‘Not bad at all’ “I’m only 16, it’s my first accident and it was my dad’s new sports car, and I wasn’t on the insurance for the car. I’m going to be grounded for life! “Well, compared to that 23 car roll-over collision I had with that semi last month, this was nothing. Not to mention that I got the collision waiver on the rental! Class: How could we fix this?

24. Questionnaire Disadvantages: Fixes • Open ended responses highly variable • Provide hints, examples to guide the responses • Use multi-part questions that are more precise • Open ended questions are difficult to code • Use more choices, or more questions, to remove need for open ended questions • Focus on variables of interest and use ‘other’ to capture low probability events that are not pertinent • Devise keyword codebooks to help code open ended data

25. Questionnaire Disadvantages: Fixes • Validation/verification difficult if study is not closely controlled • Key code questionnaires to match user and instrument • Lock-out multiple take attempts • Ask qualifying questions as part of a pretest, or branch

26. Interviews “A meeting at which information is obtained from a person” Webster

27. Interviews • Generally classified into two types: • Structured • Unstructured

28. Interview Advantages • Small numbers of interviews can be performed quickly and inexpensively (although not always) • Can be conducted remotely • Can be conducted in groups or individually • Can be conducted with little notice, if needed • Can change the interview for different levels of users

29. Interview Advantages • Open for opportunistic data discovery • Get non-verbal clues to help guide the interview • Good technique for ‘fishing expeditions’ • Useful for gathering preliminary information to guide later needs analysis

30. Interview Disadvantages • Hard to conduct a ‘good’ interview • Easy to lead the respondent, with both verbal and non-verbal cues • Easy to unknowingly ask loaded questions • “How does it feel to be ugly?” • Harder still to conduct a good group interview • herd mentality, strong leader • Hard to stay ‘up’ and focused for a large number of interviews

31. Interview Disadvantages • Data can be difficult to code and quantify • Can be large differences between interviewers • Expensive and time consuming to conduct large numbers of interviews • Usually a low upper limit to the number of questions

32. Interview Disadvantages Interview Example

33. Interview Disadvantages • Interview data may not match reality Question What was said What the data suggests How many offers do reps make per call? Reps make 1 offer per call Almost no reps make 1 offer, some do 0 some do 2+ What kind of rep is the least desirable? ‘order takers’ are the least desirable Order takers are good - they drive accessibility and are among top $ performers What makes the customer happy? Taking lots of calls is what makes customers happy No supporting data

34. Interview Disadvantages: Fixes • Hard to conduct a ‘good’ interviews • Use scientifically developed methods (i.e. Weiss’s Learning from Strangers) • Pre-develop a core set of questions • Manage group dynamics • Use hypothetical questions to elicit more detailed answers • Hard to stay ‘up’ and focused for a large number of interviews • Limit the number of interviews • Use tag teams, alternate sessions • Use well trained ‘people persons’

35. Interview Disadvantages: Fixes • Data can be difficult to code and quantify • use core set of questions • use data code books to help quantify responses • use affinity diagrams to aid in coding • Can be large differences between interviewers • use single interviewer • use visible or hidden teams • conduct pilot training to level the interviewers

36. Interview Disadvantages: Fixes • Expensive and time consuming to conduct large numbers of interviews • Usually a low upper limit to the number of questions • target exact population • supplement with other techniques • use remote techniques • use core question set, based on pre-test • Interview data may not match reality • Always verify data

37. Observational Research “An act of recognizing and noting a fact or occurrence…” Webster

38. Observational Research • Observation doesn’t have to occur in real time • Observation doesn’t have to be visually based • audio recordings • diaries • telemetry (GPS, web logs, biometrics, equipment monitors) (e.g. car snooper)

39. Observational Advantages • Good for understanding complex needs • Results are not defined by the design of the method • Capable of capturing unknown or undocumented behaviors • Gather data on specific artifacts • Can uncover inter-dependencies • If documented appropriately, can review observations as many times as necessary • Can parse expert/novice distinction

40. Observational Disadvantages • Some phenomenon is too infrequent to catch • Heisenberg Principle (i.e. driving tester) • Can be more difficult to obtain a reviewable record • Desired behavior may be interspersed with other unrelated activity • Slow, expensive to collect data • Can easily get ‘off-target’ • Data set can be overwhelming

41. Observational Disadvantages: Fixes • Some phenomena are too infrequent to catch • use high sample rates to find low probability events • ‘Create’ low probability (disaster training) • sample in areas known to have these events • be lucky • Heisenberg Principle (i.e. driving tester) • observe for extended periods of time • use remote measuring techniques • use low/no interaction models • clearly understand the cost-benefit of direct interaction • work in the users environment

42. Observational Disadvantages: Fixes • Can be more difficult to obtain a reviewable record • use alternate recording technologies • work in teams • Use advanced behavioral software (e.g. Noldus Observer) • Desired behavior may be interspersed with other unrelated activity • use specific artifact techniques • use remote data collection techniques to time compress • don’t assume that unrelated activity is actually unrelated (e.g. using the restroom while waiting)

43. Observational Disadvantages: Fixes • Slow, expensive to collect data • use remote or automated collection where possible • understand required sample size • carefully select person/situation/setting to be observed • Can easily get ‘off-target’ • use pilot observations to help identify key behaviors • be open to the fact that off-target behaviors may be pertinent

44. Observational Disadvantages: Fixes • Data set can be overwhelming • use data reduction/consolidation techniques • training samples • affinity diagrams • physical flow/sequence structures

45. Observational Research ExampleWatching Television

46. Observational Research ExampleWatching Television Nathen, et al, 1985

47. Hybrid Methodologies • Ethnography • Contextual Inquiry • Empathic Design • Participatory learning • actors • rotational management assignments

48. What happens when needs isn’t done?

49. What happens when needs isn’t done?

50. Questions?