User research for information-rich domains

User research for information-rich domains Rashmi Sinha, Consultant & Researcher-at-large Uzanto Consulting www.rashmisinha.com www.uzanto.com AIfIA Leadership Seminar, IA Summit 03, Mar 21

Special user research methods for information-rich domains. Why? • Such domains offer unique challenges and opportunities. • design of cell phone vs. design of navigation for online cell phone advisor. • Important to understand information representation in user’s minds. • Semi-structured methods borrowed from cognitive anthropology and psychology can be of use.

Project Vision • A portfolio of methods to probe user information representations and needs. • Reduced time for user research. • Same data collection phase for user categorization and persona creation. • Reuse of results. • Suggestion not prescription. • Reliable methods.

Two key questions for user research • Part 1: User Categorization • Scope & boundaries of the information domain • Structure of information domain • Differences between groups of people (different user groups, different cultures, stakeholders) • Part 2: User information needs • Are some needs more important than others? • Can users be differentiated into groups on the basis of such needs? Can this grouping be used to form personas?

Part 1: Understanding the categorical structures of the mind • Overview • Why people categorize? • The structure of semantic memory • Is understanding user categorization important for IA’s? • Methods • Free-listing. • Categorization Methods such category allocation & verification. • Card Sorting and alternatives. • Analysis of sorting data. • Identifying dimensions or facets.

Why people categorize • Cognitive Economy • Russian Psychologist Luria’s case: “Mind of a mnemonic”; Borge’s “Funius the memorius”. • Allow generalization from past experiences • By treating something as a kind, we can use what we have learned from other examples of that kind. • Facilitate communication • Describing something as a kind of thing may obviate the need to provide details of the thing itself. • Permit different levels of abstraction • difference between objects and ideas, animals and plants, dogs and cats, and terriers and collies.

Is understanding categorization useful for Information Architects • Direct use: when user categorization informs IA. • Indirect use: good to have broad understanding how users think about product even when user categorization does not directly inform IA. • Context is important, getting whole scheme right more important than specific category or node. • Important to remember: • Categorization is not static. People are good at learning new categories. If you provide the context and the right examples, they can learn new categories or alter boundaries of old categories.

Should interfaces always reflect user categories faithfully? • No. • Categorization is far too important to depend only on what user thinks. • Should also be influenced by business proposition, strategy, brand etc. • Different user groups might differ in their perception of domain. No one scheme can serve them all perfectly. • Depends on the type of site. More important to get categorization right for an architectural image database, than a small exclusive retail store e.g., Red Envelope which can push its own vision. • User research should provide alternatives rather than the one answer, allowing information architects the freedom to make choices.

Structure of human memory and how categorization fits in • Implicit / Explicit distinction • Short term / working memory limitations • Categorization is implicit • Terms: categorization and semantic memory used interchangeably

Animal (has skin, can move, eats) Fish (swims, gills, fins) Birds (wings, flies, feathers) Canary (sings, yellow) Ostrich (tall, does not fly) Salmon (edible, pink) Shark (dangerous, gray) The structure of semantic memory • Semantic distance • Hierarchical structure • Graded structure (some members are more prototypical than others) • Semantic organization builds up user expectancies regarding how things are organized • Go along with expectancies to help users understand your IA. • Violate these expectancies and you leave the user lost and confused.

Not all categories follow these rules: Goal directed categories • Example: “things to wear in winter to keep warm”. • Central tendency does not play a role in the graded structure of goal directed categories. • Graded structure is organized around ideals. For instance, exemplars are better members of category: “things to wear in winter”. • The best example of this category (down jacket) is not the exemplar that is most like other category members; rather, it is the exemplar with the most extreme value on the goal-related dimension.

Individual differences in categorization Sushi • Individual differences in categorization • Factors such as experience, learning, culture have an impact. • There is enough consistency across people for effective design. • Degree of closeness between items might vary between people, but we mostly recognize the idiosyncratic association made by others. Shark Person 1 Fish Aquarium Diving Fish Grilled Salmon Person 2 Shark

Animal (has skin, can move, eats) Fish (swims, gills, fins) Birds (wings, flies, feathers) Canary (sings, yellow) Ostrich (tall, does not fly) Salmon (edible, pink) Shark (dangerous, gray) Semantic distance as basis of categorization • At root of all categorization techniques is question: “How far is A from B?” • Proximity / similarity matrix can help visualize user semantic networks (by using cluster analysis and other statistical techniques).

Semantic Distance can be plotted as a similarity matrix (Table 1 in handout) • Also called proximity matrix. • One type of similarity matrix is a correlation matrix (look in Excel guide for instructions for creating correlation matrix) Self correlation

Plotting similarity / card sorting data: Cluster Analysis for dataset about cars Continental Buick Cadillac Mercedes Corvette Jaguar Firebird Camaro Monte Carlo Capri Chevy Vega Dart Volkswagen • Cluster Analysis • Advantages • Suggests a structural solution • Disadvantages • Prescriptive • Averages over different dimensions

Alternative method of analysis: Multidimensional Scaling (MDS)

Advantages of Multidimensional Scaling (MDS) • Hints at possible solutions rather than prescribes. Tells you the possibilities, leaves specifics of solution to you. • Dimensions (axis) can suggest facets. • Similarity maps are easy to understand • Helps identify what dimensions are important • cluster analysis can be done on results of MDS. This gives cleaner results. • Cluster Analysis and MDS are complementary

Part 1: Understanding the categorical structures of the mind • Overview • Why people categorize • Is understanding user categorization important for IA’s. • The structure of semantic memory • Methods • Free-listing • Categorization methods such category allocation & verification. • Card sorting and alternatives • Analysis of sorting data • Identifying dimensions or facets

Free-listing exercise to explore domain scope and boundaries • Goals • Explore boundaries and scope of domain across a group of people. • Gain familiarity with user vocabulary for the domain. • Use as a precursor to card-sorting, to define and limit the domain, and frame card items in the user’s language. • Method • Can be conducted as part of interview, or as written exercise • Ask respondent, “Name all the x's you know.” Give sufficient time to do so. • How many respondents? • Depends on how much agreement there is about the domain. more agreement > fewer respondents. • Stability of average rank order is a good index.

Free-listing Exercise • Think of the menu at Wendy’s. Write down all the menu items you can think of on a piece of paper. Take one minute to do this. • Some items start repeating. • Some items occur earlier in the list.

Data from free-listing exercise about McDonald’s menu (Table 2 in handout) User No 4 Chicken Mcnuggets Cheese burger Bacon cheese burger French fries User No 5 Hamburger Quarter pounder Big mac Chicken fajita French fries Apple pie User No 6 French fries Soda Big mac Quarter pounder Cheese burger Hamburger Double cheese burger Happy meal Chicken Mcnuggets Mc Flurry Ice cream Cookies User No 1 French fries Cheese burger Shake Hamburger Cheese burger French fries Chicken sandwich Chicken Mcnuggets Fish sandwich Shake Hamburger User No 2 French fries Chicken Cheese burger Shake User No 3 Hamburger Cheese burger French fries Mc rib Chicken sandwich

Analysis of data (Table 3 in handout) • Create a list of all items, sorted by their average rank (of being listed by a respondent). Examine how that rank order changes with the addition of each new respondent. If the ranks are relatively stable, then you can stop adding new respondents.

Divide items into 3 concentric circles (use your own break points): Periphery Middle Core Concept structure • Plot items according to frequency of mention

Similarity from co-occurrence • How many times did two items occur in the same free-list? • Assumption of similarity if same person mentions both • Crude similarity metric.

Other uses for free-listing • Comparing cultural or other group differences • How do two groups perceive the same domain? • Comparing two domains • How does perception of McDonald’s menu compare with Wendy’s? • Segment respondents into types based on familiarity: • Find respondents with greater domain familiarity or those who perceive domain in idiosyncratic fashion? • Recursive free-listing to explore concept structure

Variants of free-listing: Semantic Association • To find out what lies nearest to target items. Helpful in understanding problem categories and items. • Exploring at same semantic level, rather than going deeper. • Semantic Association • What words do you associate these target words with pasta: _____ _____ _____ ______ _____ _____

Part 1: Understanding the categorical structures of the mind • Overview • Why people categorize • Is understanding user categorization important for IA’s. • The structure of semantic memory • Methods • Free-listing • Categorization methods such category allocation & verification. • Card Sorting and alternatives • Analysis of sorting data • Identifying dimensions or facets

Category Generation • Goal: to understand the overall categorization scheme • Method: Open card sort • Users given items. Asked to create categories • Options: • Provide total number of categories to be created (avoid problems with splitters and lumpers). • Successive card sorts. • Ask for labels for each grouping.

Category Allocation • Goal: to understand goodness of existing information architecture and labels • Method: Closed card sort • Users given items and category labels. Asked to place each item in a category. • Do not allow creation of a miscellaneous category. • Useful for: • Understanding user categorizations when category labels are a given • Refining existing categorization scheme. • Options: • Allowing items to belong to multiple categories. • Providing category descriptions rather than category labels.

Category Verification • Goal: test goodness of an existing categorization scheme. • Method: Speed verification tests • Create the navigation scheme (the hierarchy / facets etc). Test this scheme by asking users to find items from various levels of the hierarchy in a short time. Note time, number of steps and mistakes. Compare these metrics with another scheme. • Search for utility called “Classify”. • Useful for: Comparing two designs. Testing overall goodness of an existing scheme.

Example: Card Sorting exercise for online travel guide • Example: Designing an online travel guide to help users plan their trips to various locations. • Purpose of card sort: • How to structure the website to help users find travel information, and create personalized travel guides. • Items include • lodging, entertainment, local information, When to Go, Travel by Car/Air/Bus, Music Events, Hiking, Day Trips, Skiing, Diving, Golf, Emergency Info.

Alternative for open card sorting: Online survey software • Each item (card) occupies a row • Each potential (unnamed category) is represented by a column • Disadvantage: Cumbersome with large number of categories Open card sorts: When user generates categories

Alternative for closed card sorting • Each item (card) occupies a row • Each category is represented by a column • This can be done on Excel as well Closed card sorts: When user works with given categories

Simple data analysis for card sorting • Look at data, one user at a time • At a glance you can look down a particular column, and see what items belong in it. • Also possible to create hierarchies based on user created categories

Advantages and disadvantages of using online survey software for card-sorting • Advantages • Easy to carry out, no software to install and figure out. Can be done remotely using online survey software. • Easy to get data into Excel (or your favorite application). • Can be used to conduct either open / closed card sorts • Can also be adapted to individual / group sessions • Disadvantages • Feels less intuitive to participants • Cumbersome with large number of categories

Options with card sorting and what to from future research • Options • Should each item be in only one category? • Should card sorts be done individually or with groups? • group card sorts provides an energy and enthusiasm that makes light work of an otherwise tedious task. • group card sorts can overshadow individual differences, highlighting opinion of a few. • Combining card sorting with importance ratings? • Future • Analyze differences between user groups, different cultures etc. • Individual Differences: How much consensus is there between individuals? • Stakeholder analysis. • Reuse of domain concept maps.

Identifying dimensions: Method 1, 20 questions • The user tries to guess what the item is, by asking about aspects or facets of the object. Facilitator can only answer yes or no. (method proposed by and example from Margaret Hanley on the faceted classification list) • Example: Identify facets for microscopy images • Questions asked: cellular / non-cellular, animal or not • Facets • Species • Type of electron microscope • The magnification of the microscope • The biological process being demonstrated • Part of the organism or cell being shown • The method for taking the image

Identifying dimensions: Method 2, differentiating items • Randomly select two items from your list. Ask user what differentiates them. Repeat question with other pairs till dimensions start repeating frequently. • What differentiates pair below. List one or more ways that they are different. • Pasta and rice: _______ ______ ______ • Plane and train journey: _______ ______ ______ • Good way to explore domain you are unfamiliar with. Some of the pairs will sound ridiculous!

Exercise: Identifying dimensions • 20 questions (5-6 minutes) • Middle person at each table, think of a favorite movie. Movie should be a well known one. Ask partners to guess by asking yes/no questions. • Not important if they get the correct answer or not. • Identify dimensions via questions • Pair-wise differences (5-6 minutes) • Choose one of the pairs of recipes • List differences and similarities • Identify dimensions • Does listing differences work better than listing similarities?

Identifying dimensions: Method 3, Multidimensional Scaling • Note: add text here

Conclusion of Part 1 • Overview • Why people categorize • Is understanding user categorization important for IA’s. • The structure of semantic memory • Methods • Free-listing • Categorization methods such category allocation & verification. • Card sorting and alternatives • Analysis of sorting data • Identifying dimensions or facets

Part 2: Understanding user needs, creating personas • Are some needs more important than others? • Can users be differentiated into groups on the basis of such needs? • Can this form the basis of personas? • Personas as “User Archetypes” who represents needs and goals of many other users. • Using “Personas” facilitates design for one specific person. • Who should that one person be? How should that person/s be identified? • Not the average user • Not a real user • Derived from background user research (interviews etc.)

Current methods for persona creation • Method • Conduct interviews with various stakeholders. • Find patterns. • Pick a nugget, interesting tidbit and build persona around it. • Problems with method: • Interviews not economical way to find representative users. • No tight coupling between user research & personas. • Would two designers using same user research create same personas? • This method works well when you have time on your hands, resources to interview many people, and skilled persona developers.

Will market segmentation techniques prove useful? • Market segmentation is used to identify clusters of people product can appeal to. • Mostly on basis of demographics. • Sometimes on the basis of psychological variables (also called psychographic variables). • Techniques can forecast marketplace acceptance of products and services. Can also help convince executives to build product. • Such techniques often not helpful in defining product

Deconstructing marketing techniques • Questions focus on like / dislike of product concept • what do you think of vanilla coke or green Heinz ketchup? • Statistical techniques used in market segmentation cluster users according to demographic variables, not according to user needs from products. • Concern themselves with product at a high conceptual level. For interaction design, one needs to focus on the specifics of the interaction: How will product satisfy needs / goals.

What is needed is a method that will… • Ground the personas in reality. • Focus on motives/needs of users. • Complementary to existing methods e.g., interview, observation. • Concrete series of steps that any designer can follow to develop personas. • Reliable: When used by other designers, at other times will lead to similar personas.

Creating personas for a Bay Area restaurant finder • Goals of site: Become the premier site for Bay Area residents & visitors to find restaurants of their choice. • Design quick restaurant-finders (a la product advisors) apart from conventional advanced search / browse options • Method: • Step 1: Preliminary phone interviews • Step 2: Survey of User Needs • Step 3: General data analysis • Step 4: Identify user archetypes • Step 5: Add details to personas through interviews and observation (optional)

Step 1: Preliminary phone interviews • Goal: Initial exploration of the domain. Identify questions that need to be asked for next stage. Get acquainted with user vocabulary for a domain. • Phone Interviews: Ask people to describe a specific restaurant experience. Note how they chose the restaurant, what factors they indicated as important. • (Note: related to concept of schemas and scripts from social psychology) • Review of other similar websites to identify factors that are important. • User free-listing?

User research for information-rich domains