Presentation Transcript

1. Findings: Baseline Usability TestingDecember 3, 2010

2. Objectives + Methodology

3. Objectives The central purpose of this study is to measure any change in usability between the current and redesigned sites. Data were gathered on the ease of finding and completing the most common tasks that surveys have shown are needed by different user-types. A near-identical protocol will be run for the Post-test after the Ramp redesign is launched into production. Additional goals are to: Inform Ramp’s redesign in terms of addressing common pain points and/or better align the interface to users’ needs. Inform future incremental improvements (bug-fixing, tweaks) to the site. Gather subjective reactions to the branding and look-and-feel. Explore ways to help data, products, information and applications meet priority users’ needs. Deliverables available for review include: This Powerpoint report of the findings. The script used for all interviews. The notes and raw data that were taken during the interviews. Links to digital recordings that may be streamed or downloaded. A digital “highlights video” editing together illustrative clips. Data (including PII) from the volunteers for the test.
4. Methodology Setting. Participants were individually interviewed on-site at a facility in the University District of Seattle. Their screen activity and video thumbnails of their faces were digitally recorded. Observers who did not interact with participants watched via a conferencing feed both on-site and remotely. Participants. The 30 participants who completed the protocol of 40 who were scheduled included: 8 job seekers recruited via a survey posted on Workforceexplorer.com 2 low-vision job seekers recruited via contacts at ESD and UW 9employers recruited via the survey and contacts near facility 7 ESD job counselors recruited via contacts at ESD 4 ESD business outreach managers recruited via contacts at ESD Protocol.First, participants’ eye-gaze patterns were assessed from starting points at the Home Page and Data Library with prompts such as “where would you look to find where to post a job opening.” Second, the same prompts were repeated while key usability metrics were recorded, including success (defined subjectively), paths, clicks (links and enter keys), time (in seconds), subjective difficulty ratings (1-5 scale where 1=easy and 5=difficulty), and any verbatim. Incentive. Job seekers and employers were thanked with a $50 Fred Meyer gift card; ESD staff received no gift.
5. Digital Recordings. Double click to activate. See Patty Henson for password.
6. Method: Metrics, Graphing, and AnnotationTask #: “Actual verbal prompts given to participants.”Sample: Who this graph is based on (N=number given task) Percent of N who completed task to their satisfaction. Percent of survey respondents who indicated this task is important to them. Avg. time to task completion in seconds. Includes successes and failures. Avg. number of inputs including mouse clicks or enter keys Redesign results are blank in the rest of this report; they will be filled in when redesign is complete. Avg. subjective rating of difficulty on 5 pt scale (5 most difficult). Includes successes and failures. Minimum clicks to success: this number indicates the click count on the shortest known path to success Common wrong paths…codes used include: mnmain navigation tabs at top of page; rnright navigation link; lnleft navigation link; bodylink found in the body of the page; > demarks clicks; search ‘string’ <back> example: body find labor market info >body numbers and trends > Representative verbatim: “statements made by participants during or after the task”
7. Tasks given to all participants
8. Task J10: Find out how much a specific job pays in your fieldSample: All Participants (N=26) Minimum clicks to success: 2 (pulldown Occupation Explorer OR rnjob seekers > body find detailed) Common wrong paths: body find labor market info > body numbers and trends > Representative verbatim: “very confusing (wages is used as a link or descriptor three times on home page)” “they all lead to the same place, but a lot of redundancy” “it didn't jump out at me, and what did didn't give me the information that I needed” “hmm, not finding search very helpful…seems weird to have to click to show search criteria”
9. Task J10: Find out how much a specific job pays in your fieldBreakout: By user type Baseline Redesign Job Seekers + Job Counselors Employers + Biz Outreach Job Seekers Job Counselors Employers Biz Outreach
10. Task J10: Find out how much a specific job pays in your fieldObservations & Recommendations Press release in top body is a low priority for core user types. Use real estate for identification of Who user is & What is needed, rather than right navigation. Tools pulldown overlooked: Only 1 of 10 JS used Make larger, do not place where search box is expected MariaM “Tools” term appears twice on page; use only where it points to a wizardy like application. Replace “Occupation Explorer” with a more self-explanatory term: few gazed at or chose it term is similar to whole site key data found here; surface “Wages” term appears 3 places on page: Users confused which one is right Sabrina, Andrew, Pam, Robert. None of these links are as good as Occupation Explorer. Video: Hildo16:15
11. Task J11: Find out which jobs are increasing/decreasingSample: All Participants (N=21) Minimum clicks to success: 2 (pulldownOccupation Explorer) Common wrong paths: bodyfind labor market > body numbers and trends> rnnumbers and trends> pulldowntools – industry trends search field “demand decline” < back to home > Representative verbatim: “(on wilma) again, more scrolling seems like there would be something up here (top) that I could type and it would find it for me” “i don't see anything for counseling, maybe it's under mental health” “but it doesn't tell me why this is a demand industry
12. Task J11: Find out which jobs are increasing/decreasingBreakout: By user type Baseline Redesign Job Seekers + Job Counselors Employers + Biz Outreach Job Seekers Job Counselors Employers Biz Outreach
13. Task J11: Find out which jobs are increasing/decreasingObservations & Recommendations Video: Pam13m Avoid multiple links to same place: Rename “Career Center;” taken by 0 job seekers or employers Also, the list under the link does not match the page it goes to in copy or sequence Avoid long non-linked list of content you’ll find under body links…instead improve navigation. Demand/decline term little understood by job seekers Make “Occupation Explorer” easier to find: This is the most expedient link Overlooked here and in Tool… pulldown. Has the same (better?) data than the Demand/Decline list. Users tended to take links with the word “trends” in them for this task. Gary, Pam, Mary, Mike
14. Task J11: Find out which jobs are increasing/decreasingObservations & Recommendations Wilma look and feel disorienting. Called “jocular” Andrew back button not working Maria Lots of scrolling. “Decline” way below fold, making it likely that they don’t see contrast to “Demand.” Users want data to validate the labels here…as found on Occupation Explorer. Chris, Rhea, Colleen.
15. Task J12: Find the unemployment rate for your countySample: All Participants (N=24) Minimum clicks to success: 2 (pulldown Local Area Profile) Common wrong paths: rnnumbers and trends > imagemapking < back > body LAUS < back> Representative verbatim: “I don't see anything for my county, it would seem like they would have a list for my county” “not sure that these graphs are helpful to me”
16. Task J12: Find the unemployment rate for your countyBreakout: By user type Baseline Redesign Job Seekers + Job Counselors Employers + Biz Outreach Job Seekers Job Counselors Employers Biz Outreach
17. Task J12: Find the unemployment rate for your countyObservations & Recommendations Video: Colleen44:30 Regina15m Page after this link has no county selector. Despite “Local Area” in name. Similarly labeled link below fold in right navigation DOES take you to country selector. Avoid similar links that take you different places. Below PDF map with unemployment by county found via Google search – surface this better. search term “unemployment rate king county washington.” http://www.workforceexplorer.com/admin/uploadedPublications/3672_statecounty.pdf
18. Task J13: find “green jobs” informationSample: All Participants (N=16) Minimum clicks to success: 2 (body Find Publications & Reports > PDF) Common wrong paths: body find publication reports > lnsearch “green jobs” > “green economy initiative” PDF > close PDF Representative verbatim: “if they really wanted to stress that they would probably have a green button” (green jobs) “could be more prevalent on the home page” “there were some articles (in search results) that seemed unrelated” “who cares about publications, I want a job” “a 58-page report. wow that’s a commitment.”
19. Task J13: find “green jobs” informationBreakout: By user type Baseline Redesign Job Seekers + Job Counselors Employers + Biz Outreach Job Seekers Job Counselors Employers Biz Outreach
20. Task J13: find “green jobs” informationObservations & Recommendations Many used search to perform this task Since few links yet on home page. The sort-by-relevance is impaired by the grouping by media type. Unlike expected search results. Recommend results be in HTML page not PDF. Search engines do not scan copy in PDF. Users click out of PDFs before reading through them, often for length Mike, Scott, Sarah, Andrew, Pam, Michael.
21. Tasks only given toJob Seekers + Job Counselors
22. Task J2: Find job openingsBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 75% of Job Seekers according to User Survey Minimum clicks to success: 2 (rnjob seekers > body search job listings) Common wrong paths: bodyfind career and occupation info <back> job seekers Representative verbatim: “first off, it's really slow” “buried really deep, should be accessible earlier, it's for people going for employment, it should just be the home page” “i would go2worksource, not here”
23. Task J2: Find job openingsObservations & Recommendations Video: Carlos2:30m Those who took link for “Job Seekers” tended to finish this task easily. But almost all expected placement to be more prominent. Lawrence, Maria, Louis, Sabrina, Liz, Carlos. There is little reason to repurpose WFE for finding Job Listings, since all unemployed people are driven to Go2WorkSourse already. Rebranding effort not worth it, especially with similarity in names.
24. Task J3: Find out how to contact an employerBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 49% of Job Seekers according to User Survey Minimum clicks to success: 2 (pulldown find employers) Common wrong paths: body find career and occupation info > body occupation explorer > Representative verbatim: “again, it's very slow, when I click I want it to show up right away, other sites are instantaneous” “seems like a lot of steps” “I can't believe Boeing is 1-4 employees”
25. Task J3: Find out how to contact an employerObservations & Recommendations Video: Carlos4:30 If most of the page is blank, should not need to click on map to show slightly larger map. Just show larger map to begin with. Doubts about data Amazon and Boeing have more than 100-249 employees. Should sort descending by size, otherwise show “other” Amazon & Boeings at top. The above cast doubt on the validity of the information Lawrence, Louis, Mike.
26. Task J4: Find out how to get help with a resumeBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 41% of Job Seekers according to User Survey Minimum clicks to success: 2 (rnjob seekers > body get resume help) Common wrong paths: search resume tools dropdown <back> Representative verbatim: “can it write it for you if you put the information in?” “honestly, what I would do is go2worksource.com and use those resume tools” “not the best website for resume assistance,” “not very pretty, not very effective” “would assume it's under tools but I don't see them”
27. Task J4: Find out how to get help with a resumeObservations & Recommendations Video: Sabrina15m Searches produce no results resume résumé Some expected this to be an application or a template rather than a FAQ like list of reference links. Sabrina, Christina, Stephanie, Robert,
28. Task J5: Find training opportunitiesBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 18% of Job Seekers according to User Survey Minimum clicks to success: 2 (pulldown training programs) Common wrong paths: friction occurred on entering valid search strings [mathematics produces no results for pages] failure due to slow load time and timeout Representative verbatim: “it's kind of confusing, you've got training programs, training providers, the training resources” “was thinking enter here would be a drop down”
29. Task J5: Find training opportunitiesObservations & Recommendations Word “Training” in 3 places confuses people as to which is the right one. Robert, Hard for first time users to enter training program title when it is not known. Carlos. New system mentioned: WOIS. Lenny.
30. Task J6: Fill out a questionnaire to match jobs to your interestsBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 38% of Job Seekers according to User Survey Minimum clicks to success: 3 (mncareer center > body assessments > body interest profiler) Common wrong paths: mn career center <back> rn assessment tools <back> rnstudents <back> Representative verbatim: “a lot of times job seekers don't know what an assessment tool is” “that's once you login, I guess”
31. Task J6: Fill out a questionnaire to match jobs to your interestsObservations & Recommendations Tabs selected bear no discernable relationship to pages, and are not consistent. mn Home (Home) rn Assessment Tools (WA Labor Market) mn Home (Home) mn Career Center body Assessment (Data Library) body Find Occupation by Skills (Career Center) mn Home (Home) rn Job Seekers (WA Labor Market)…on this page there is no link to Assessments Interest profiler fits poorly in screen, covering other copy.
32. Task J7: Find occupations that require skills you haveBreakout: By user type Baseline Redesign Job Counselors + Job Seekers Job Counselors Job Seekers Have Goal: 32% of Job Seekers according to User Survey Minimum clicks to success: 3 (mncareer center > body assessments > body find by skills) Common wrong paths: mncareer center > lnoccupation career center rncareer clusters > clusters rnjob seekers > body job listings > search field > rnoccupation explorer < back > rnassessment tools > body skills profiler > Representative verbatim: “this isn't the website that I would use for that” “i still prefer the left screen”
33. Task J7: Find occupations that require skills you haveObservations & Recommendations Without their own heading, these links look like they are under “Wages & Training.”
34. Tasks only given to Employers + Business Outreach Managers
35. Task E2: Find where to post a job openingBreakout: By user type Baseline Redesign Employers + Biz Outreach Employers Biz Outreach Have Goal: 53% of Employers according to User Survey Minimum clicks to success: 2 (rnbusiness owners > bodypost a job opening) Common wrong paths: could not find link from home page Representative verbatim: “i would think posting a job shouldn't be below information for business owners... posting a job couldn't be more important to this site” “i don't know from this page” “for my business, small as it is, business owner and employer are synonymous, but if I was larger, and an hr person, I wouldn't have thought of business owner as a place to go”
36. Task E2: Find where to post a job openingObservations & Recommendations Video: Gary :15s 31:15 Those who took link for “Business Owners/Employers” tended to finish this task easily. But said that quite often the HR person is not the business owner Andrew, Rhea, Pam, Michael.
37. Task E8: Find demographic and education info for a countyBreakout: By user type Baseline Redesign Employers + Biz Outreach Employers Biz Outreach Have Goal: 14% of Employers according to User Survey Minimum clicks to success: 2 (pulldown Local Area Profile) Common wrong paths: body Local area Household < back > rnpopulation < back > rnimagemap > Representative verbatim: “that page didn't speak to me about general demographics” “i don't see anything that specifies education, I wouldn't know where to start” “no idea where to go” “not easily found”
38. Task E8: Find demographic and education info for a countyObservations & Recommendations Assuming this content is under “Local Area Profile” this term was not self-explanatory enough to find. However, when guided to this content it was found to be valuable.
39. Agreement Ratingswith Branding Statements
40. BrandPerceptions. How much do you agree with the following statements? (1=don’t agree at all; 5=strongly agree)
41. Summary of Eyetracking Findings (Full Report in Separate Powerpoint)
42. Percent of Time Gazing at Correct Areas.
43. Time to locate.
44. Task J1: Log inBreakout: Job Seekers + Job Counselors Sample = 10
45. Task J2: Find job openingsBreakout: Job Seekers + Job Counselors Sample = 10
46. Task J3: Find out how to contact an employerBreakout: Job Seekers + Job Counselors Sample = 10
47. Task J4: Find out how to get help with a resumeBreakout: Job Seekers + Job Counselors Sample = 10
48. Task J5: Find training opportunitiesBreakout: Job Seekers + Job Counselors Sample = 10
49. Task J6: Fill out a questionnaire to match jobs to your interestsBreakout: Job Seekers + Job Counselors Sample = 10
50. Task J7: Find occupations that require skills you haveBreakout: Job Seekers + Job Counselors Sample = 10
51. Task J8: Find search toolBreakout: Job Seekers + Job Counselors Sample = 10
52. Task J9: Find general information about occupationsBreakout: Job Seekers + Job Counselors Sample = 10
53. Task J10: Find out how much a specific job pays in your fieldBreakout: Job Seekers + Job Counselors Sample = 10
54. Task J11: Find out which jobs are increasing/decreasingBreakout: Job Seekers + Job Counselors Sample = 10
55. Task J12: Find the unemployment rate for your countyBreakout: Job Seekers + Job Counselors Sample = 7
56. Task E1: Log InBreakout: Employers + Biz Outreach Sample = 7
57. Task E2: Find where to post a job openingBreakout: Employers + Biz Outreach Sample = 6
58. Task E2: Find general information about various occupationsBreakout: Employers + Biz Outreach Sample = 7
59. Task E4: Find out how much a specific job pays in your fieldSample: Employers + Biz Outreach Sample = 7
60. Task E5: Find out which jobs are increasing/decreasingSample: Employers + Biz Outreach Sample = 7
61. Task E6: Find the unemployment rate for your countySample: Employers + Biz Outreach Sample = 7
62. Task E7: Find the labor pool informationSample: Employers + Biz Outreach Sample = 7
63. Task E8: Find demographic and education info for a countyBreakout: Employers + Biz Outreach Sample = 7
64. Task E9: Find out which jobs are increasing/decreasingBreakout: Employers + Biz Outreach Sample = 7
65. Task E10: Find Consumer Price Index (CPI – target below fold) Breakout: Employers + Biz Outreach Sample = 7
66. Task E11: Find out how much a specific job pays in your fieldBreakout: Employers + Biz Outreach Sample = 7
67. Accessibilityfor Low Vision Users
68. Recommendations. How to construct the page code to be more friendly to screen-reading software (e.g. JAWS). Decide the purpose of the page. “Well, what I wonder is what you wanted this page to be about.” “I hate scrolling, scrolling, scrolling, scrolling.” Use Headers for page sections that follow Cascading Style Sheets (CSS) and adhere closely to headers (H1, H2, H3). Just like sighted users, screen readers can look at higher-level headers first and let users decide whether to listen to the links under one, or move on. “From my standpoint, it's really good to break the page into clear sections, it's good to have headings... for me, I could just hit h and go through the various sections of the page... then I could read some of that and read the links.” Don’t use tables; they are non-standard and cause JAWS to skip cells. Also use landmarks and anchors. These also embed easy spots to skip to. Include a link to “skip to main content” which can be hidden from sighted users with CSS . This will prevent JAWS from reading all the main navigation links each and every time the page loads. “A link to skip to main page content is a sigh of relief.” Tabs have no meaning for blind users; their color treatment is not set apart with JAWS. Don’t overuse floatover states or tooltips. “A bit of trepidation with a mouseover because I'm not really sure how that's going to interact with JAWS.” Web resources: http://webaim.org/techniques/css/invisiblecontent/ http://www.webcredible.co.uk/user-friendly-resources/web-accessibility/screen-readers.shtml http://www.usability.gov/pdfs/chapter3.pdf Video: JAWS1:20m Tracy45m
69. Agreement Ratingswith Branding Statements
70. BrandPerceptions. How much do you agree with the following statements? (1=don’t agree at all; 5=strongly agree)
71. Summary of findings
72. Summary of Findings Overall, the usability of the site as it is currently in production is severely impaired. Users are often taking over 2 minutes to complete core tasks. Success rates of 66% are common, dipping as low as 25% among Employers and Business Outreach managers. Baseline measures were taken for later comparison to redesign. The impairment stems mainly from: Slow load times. Load times over 30 sec are common, as are 500 errors common. Failure to identify Who (user type) and What is desired (common tasks). Fundamental flaws in navigation elements include: misleading copy and copy that holds little meaning to users (main nav) varying navigation across pages (left nav) placement of navigation in overlooked areas (right nav) descriptive copy does not match destinations (body nav) key real estate is used by seldom-needed links for core users (body nav) non-standard search results redundancy (right nav) Many-to-many correspondence between links and destinations. Different links that go to the same place, and similar links that go to different places. Absence of indicators of the user’s location in the site map.

73. Global NavigationObservations & Recommendations

74. Global NavigationObservations & Recommendations Labels on main navigation tabs hold little meaning for users. BillW, Maria, Sarah, Andrew, Colleen It is disorienting for the left navigation to differ across tabs. Contributing to very infrequent use among users. Go2worksource also different. People who took persona links were far more likely to succeed at the tasks. But it is unclear here whether the top group of links (e.g. Job Seekers, News Media) is who should go there or what you will find there. Left navigation also redundant with right navigation. Avoid different links that take you to the same place. Same problem with poor copy. Replace many poor, redundant navigation with one good one.
75. Global NavigationObservations & Recommendations If you click “WA Labor Market” in main navigation, the left navigation and body appear one way. But if you click “Find Employer” the left navigation and body are very different, despite appearing to be on the same tab. Confusing, disorienting, and impairs ability to find one’s way back to this page.
76. Global NavigationObservations & Recommendations Drawbacks of long lists of links Users must read them all, no meaning conferred by hierarchy, grouping, or sorting. After users find the one they think is right, they still read others and often make wrong choice Hildo No focus to draw the eye; flat & overwhelming. Copy is not meaningful in same way to all users. Location on lower right side known to be overlooked. Design reminiscent of a personal bookmark list. For use only by creator.