CityofBoston

1. CityofBoston.gov City of Boston.gov Team 2 Testing & Assessment HFID-750 Usability Testing & Analysis Team 2 HF750 Testing & Assessment 12.08.08

2. Table of Contents • Overview • Executive Summary • Goals & Objectives • Methodology • Findings & Recommendations • Conclusion • Next Steps • Lessons Learned • Q&A

3. Overview Engagement: We engaged with the development and management teams for the cityofboston.gov website to conduct usability testing. Website: CityofBoston.gov General Audience: Residents, Business interests, Visitors, Students Purpose: To facilitate the dispensing and communication of information about the City of Boston… To anyone who might have a question about how to perform common tasks such as paying fees, finding a job, getting permits, licenses, etc. To provide contact information for city officials to facilitate communication between the city government and concerned citizens To find out when and where public events will occur To physically get around the city and obtain information on local communities

4. Executive Summary AVG: B • Participants found the website very informative. • “It gives you basically the whole…whatever you’re looking for in Boston. It is informative.” • “Seems to have a lot of information that people would want to use, like most requested pages.” • Participants liked the visual design (colors, graphics etc.) of the portal and commented that it aided in navigation • “The color scheme really made it easy. You could just follow the colors.” • Organization of information was not intuitive and users relied on‘Search’ function to find what they were looking for. • “Certain things were not where I expected them to be.” • Participant’s expectations of online services on the website werenot met. • “I don’t see an online application…I thought there would be an online form.” • Participants reported that their overall experience with the site was positive and gave it an average grade of “B.”

5. Goals & Objectives

6. Goals & Objectives - Site Goals • Increase efficiency of completing commonly performed tasks • Provide overall ease of use • Provide an easy method to locate desired information • Establish a forum of communication for site users • Deliver useful and informative content that incentives users to return to the site

7. Goals & Objectives – Business and User Goals Business Goals Increase Site Traffic Create opportunities for partnerships within the community Provide comprehensive information for users to access desired information to alleviate support calls to government offices Ensure that all user/audience goals are being met Ensure that Business and User Goals are accurately aligned To be a best-in-class city destination website User Goals To learn more about Boston and what the city has to offer To obtain desired information with respect to each audiences’ needs To complete online transactions with ease To improve community interaction with residents, business interests, and visitors

8. Goals & Objectives - Test Objectives • Gather user feedback on the City of Boston online portal through observations and questions asked during the interview session • Identify usability issues especially those related to information architecture, and navigation • Understand users’ overall subjective satisfaction with the website through their responses to a post-test questionnaire • Uncover opportunities for improvement and/or enhancements • Compile and present findings to the CityofBoston.gov stakeholders

9. Methodology

10. Methodology - Testing Protocol Where: Tests were conducted remotely as well as in person at Bentley’s DUC. When: October 30th and November 5th, 2008 Who: Participants recruitedfrom Bentley student population as well as our friends. They were tested in one-on-one usability sessions. 6 participants were recruited to represent 4 types of user groups, Businesses, Residents, Students, and Visitors ranging in age from 18-55+ What: Participants signed a consent form and responded to a pre-test questionnaire that identified their level of experience with using the internet and government related web sites How: Participants were asked to use think-aloud protocol while they completed each task. Scenarios were constructed for the participants to replicate normal tasks on the CityofBoston.gov. Operating within a given scenario, users were asked to perform independent tasks with clearly defined measures of success or failure.

11. Participant Demographics

12. Methodology - Conducting the Test During the usability evaluation, 6 participants, matching the user profile(s), were asked to spend one hour with the site. During this hour, participants: Completed a user background questionnaire Answered questions about initial site impressions Performed real-world tasks on the site while thinking aloud Answered questions about their overall satisfaction

13. Methodology - Analysis and Reporting During the sessions we focused on capturing the following qualitative metrics: Task initiation and user expectations Points of confusion (points where user is stalled, isn’t sure what to do) User’s perceived ease of use Capturing Information through Video: The computer screen that the user sees as he/she is performing tasks The participant him/herself. Locating parts of the tasks where participants are either satisfied or frustrated Locating assists and failures Severity Ratings: High, Medium, Low ratings Based on our views of importance and the amount of difficulty seen over the course of testing

14. Methodology - Pre-Test Interview At the beginning of each session, we asked participants five questions, including:

15. Methodology - Post-Test Interview At the end of each session, we asked participants 9 questions:

16. Findings & Recommendations

17. Findings & Recommendations - Task List During the usability evaluation, participants were asked to complete eight (8) scenarios or Real-life tasks on the site. The tasks were presented in the same order for each participant. • You have just learned from a friend that you can pay your personal property tax online through the CityOfBoston.gov web site. How would you go about finding where to do this? • You just bought a new puppy from a store in Boston and have to get him licensed, but you aren’t sure how. How would you go about finding this information on the City of Boston web site? • You’re planning a trip to Boston in a few weeks, and heard about an attraction called the “Freedom Trail.” How would you go about finding more information? • You want to find out what special city events are planned for the month of November. How would you go about finding this information? • You’re researching the quality of Boston’s public school system, and need to find a list of all of the public schools in the city. How would you go about finding this information? • The upcoming election is only a few weeks away, and you’re going to be out of town on Election Day. Using the City of Boston web site, find out how to apply for an absentee ballot. • You’d like to look at the minutes from the last Boston City Council meeting. How would you go about finding this information? • You have been injured on the job and will be unable to continue working for a few months. Find out how to apply for worker’s compensation.

18. Findings & Recommendations - Task One: Payment of Personal PropertyTax Recommendations: • Because the Business ID is labeled as Identification Number (note the instructions in the right column) on the paper form required to make a payment, change the Business ID label in both columns to Identification Number. • This will help reduce confusion as to whether individuals are on the correct page to pay their property tax. Severity: High Findings: • Participants did not encounter difficulty reaching the Personal Property Tax page, but were confused by the page, as it asked for a Business ID. 33% of participants did not know they were on the correct page, and failed to complete the task. “This isn’t for residents because you need a business ID. Hm. And you need a bill number. It says personal property tax, not business.” –P6 “This brought me to a business ID…this is confusing, because I’m not a business.” –P2

19. Findings & Recommendations - Task Two: Licensing a New Puppy Recommendations: • Move the general dog license application above the pit bull license application so that no scrolling is required to get to the form link at a resolution of 1024x768. • Feature Request: Consider allowing dog license registrations to be submitted with payment online Severities: Medium Findings: • 83% of participants completed this task without assistance. • The location of the general dog licensing form is at the bottom of the page, under a specialty pit bull license. Participants commented on this: “All other dogs should have been at the top, pit bulls towards the bottom.” –P3 “I would have given it a 1 [easiest] but general license was below pit-bull license application.” –P1 • Participants expected an online form or application instead of a PDF that needed to be printed and mailed: “I clicked on license application and expected a form.” –P1 “I don’t see an online application…I thought there would be an online form.” –P4

20. Findings & Recommendations - Task Three: Information on the Freedom Trail Positive Feedback Findings: • All participants completed this task without difficulty. “It was pretty easy to navigate where I wanted to go.” –P2 “Didn’t have to go anywhere else…was right there, pretty easy” –P5 “It was right on the [Visitors] home page.” –P6

21. Findings & Recommendations - Task Four: November Special Events Severities: High Findings: • 83% participants completed this task, but all expressed concerns with the date range displayed in the calendar and the general presentation of the list of events. • Participants were confused as to what the checkboxes on the left-hand side of the calendar were for: “Curious as to what check box does? Not sure why it’s here or what it does.” –P1 “Easy to get to, but once I was there I didn’t like the check boxes, I didn’t feel like I got any feedback, loading in the background. Hard to find specific event.” –P4 • Participants noticed that the calendar, by default, only displays events for a two week period. They expected a full month’s worth of events to be shown: “I was unsure if I had everything on that one page (all of the events).” –P2 “This looks like…this is only one week...” –P6 • Participants expected to see an actual calendar instead of a list of events: “I would have expected a calendar I could click on, day, week month for whatever view you want and then you could see all the events.” –P4 “Easy to get to, but with a calendar she would expect a sign. The page opens to a list, she would like to see this in a different format. “ –P2

22. Findings & Recommendations - Task Four: November Special Events Recommendations: • Display a full month’s worth of events by default; participants expected the calendar to default to a “month” view. • Consider presenting events in an actual calendar format. This makes it easier to determine when events start and end. • Add end times (when available) to event listings • Add a heading under the Search box on the left-hand navigation titled “Filter Calendar Events” in larger type (to draw attention). • Consider displaying different categories (e.g. Arts, Sports, business, community event) in different colors to allow users to select and filter on a higher level of genre

23. Findings & Recommendations - Task Five: List of Boston Public Schools • Severities: High • Findings: • Participants remarked that it wasn’t clear if this information should be located under the Residents or Students section. • “If you’re looking for schools…would you expect to find it under students? I was a little bit confused.” –P5 • “I went to residents as residents would be interested in this info.” –P3 • “There were a number of things that seemed like they were going to give me a list of schools. They were really for finding which schools to register for. I guess choose a school might somehow suggest registering for a school? It wasn’t clear to me that it wasn’t gonna be a list of schools.” –P6 • 33% of participants (because of the path they took initially) noticed that the link to the list of schools on the Schools, Colleges, and Universities page was broken.

24. Findings & Recommendations - Task Five: List of Boston Public Schools • Recommendations: • Consider placing links to this information in both the Residents and Students sections of the site. In the Residents section, place the list link on the front of the Education and Schools page (83% of participants took this path). • Fix the broken link on the Schools, Colleges, and Universities page to point to the new list on the Boston Public Schools web site. • Task completion required leaving the CityofBoston.gov website and moving to the bostonpublicschools.org site. This was a transparent change. Consider a message letting the user know that they are leaving the current site.

25. Findings & Recommendations - Task Six: Absentee Ballot • Severity: High • Findings: • 83% of participants completed this task successfully • Participants expressed concerns about placing the Absentee Ballot form under the Register to Vote section: • “Register to vote seems logical, but that isn’t what I really want to do.” –P2 • “I might have expected it to have a link pointing to it at the same level as the voter registration. • You don’t have to register to vote with an absentee ballot.” –P6 • “These links were not intuitive.” –P1 Recommendations: • Place the Absentee Ballot form link on the front of the Elections and Voting section of the Residents area of the site; not all absentee voters are registering for the first time. • The text informing the user that they could apply for an absentee ballot was always listed after, and in the same sentence with registering to vote. Registering to vote for the first time and requesting an absentee ballot are separate tasks.

26. Findings & Recommendations - Task Seven: Meeting Minutes Recommendations: • Consider moving the text size adjustment tool (in the lower left corner of every page) to the title bar about 10% of the way down from the top of the screen. • This will allow users to adjust the default text size on the site to suit their needs. In its current location, the tool is not visible (no participants commented on its presence). Severity: Low Findings: • 83% of participants completed this task successfully • Participants who completed this task remarked that it was “pretty easy to get to” and “very straightforward.” • One of the older participants noted that the small font size that the Meeting Minutes link was in could cause issues: “The link was small and down at the bottom, difficult to find.” –P4

27. Findings & Recommendations - Task Eight: Worker’s Compensation Recommendations • Place a link for Worker’s Compensation in both the Forms section of the site and on the Business section (under Forms and Applications). Severity: High Findings • Only 16% of participants completed this task successfully. All participants who failed said that they would use the Search tool; no participants looked in the Services drop-down. • Participants expected to find the application in the Business or Forms section of the site. “Didn’t exactly complete task, would have expected it to be within forms permits and applications page, not sure where else it would be.” –P4 “Would think it would be under applications, but must be wrong because it is not there.” –P2 “Looking for a form or something like that ” –P5

28. Task Success / Failure Pass = User was able to accomplish task without an assist Fail = User was unable to complete task, needed an assist or relied on search

29. Task ratings 1 = Very Easy 2 = Moderately Easy 3 = Neutral 4 = Moderately Difficult 5 = Very Difficult

30. Next Steps Review the list of findings in detail and consider refining the website based on the recommendations provided. Consider expanding the user test group to include business audiences Include differently-abled groups for next usability study (to check for Section 508 compliance) Work with development team to retest redesigned website

31. Lessons Learned Rotating roles of moderator, observers and note-takers helped us to experience the roles first-hand Critical to check equipment prior to test (i.e. Phone for remote testing) Important to adequately space out participants during testing sessions (esp. Remote Testing) Having a test script prepared was very helpful and helped ensure continuity to our testing approach Explaining testing logistics to participants up front was helpful Pre-planning with group members helped streamline testing process Giving $10 Starbucks gift cards as honorarium was appreciated by participants Keeping some snacks and candies handy were good. It helped with blood sugar level and ability to concentrate Don’t put the most difficult task at the end of the list of tasks; participants should leave on a ‘high note’ Make it more obvious that participants should return to the home page before starting each new task

32. Q&A

33. Appendices • Compiled Session Notes • Test script • DVD recordings of participants • Consent forms