Newspaper in Education Web Site (NEWS) Usability Evaluation

1. Newspaper in Education Web Site (NEWS)Usability Evaluation Conducted by Terry Vaughn School of Information The University of Texas at Austin November 29, 2004

2. What we’ll talk about today • Newspaper in Education Program Overview • Test Description • Participant Background • First Impressions Interview • Task Performance • Problems and Their Sources • Final Reactions • Conclusions • Recommendations • Questions from the audience

3. Newspaper in Education Program Overview NIE • Program of the Orlando Sentinel (major newspaper) • Promotes literacy in the central Florida area • Supplies newspapers and teaching materials to area schools NEWS • Public can learn more about NIE programs • Register for NIE programs • Purchase newspapers and other classroom materials • http://extra.orlandosentinel.com/nie/

4. Test Description • Research Questions • Goals • Participants • Environment and Equipment • Test Procedure • Data Collection

5. Participant Background Data • 4:1 female to male • 30 year median age • All college educated • 10 - 21 years of computer use • 8 - 10 years of Web use • All used the Web in their work • All claimed the Web was important - very important part of their work

6. Participants’ First Impressions Positive • “Visually appealing” • Appropriate colors and graphic • Grouping of left-side navigation • Clear audience Negative • Headline looks like a run-on sentence • Multiple layers of navigation on top may confuse

7. Task Set Tasks 1 - 5 • Finding information • Example:You are interested in taking a workshop to learn how to incorporate the news into your curriculum. How many participants must you have for a Teacher Workshop? Tasks 6 - 10 • Filling out forms • Example:You decide to order a newspaper subscription for your classroom starting after the Christmas break (1/4), suspending service over Spring Break (3/14 – 3/18), and ending on the last day of school (5/25). Find the page where you can order a subscription and fill out the form.

8. Task Performance (ranked according to difficulty) Most problematic tasks: • Task 6.Find the page where you can order a classroom subscription and fill out the form. • Task 2.What is the minimum order of newspapers per classroom? • Task 1. In addition to daily newspapers, what else will you receive with your classroom subscription?

9. Most Problematic Tasks Task 6.Find the page where you can order a classroom subscription and fill out the form. • Errors: invalid, incorrect, and omitted data • Sources: vague labeling and directions; customers had to manually calculate data; unintelligible server error page • Quotes:"I don't get this at all.”"I'm supposed to calculate all this myself?” Classroom Subscription Order Form

10. Most Problematic Tasks Task 2.What is the minimum order of newspapers per classroom? • Errors: navigation - clicked wrong page • Sources: vague labeling; information disorganization - relevant information is located on the FAQ • Quotes:“An FAQ is my last resort.” Newspaper in Education FAQ

11. Most Problematic Tasks Task 1. In addition to daily newspapers, what else will you receive with your classroom subscription? • Errors: navigation - clicked wrong page; wrong answer • Sources: poor information organization • Quotes:"I have no idea where to go.”"How do I begin?” Teacher Home Page

12. Identifying Problems Errors Ranked According to Criticality Severity Scale 1 - Minor inconvenience, easy to circumvent 2 3 4 - Major problem prevents user from completing task

13. Identifying Problems Sources of Errors

14. Final Reactions Overall grade: C+ Most appreciated: visual design Least appreciated: information disorganization “Will you return to the site?” Yes - 2 Maybe - 2 No - 1 “Would you recommend the site to a colleague?” Yes - 2 Maybe - 1 No - 2

15. Conclusions Q:Is there enough information for customers to make informed decisions about the purchase of NEWS content and services? A: Yes, but it is difficult to gather because relevant information is not adequately grouped and labeled. Q: Can customers navigate the site and find the information they need? A: Yes, but sometimes customers get disoriented because of the poor labeling and information organization. Q: Can customers find the information in an acceptable time frame? A: Yes, but not without difficulty and/or assistance.

16. Conclusions Q: What problems do customers encounter performing tasks on NEWS? A: Customers mainly encounter problems completing the order forms. Q: Can customers complete the online subscription forms. If so, how many? A: None without assistance. Q: How do users perceive NEWS? A: Due to the flaws in the site and the problems participants encountered, they we’re left questioning the professionalism of the NIE program. Since the forms were so difficult, participants questioned NIE’s commitment to customer service.

17. Recommendations Labeling • plain English - no jargon • consistent • descriptive Information Organization • group relevant information according to service offering • don’t scatter related information around Hyperlinks • change the Contact link to direct to a web page • no “click here”

18. Recommendations Form Design • improve layout - spacing and grouping • clear, descriptive labeling of text boxes • don’t make customers do work that a computer could toe.g. calculating data • break Classroom Subscription Form into two parts, with an interactive calendar to help customers with start/stop dates Writing style • clear and concise • employ a copy editor Feedback • order form confirmation

19. Questions Thank you!