Usability Test – UNESCO Full Report June 8 2006

1. Usability Test – UNESCO.org Full ReportJune 8 2006 Laetitia Giannettini [giannettini@axance.fr] Usability SpecialistCyril Alegre [alegre@axance.fr] Usability Specialist

2. Table of Contents • Study Background and Objectives • Methodology • Executive Summary • Detail of Results • Main Recommendations

3. 1. Study Background and Objectives

4. Background of the Study and Objectives • The United Nations Educational, Scientific and Cultural Organization (UNESCO) is a specialized United Nations agency that promotes peace through Education, Social and Natural Science, Culture and Communication programmes across the world. • UNESCO’s Bureau of Public Information (BPI) is the service in charge, among other functions, of public relations and providing audiovisual media. Its also in charge ofthe administration and editorial coordination of the UNESCO web portal: www.unesco.org. • The web portal aggregates a great amount of distinct UNESCO websites with different contents and look and feel. BPI wishes to create astandard setting and a capacity building process to harmonize standardize these websites. • A framework has been developed to migrate all the websites’ pages into a new look and feel essentially based on usability and accessibility criteria and previous user tests. The home page and the Worldwide and Organisation sections have been migrated so far. • In addition, the UNESCO portal now provides content in 6 languages: Arabic, Chinese, Russian and Spanish (the four other official languages) were added to French and English (the Organization’s working languages languages).

5. Background of the Study and Objectives • BPI wished to assess the perception and understanding of the portal by the representatives of the UNESCO Executive Board • The study’s objectives were to assess: • Perception and understanding of the home page and its main items (access, tools, navigation…) • Perception and understanding of pages in the main sections (Themes, Worldwide, Organisation, Services) • Opinions toward the online version of the UNESCO Courier • Opinions toward the availability of multilingual contents • Opinions toward the legitimacy of BPI in coordinating the portal • The applied methodology was user testing (see section 2. Methodology)

6. Background of the Study and Objectives • This actual user testing follows a previous one, which took place in February 2004 • The main objectives of the first study were to assess the utility (usefulness) and usability of the portal by UNESCO’s target public. • 18 users divided into 6 profiles (3 participants per profile) were consulted for the first study: • Researchers/Scientists/Academics • Media • Teachers/Educators/Trainers • Decision-makers/Civil Servants in Supervising Ministry/Governing bodies • Development agents (NGOs, IGOs, UN) • Students/Young people

7. 2. Methodology

8. Background of the Study and Objectives • User testing is mostly a qualitative method based on observational and oral data • It assesses users’ experience through two major criteria: • usability (usefulness) of the website: is the website easily and efficiently usable? • utility of the website: does the website meets users’ needs and expectations (contents, tools, etc.)?

9. Methodology > Usability Testing • Individual sessions in three steps: • Step 1: Pre-test questionnaire • User’s profile and their use of the UNESCO web portal • Step 2: Website browsing • perception and understanding of the home page • perception and understanding of pages in the main sections • opinions toward the online version of the UNESCO Courier • Step 3: Post-test questionnaire • General opinion of the website • Performance and satisfaction rating (questionnaire for performance indicators) • Duration of Sessions : variable – between 10 and 45 minutes according to the participants availability

10. Methodology > User Profiles • User profiles (cf. Excel spreadsheet for details) : • 55 participantsto sessions of the UNESCO’s Executive Board • 22 female and 33 male • From 39 Permanent Delegations (cf. list on next slide) • Age 22 to 66 • 30 users saw the portal in English and 25 users saw the portal in French

11. Methodology > User Profiles • Participants to this test are from the following Countries:

12. 3. Executive Summary

13. Strong Points • Plenty of information available – the portal is a useful working tool • Some information is available in several languages • The home page layout is clear • The work done in order to add some consistency across the portal (the Framework) was noticed and appreciated • The “Worldwide” section is found useful (the possibility to find information about members states, its programs) • The online UNESCO Courier was appreciated

14. Weak Points • Guidance needs some improvement • The portal is rather expert-oriented – outsiders and new comers need some time to understand it • Some contents are outdated or missing • Users have difficulty finding certain information and documents and the search engines are not efficient enough • There is still a lack of consistency between the different sections – this hurts understanding and learnability • Users expect more multilingual content

15. 4. Detail of Results

16. Results Details – Severity ratings The results are classified according to their severity. This classification should also be seen as an emergency rating – the higher the number the sooner rework should be done: 1 Good point Graphical issue 2 Minor usability issue 3 Medium usability issue 4 Major usability issue

17. Results Details – Table of contents • Pre-test questionnaire – Users’ profiles and web habits • Browsing observation – Perception and understanding of the home page • Browsing observation – Perception and understanding of the “Worldwide” section • Browsing observation – Perception and understanding of the UNESCO Courier • Browsing observation – Transversal issues • Post-test questionnaire – General opinions • Post-test questionnaire – Performance and satisfaction rating

18. A. Pre-test questionnaire • The age spread was as shown in the graphic opposite • The average age was around 41 • Half of the participants had spent more than 2 years working for UNESCO • The average seniority was around 3.5 years

19. A. Pre-test questionnaire • Most participants were web experts: 83% of them have been using for more than 5 years – on average for 10 years and for 16 hours per week nowadays • Spreads as shown in the graphic opposite

20. A. Pre-test questionnaire • Almost all participants had already browsed UNESCO.org. • Most often, participants started using the portal as they started working at UNESCO. On average, they have been using it pretty often (on a daily basis for most respondents) for 4.5 years • Spread is as shown in the graphic opposite

21. A. Pre-test questionnaire • Mostly, the participants browse the portal to find information and documents. The most visited sections are: • Themes (mentioned by 22 participants) to get information about UNESCO’s programs and activities • Organization (22) to collect information and documents from the Executive Board, the General Conference (meetings, resolutions, speeches…) • Services (20) to find some documents and publications (conventions, resolutions…) and information about legal instruments • News (14) and Events (11)to check what’s happening (UNESCO’s activities, exhibitions, conferences…) • Communities (7) to find contacts

22. A. Pre-test questionnaire Strong points • The main strong points reported by the participants according to their own experience were: • The great amount of information availableThe portal is seen as a useful source of information to keep up to date about UNESCO news, about “what’s currently going on at UNESCO”. Furthermore, the UNESCO portal is often seen and used as a regular working tool: some users, for example, use it to collect information in order to prepare Executive Board meetings • The overall ease of use and easy access to the information • The contents available in 6 languages

23. A. Pre-test questionnaire Weak points • The main weak points reported by the participants were: • Overall complexity, especially for outsiders or newcomers – the portal was often said to take some understanding of UNESCO structure to be efficiently used • Difficulty in finding certain information and documents: all the participants said they could find most of what they were looking for but many also reported some problems in finding some information • Outdated contents: many users deplored the slowness – or lack, at worst – of updates • Lack of multilingual contents

24. A. Pre-test questionnaire • Most participants were aware that BPI was in charge of the UNESCO website • Most of them also had some general knowledge of BPI activities: “Press information, Public Relations” (Holy See) “Publishing, Communications” (China) “Public information about UNESCO’s activities” (Italy) “Promotion of UNESCO’s image” (Sri Lanka) • Other main activities reported: website administration, content feed, updates and coordination

25. Results Details – Table of contents • Pre-test questionnaire – Users’ profiles and web habits • Browsing observation – Perception and understanding of the home page • Browsing observation – Perception and understanding of the “Worldwide” section • Browsing observation – Perception and understanding of the UNESCO Courier • Browsing observation – Transversal issues • Post-test questionnaire – General opinions • Post-test questionnaire – Performance and satisfaction rating

26. B. Perception and understanding of the home page The home page is clear • Most users found the home page layout clear and user friendly • They understood the 3-column layout and the left and right menus • The sections’ organisation and the wordings were well understood “It’s clear, one can find almost everything he needs” (China) “It’s practical, I can find what I want” (Indonesia)

27. B. Perception and understanding of the home page The home page is clear • The languages available on the home page were well perceived and the work done to provide multilingual contents highly appreciated “They are very useful” (Italy) • The tools are well perceived – the search engine is the most attractive “Practical, I find what I want” (Indonesia)

28. B. Perception and understanding of the home page The home page is regularly kept up to date • The home page is seen as being kept up to date (with changing pictures, themes, news and events), which gives the website a dynamic look that was appreciated

29. B. Perception and understanding of the home page 3 High informational density • Users found the home page had too much information and links on it, especially in the right menu “Too many things – I can’t get any information” (Holy See) “It’s compact, there are too many things on it, which distracts one’s attention – even more for someone browsing it for the first time” (India) “The right menu offers too many things – nothing shows up” (Switzerland)

30. B. Perception and understanding of the home page 3 High informational density • A few users did not understand the wording “Communities” nor anticipate the content of this section • The right menu is too heavy for non-UNESCO users – it takes some good knowledge of the UNESCO organisation to understand the menu • The high density in the right menu restrains the visibility of the links of interest – they do not attract attention (e.g. “General Conference” and “Executive Board”)

31. B. Perception and understanding of the home page 3 High informational density • Hide the submenus (under “Secretariat”) – and make it available by a click on “Secretariat”? • By default, the Organisation menu should only offer “About UNESCO”, “General Conference”, “Executive Board”, “Director-General” and “Secretariat” “The home page should focus: it should only give access to UNESCO’s main sectors and organs” (India) “The ‘General Conference’ and ‘Executive Board’ links should show up more” (Switzerland) • This 3 column layout is more adapted for 1024 resolution.

32. B. Perception and understanding of the home page The home page does not communicate UNESCO’s spirit enough 3 • A few users deplored the fact that the home page like the whole website did not convey efficiently UNESCO’s mission, values and messages “I would prefer a global picture in the central area, a more universal look to it” (Kazakhstan) “The home page fails at showing what UNESCO is” (Canada)

33. B. Perception and understanding of the home page The home page does not communicate UNESCO’s spirit enough 3 • The home page content is rather insider-oriented: non-UNESCO users may not get a good insight of what UNESCO is and does • A baseline on top of the page could provide a good introductory overview of UNESCO’s definition missions:“UNESCO - the United Nations Educational, Scientific and Cultural Organization (UNESCO) was founded on 16 November 1945. For this specialized United Nations agency, Education, Social and Natural Science, Culture and Communication are the means to a far more ambitious goal : to build peace in the minds of men”

34. B. Perception and understanding of the home page The home page does not communicate UNESCO’s spirit enough 3 • The central area does not give an overview of UNESCO’s core activities (i.e. key international programs) • The central area is not well perceived, it doesn’t catch UNESCO people’s attention a lot – insiders are more attracted to the menus “I don’t browse the central area” (Holy See) “It doesn’t catch my attention” (Kazakhstan)

35. B. Perception and understanding of the home page The central boxes’ layout is confusing 3 • The central area layout is not clear: • The articles in the “News” and “Events” boxes are separate with white lines and look like they’re discrete boxes – therefore, the second article, for example, does not seem to be part of the news • The boxes should be united

36. B. Perception and understanding of the home page Some accesses are not perceived 2 • The themes tabs on top of the page were not well perceived by some users – the lack of consistency across the portal restrains learnability of the navigation: • Different graphical aspect • Different tabs – some sites offer an “The Organisation” tab or a “Site map” tab, which could be extended to the whole portal

37. B. Perception and understanding of the home page The home page does not mention some current events 2 • Some users deplored the fact that the home page did not mention the ongoing Executive Board (Palestine, European Commission)

38. B. Perception and understanding of the home page The French “Plan” tab is not well designed 2 • The tab’s icon and label are so far from each other that they don’t seem to be linked together • The site map icon needs some graphical readjustment

39. Results Details – Table of contents • Pre-test questionnaire – Users’ profiles and web habits • Browsing observation – Perception and understanding of the home page • Browsing observation – Perception and understanding of the “Worldwide” section • Browsing observation – Perception and understanding of the UNESCO Courier • Browsing observation – Transversal issues • Post-test questionnaire – General opinions • Post-test questionnaire – Performance and satisfaction rating

40. C. Perception and understanding of the “Worldwide” section The “Worldwide” access was well perceived and appreciated • The “Worldwide” access was well perceived by most participants • It was also well understood: most participants had a good insight of what the section offers – local information about UNESCO organisation and programmes • The Worldwide section available from the left menu was generally appreciated as a useful source of local information

41. C. Perception and understanding of the “Worldwide” section The “Worldwide” access is mostly clear • The “Worldwide” accesswas easily used by most users – they had no difficulties accessing the page dedicated to their country

42. C. Perception and understanding of the “Worldwide” section The “Worldwide” access is also confusing 2 • A few users got confused by the“Worldwide” access, which follows UNESCO’s own logic in aggregating regions of the world, which does not match the common logic about world’s continents and therefore may confuse non-UNESCO people – e.g. Europe and North America form a single region and the Russian Federation and Turkey are listed twice, both in the “Europe and North America” and the “Asia and the Pacific” regions.

43. C. Perception and understanding of the “Worldwide” section The “Worldwide” access is also confusing 2 • Several interpretations were given as for UNESCO’s own logic in aggregating regions of the world: • Regions are voting groups • Regions follow a “political” logic

44. C. Perception and understanding of the “Worldwide” section The cross-referencing of information principle was appreciated • The availability of country-specific information (themes) was mostly appreciated

45. C. Perception and understanding of the “Worldwide” section The themes were not easily perceived by users 3 • Some users did not notice that some more content was available in the lower area of the page • The page layout should allow users to notice the content below – the browser should “cut” the page where some content is partly visible in order to suggest some more content

46. C. Perception and understanding of the “Worldwide” section The themes’ content was not easily anticipated by users 2 • A few users deplored some difficulties anticipating the contents “behind” the themes links on the country-specific pages • The links are not clear enough, not understood as being contextual, country-specific • There is no clear introduction • Explanatory sentences should introduce the themes’ content (what about the type of links is)

47. C. Perception and understanding of the “Worldwide” section The “Organisation” menu lacks visibility 3 • Many users did not see the “Organisation” menu on the country-specific pages • The background colour and the search engine and “UNESCO at 60” picture above the menu make it “secondary” (Austria) • The “organisation” menu lacks consistency • “The Organization” label is not country-specific and the “UNESCO in this Country” label is not well located – it applies only to the 2 links underneath it

48. C. Perception and understanding of the “Worldwide” section The “Organisation” menu lacks visibility 3 • The menu should be moved upward in the place of the search engine and picture and gain consistency from one page/section to another • The menu label should be more specific

49. C. Perception and understanding of the “Worldwide” section Content was sometimes assessed not to be full-accomplished 2 • According to the users, some contents are poor or information is missing or not enough emphasised : • The countries’ main sectors of activity and the key programmes (Luxembourg, Greece, Holy See, Poland, Slovenia, Ethiopia, etc.) • The statistics should come along with interpretations (Indonesia) • The National Commission programme (Netherlands) • A link to the country website from the National Commission page (Argentina, Kazakhstan)

50. C. Perception and understanding of the “Worldwide” section Some country-specific pages content is outdated 2 • Some contents are not up to date – and some pages are still empty (Slovenia) • Some users (Slovenia, Luxembourg) suggested the National Commissions should be involved and provide BPI with information – it could be through a form –, given the National Commissions are in charge of defining their status, composition principles and main sectors of activities