structure

1. RURAL WINGS Final Review Meeting Usability EvaluationThomas Köhler, Nina KahnwaldMedia Center, Dresden Univ. of Technology 1

2. structure • Introduction • Usability Questionnaire • Procedure • Response rates • Results • Personal data, Infrastructure, Usage Profiles • Usability of Rural Wings infrastructure, training and support • Usability of Rural Wings applications • Usability monitoring during user training • Procedure and Results • Heuristic Evaluation • Procedure and Results • Summary 2

3. procedure • Overall usability evaluation activities: • Usability Assessment through Questionnaires – after a few months of usage (11/2007-2/2008, 12/2008-2/2009 and 10/2009-12/2009) • Usability Testing through monitoring of questions and problems – during implementation and training sessions (6/2007-12/2009) • Usability heuristic Evaluation of Rural Wings CAP – prototype stage (8/2007) and implementation stage (12/2009) • Technical Evaluation – continued monitoring during test runs (11/07- ongoing) 3

4. usability evaluation questionnaire • Usability questionnaire • Structure of the questionnaire (based on „Questionnaire for User Interface Satisfaction“ [QUIS]): • Personal data • Infrastructure • Usage profile • Performance/Usability of technical infrastructure • Usability of CAP • Usability of applications • Scales are applicable to • all user groups identified • in the user needs • analysis (WP3). 4

5. results • Response rates • data entry that was completed until 20th of December 2009 • analysis of all questionnaires where users specified their pilot site • questionnaire was completed by 159 end users from 90 pilot sites 5

6. response rates I 6

7. response rates II 7

8. response rates III 8

9. response rates IV

10. response rates V 10

11. response rates VI

12. response rates VII

13. response rates VIII

14. response rates IX 14

15. response rates X 15

16. response rates XI

17. personal data I • Demographic aspects: • Gender: 66 female, 86 male participants • Age: majority (69%) is between 25 and 49 years old. • Professions: • Other: i.e. civil servant, school consultant, journalist or librarian 17

18. personal data II 18

19. infrastructure • Infrastructure characteristics: • 75% of the RW users use MS Internet Explorer as webbrowser, Firefox is used by 22% of the users. Netscape and Opera are used by only 1,3%. • Nearly all responding RW users (96%) have MS windows installed as operating system. Five (3%) use a Linux system and no one uses a Macintosh. • Connection to the RW internet access: • via Sat modem: 36,5% • via local area network: 32,1% • via indoor wireless network connection: 14,5% • via outdoor wireless network: 6,9% 19

20. usage profiles I 20

21. usage profiles II Services used: • web browsing (88,7%) • e-mail (74,8%) • RW applications and tools (41,5%) • Instant messaging (31,4%) • software updates (28,9%) • Websites visited (3 most visited): • search engines • news-sites • online-mailing • RW-CAP • weather forecast 21

22. usage profiles III CAP use: Duration per week User Location 22

23. usability of infrastructure I • All categories are rated as excellent or good by at least 39% • Highest satisfaction can be seen in the area of assistance ratings which every fifth user (19.5% and 22%) judged as excellent. The initial installation was rated good or excellent by 69% of the users. • Deficits or problems seem to exist with the availability of the service (rated “poor” by 17.6% of the end users) and the reliability of the satellite terminal (rated “poor” by 7.5%). • Also free text answers and comments refer to low bandwidth and unreliable connections with repeated failures. 23

24. usability of infrastructure II 24

25. usability of infrastructure III Summaryusabilityratings „poor“ (part I) 25

26. usability of infrastructure IV Summaryusabilityratings „poor“ (part II)

27. usability infrastructure V 27

28. usability RW-CAP I • RW applications are often used independently from the CAP: • Data from Google Analytics has shown that slightly more than 50% of the users did access these RW applications directly. The following evaluation results thus reflect the experience of the remaining up to 50% users. • Some users perhaps don‘t see the added value of the CAP: „Do not see a strong need to always enter through the portal when you already know where you want to reach out.” • RW-CAP ratings overall impression was similar to the interim evaluation: • A critical topic was the missing localization and translation of RW-CAP contents, as stated by several users in the free text comments: “The CAP is in English, because of that we cannot judge its relevancy.” (Spain) / „Not available in French.“ (France) / „The main problem in using the site is the language barrier.” (Poland) • However this has been solved by the beginning of 2010 with a new Greek + French version. Also national partners may insert translations by themselves easily if needed. 28

29. usability RW-CAP II 29

30. usability RW applications I • 15 different RW applications were used and rated during the final runs. • 39% of the participants in the survey had not used any RW application when answering the questionnaire (32% in final runs Phase A evaluation). • On the whole the applications were rated positively. • less positive rating than during the last evaluation: Agroweb (15 users), VEMUS (14 users), WebTV (30 users), Xplora (29 users), D-Space (13 users) and Teacher eTraining (13 users) • lost slightly: YouRA (27 users), UNITE (2 users) and ExperiNet (8 users) • rated as positive as the last time: NEMED (25 users), AgroTeleDiag (3 users), the Health Training (12 users), RCCM (9 users) and MEDSKY (6 users) • See Annex A of this presentation for detailed usability ratings for each application. 30

31. usability monitoring • Principles of RW usability monitoring • User problems and questions can be indicative for aspects of the technical infrastructure, the manuals, the central access point or the rural wings platforms and applications that are not designed intuitively. • It can thus be used to identify possible usability problems and areas of improvement during the training of the end users. • A structured template for the • collection of user questions • and problems arising during • the training of the end users • was provided to all NCs. • Data collection realized after • February 2009; in final test • runs feedback only from • Spain. 31

32. results usability monitoring • Emerging topics: • Language was not a problem as localized sites are available; • PC-skills become an issues; • RW applications and CAP (problems with D-Space and CAP-Structure); • RW Infrastructure (Problems with network - suggested usage of a faster browser did improve the performance somewhat but reportedly not enough). 32

33. heuristic evaluation • Design of heuristic evaluation • Expert evaluation of the RW Platform by usability experts of TUD: • checklist of recognized usability principles (heuristic) will be used to identify possible usability problems (see Schweibenz/Thissen 2003 and Nielsen/Mack 1994). • identified usability problems and possible recommendations will be aggregated and communicated to FORTHnet and all NCs by TUD. • Expert evaluation was conducted twice: at prototype stage (2007) and end of the project (2009) after improvements were made. • Results • overall better impression than during 1st usability evaluation: lucid, structured, not overloaded. • but information offered and options could be structured and grouped better and adapted to the needs of the users • often available in English only, even the help function (!) • language specific (sub) sites should be developed • for Further details of the heuristic evaluation cp. annex A 33

34. Conlcusions I • Language and localisation • Language barrier had been identified as one of the most crucial issues in previous evaluations. • RW Infrastructure • RW services are mostly used during the week and earlier in the day, while the usage during the weekend has increased as compared to the last evaluation. • Apart from language barriers, problems with network performance and wifi coverage most dominant in user feedbacks. • Availability of the RW internet service has been rated as “poor” by 17.6% of the end users (compared to 11% during the last evaluation phase). • Negative experiences with bandwidth reliability did again retard local activities, hinder training sessions or even prevent proper use of RW services most of the time. 34

35. Conlcusions II • RW-CAP • Development of CAP-usage: • in test run evaluation nearly all participants had used RW-CAP, in the final runs 18% stated “not applicable”; • only 29% of the participants compared to 40% in the test runs did name it as one of the three most visited websites. • Language barriers and navigation problems remain (monitoring and heuristic evaluation). • Overall feedback less critical than in the test runs (higher satisfaction or lower relevance of CAP?). • Suggested improvements: • Single log-in for all applications via CAP which would provide an added value; • Community features (have been implemented roughly but not promoted). 35

36. Conlcusions III • RW Applications • 16 different RW applications were used and rated – predominantly positive, albeit slightly lower than during final phase A. • Although 39% of the participants in the survey had not used any RW applications when answering the questionnaire. • Free text comments division between users who found the applications “very good”, “marvellous” or “motivating” and those who, for various reasons, didn’t manage to get some of the applications working at all. • RW training and support • The overall rating of the RW installation services and the RW support was very positive. About a fifth of the participating users did rate it as “excellent”. Suggestion for successive projects: • to ensure sustainability meta-competencies should be imparted (i.e. problem-solving strategies in case of computer problems) to enable users to help themselves outside of training sessions. • Supported by community features in RW CAP to allow end-users to share best practice with other pilot sites and support each other when problems occur. 36

37. Conlcusions IV • Positive feedback • despite all the criticism that an evaluation has to reflect we can conclude the usability evaluation of the Rural Wings Project with positive comments of users on the RW-project: • “Thanks to this project we were able to contact other different realities and to share experiences and knowledge gained with them .” • “It is an extraordinary help for rural localities such as ours.” • “It's good to feel myself as a part of international project.” • “It is a great project that allows the educational rural communities to have access to Technologies of Information and Communication.” 37

38. Thank you for your attention. • Prof. Dr. Thomas Köhler • Technische Universität Dresden • Media Centre http://mz.tu-dresden.de • Phone: +49-(0)351-463-32772 Fax: -463-34963 38

39. ANNEX A • Annex A • usability ratings for RW applications

40. usability RW applications 40

41. usability RW applications 41

42. usability RW applications 42

43. usability RW applications 43 RURAL WINGS IP

44. usability RW applications 44

45. usability RW applications 45

46. usability RW applications 46

47. usability RW applications 47

48. usability RW applications 48

49. usability RW applications 49

50. usability RW applications 50