JISC Distributed e-Learning Tools Software Quality Evaluator

1. JISCDistributed e-Learning ToolsSoftware Quality Evaluator e-Learning Programme Meeting: Framework & Tools & Distributed e-Learning Strands Jury’s Inn Birmingham 5th and 6th April 2005

2. Who are we? • John Harris – Epic plc • Suzi Wells – Futurate ltd • Jonathan Grove - Futurate ltd • Nicky Ferguson – Clax ltd

3. Our brief • Act As Software Acceptance • Check if the project has delivered the software it proposed in the project plan and ensured its quality as defined in their quality assurance plan. • Summative Evaluation Of Software Quality • Comment on the general quality of the software and the quality assurance plan for each project. • Lessons Learnt • What can be learnt from the quality assurance process used with the projects?

4. Our evaluation remit • Quality Planning, before and during • Quality Control and Auditing • Extendibility and Maintainability • Does it work? • Usability • Robustness • Open Source and Open Standards • Documentation

5. Our feedback remit • test results to individual projects • quality planning process • quality control • open source policy – compliance and workability • tools, techniques and best practice • evaluation process itself …

6. Evaluation Criteria • Match of actual software to planned output • Ongoing quality planning • Open Source compliance • Open Standards compliance • Quality control procedures • Specific and adequate documentation

7. Evaluation Criteria (continued) • Fit for purpose • Maintainable • Extendible • Robust • Usable • Accessible • Comprehensible • Future Plans

8. Methods and Tools - transferable • Prior contact by email • this is what will be happening • summary of methods • On-line survey/questionnaire • Interview with project staff • Selective audit of project documentation, version tracking and testing procedures

9. Methods and Tools - not so transferable? • Lab testing the software on various platforms and in various conditions • Selective checking of sections of code • Usability walk through and application of heuristics

10. About Epic and EpiCentre • Epic is the UK’s leading e-learning production house • Private and public sector clients • Over 200 staff • ISO9001 since 1992 • Dedicated test house

11. Epicentre test lab • All major OEM PCs and Apple Macs • Windows, Mac and Linux environments • Range of branded and configurable hardware • Range of components and peripherals, e.g. graphics cards, sound cards, processors, modems • Isolated networks - Windows/Mac, LAN/Wireless • Full set of native language operating systems and browsers • Electronic and physical isolation • Sophisticated test lab infrastructure

12. The test lab

14. What does the testing address? • Does it work? • Is it robust? • Is the code extensible? • Is the code maintainable?

15. Lab testing • Robustness and Stress Testing • Functional Testing • Compatibility Testing • Destructive Testing • Stress Testing • Accessibility

16. Code reviewing • How well is the code structured? • How efficient is the code? • How readable is the code? • How clear are the comments in the code? • How do the answers to these questions affect the extensibility and maintainability of the code?

17. Introduction • Futurate’s role • What is usability? • Producing usable software • How we evaluated usability • Alternatives

18. About Futurate • Involved in interviews, desk audit and usability testing • Applications, elearning and web content development • 8 people • Wide experience and multi-disciplinary team • Usability and accessibility consultants • Extensive technical expertise • Experienced trainers and mentors • Education, commercial and not-for-profit clients

19. Usability “Usability is the degree to which something - software, hardware or anything else - is easy to use and a good fit for the people who use it.” Usability Professionals’ Association

20. Producing usable software • Know your users and their tasks • Test early and test often • “Eat your own dog food” • Encourage criticism

21. How to test • Walk-through use scenarios • Consider different types of user • skills • attitudes • Testers should not be familiar with the software • User guides and on-screen help may be provided

22. Usability heuristics • Heuristics, or rules of thumb, can help guide testing • Example heuristics: • error messages should be meaningful and should propose a solution • use of colour should be consistent and appropriate • navigation items should be logically grouped

23. Report structure • Interaction • Design • Navigation • Written content

24. Usability testing is… • Someone unfamiliar with the software • Working through use scenarios • Testing against heuristics • Producing a structured report, that includes suggestions for improvements … or at least it was for this project.

25. Alternatives • Test with real users • Review projects’ own usability testing

26. Methods and Tools – transferableWhat can we offer you? • Prior contact by email • our text and summary • On-line survey/questionnaire • our text and suggestions for additions from interviews (we used Survey Monkey – poss. BOS?) • Interview with project staff • A Word template which contains all the questions and can be used as a summary of the answers too • Other • Complete criteria and methods documents

27. Methods and Tools – not so transferableWhat can we offer you? • Suggestions: • Note down all your blind alleys • Keep in mind maintainability and extendibility • Keep a diary, useful to you, the developers and any users coming after you • For discussion • Best tool for on-line diaries allowing developer comment, good searching and retrieval and good readability for further new users

28. Questions,comments,discussion