Evaluating usability through claims analysis

1. Evaluating usability through claims analysis Suzette Keith Ann Blandford, Bob Fields, Richard Butterworth, Yin Leng Theng

2. Aims • Testing claims analysis as an analytical usability evaluation technique for digital libraries • Validating claims with user trials • Testing usability of claims analysis for designers • … towards developing a suite of usable usability tools for digital libraries

3. Claims analysis “claims analysis is an analytical evaluation method involving the identification of scenario features that have significant positive and negative usability consequences” (Rosson, Carroll 2002)

4. Case studies • Exploring claims analysis with the developers • Applying claims analysis to an add-on feature • User trial to investigate claims

5. Claims analysis: overview of method 1 Create scenarios - users, goals, story, action 2 Develop description of the artifact 3 Generate claims 4 Analyse claims 5 Reason about design rationale

6. 1 Create scenarios Data sources: • Interviews with users • Cognitive processing model • Research on information seeking • Research on digital library usage …Natasha is a recent graduate investigating a new project. She decides to try the digital library and finds some results…

7. 2 Description of the artifact • Design rationale • Usability specification • Functional specification • Mock-ups • Scripts • Partial implementation • Prototyping …she sees the search-box and types in a query…

9. 3 Generate claims The Action cycle: How does the artifact: • evoke goals in the user? • support planning? • indicate action to take? • guide interpretation? • help evaluation and error recovery?

10. Goal claim ...Natasha is seeking information on a topic to support on-going project activity.. Claim about users goals: The digital library holds appropriate information accessed by keyword searching and browsing

11. 4 Analyse goal claim • User can search or browse for information • but • May not know appropriate terms • Content may not be appropriate

12. 5 Reasoning about design rationale • What factors might cause the user to fail • How can the effect of negative claims be moderated • What trade-offs might effect other parts of the system design

13. Case study 1:Exploring claims Findings from developers: • Developers had designed the ‘best possible’ system • Informal use of novice scenarios • Focus on problems of awareness and exploratory use • Difficult to identify negative consequences of design decisions

14. Case study 2:Applying claims Add-on feature in development: • Hypothetical novice user scenarios • Prototype interface • Developers wanted support for design decisions • Analysis strategy: plan, execute and evaluate

15. Plan? Action? Evaluation?

16. Claim for evaluation ….Natasha enters details into a form, the web-site or document url is notified to all profiled users.. Implemented feature: automatic completion sequence: + rapid execution minimises user effort but - reduced opportunity to correct errors, maximum opportunity for public embarrassment

17. Reasoning about evaluation claim • Trade-off between speed and accuracy • Importance of feedback and consequence of error • Exploration of alternative scenarios • Examination of technical issues • Proposed changes to interface

18. Case study 3:Novice user trials The ‘ Google generation’ • Positive and negative effects of prior experience on goal, planning, action, evaluation • Very specific information seeking task • Difficulty dealing with very large results sets • Progress measured by relevant full-text documents found

19. The challenge …what is exciting about digital libraries is the power they offer ordinary users… The promise that information can be stored,accessed, manipulated and moved in ways that were hereto impossible… (Dillon)

20. Claims analysis: reprise Can claims analysis contribute to the understanding of information seeking and the design of digital libraries ? - task-artifact cycle Can claims analysis be specified to support the development of innovative and creative solutions to usability issues? - concrete yet flexible

21. Thank yous.keith@mdx.ac.ukwww.cs.mdx.ac.uk/ridl