How do lawyers behave when using electronic resources?

1. How do lawyers behave when using electronic resources? Stephann Makri s.makri@ucl.ac.uk University College London Interaction Centre Department of Information Studies Work supported by an EPSRC DTA Studentship. GSLIS visit supported by an ESRC Post-Doctoral Fellowship. 9th June 2009

2. A naturalistic study of lawyers’ interactive information behaviour when using electronic resources… … leading to the development of two novel methods for evaluating the functionality and usability of these resources.

3. Outline • Research motivation. • Study of lawyers’ information behaviour, feeding into…. • The development of the Information Behaviour evaluation methods. • Formative evaluation of the IB methods with a small group of stakeholders working for LexisNexis Butterworths. • Future work.

4. Research motivation (1) • An understanding of users’ information work can feed into the design or evaluation of electronic resources to support this work. • Information work is highly important for lawyers, but electronic legal resources are complicated to use. • Improving these resources can make lawyers’ information work easier.

5. Research motivation (2) • We don’t blame lawyers for information-seeking difficulties. • We don’t seek to design users. • We are motivated by the prospect of improving electronic legal resources by ensuring they are designed to fit users’ needs. • Other projects provided exciting opportunities for: • Working with the large London office of a multi-national law firm. • Working with LexisNexis Butterworths.

6. The Justis digital law library Screenshot used with permission

7. Methods employed Research structure Combined in-depth interviews and naturalistic observations of lawyers using existing resources Naturalistic interviews/ observations of lawyers Identification of information behaviours Development of Information Behaviour methods Evaluation of methods Refinement of Ellis’s model based on lawyers’ behaviour Identification of law-specific and Information use behaviours Identification of behavioural subtypes and levels Key contributions

8. Methods employed Observations of IB methods in use Combined in-depth interviews and naturalistic observations of lawyers using existing resources Focus groups and summary questionnaires examining the usefulness, usability, learnability and likeliness of future use of IB methods Naturalistic interviews/ observations of lawyers Identification of information behaviours Development of Information Behaviour methods Evaluation of methods Refinement of Ellis’s model based on lawyers’ behaviour IB functionality method Evaluation of IB functionality & usability methods Identification of law-specific and Information use behaviours IB usability method Identification of behavioural subtypes and levels Key contributions

9. Examining lawyers’ information behaviour (1) • Combined in-depth interviews/naturalistic observations of a vertical slice of academic and practicing lawyers. • 32 academic lawyers (students/staff at all levels). • 24 practicing lawyers working for London branch of a large multinational law firm, split across contentious and non-contentious departments and from Trainees to Associate level. • During observation, lawyers used existing electronic legal resources to ‘find information currently or recently needed’ for their work. • Lawyers verbalised thoughts, feelings and actions. • Observer asked opportunistic questions about what lawyers were doing and why.

10. Examining lawyers’ information behaviour (2) • Goal-driven adoption and adaptation of methodological approaches: • Strauss and Corbin’s (1998) Grounded Theory methodology • Beyer and Holtzblatt’s (1998) Contextual Inquiry method • Ericsson and Simon’s (1984) Protocol Analysis. • Encapsulated many core principles, but deviated from some aspects to achieve our goal. • Deviated from GT by only using open and axial coding (i.e. stopped short of generating a theory). • Deviated from CI by asking participants to think aloud. • Deviated from PA by asking questions during the think-aloud session and not using a pre-defined coding scheme.

11. Examining lawyers’ information behaviour (3) • Study identified a number of information behaviours. • Many similar to those previously found in a variety of disciplines: • Accessing, Surveying, Monitoring, Searching, Browsing, Distinguishing, Filtering, Extracting, Chaining. • Some new behaviours pertinent to information-seeking in the legal domain: • Updating, history tracking. • Some new information use behaviours: • Analysing, synthesising, recording, collating, editing, distributing.

12. A law-specific behaviour: Updating Screenshot used with permission

13. An information use behaviour: Distributing Screenshot used with permission

14. Examining lawyers’ information behaviour (4) • Study also: • ValidatedEllis’s model through a new method of observations as opposed to interviews. • Enhanced potential analytical detail of Ellis’s model through identification of a range of ‘subtypes’ of behaviours and levels at which behaviours can operate.

15. Resource level Source level Document level Content level Levels at which behaviours can operate

16. The IB functionality method (1) • Uses information behaviours as a framework for evaluating electronic resource functionality. • For each behaviour, we can ask whether the resource currently supports the behaviour. • If it does currently support the behaviour, we can consider: • The ways in which it currently supports the behaviour. • The additional ways in which it might support the behaviour. • If it doesnot currently support the behaviour, we can consider the ways in which it might support it.

17. The IB functionality method (2) • We can also consider: • Whether there are any behaviours that may no longer be necessary to support • Whether there are any ways that the resource currently supports a particular behaviour that may no longer be necessary.

18. The IB usability method • To evaluate the usability of electronic resources it is possible to: • Set behaviour-focused tasks to intended/actual users and ask them to think aloud whilst performing them. • Then identify usability issues from the think-aloud data. • Example updating tasks include: • Try to find out if a particular legal case is still good law. • Try to find out if a particular piece of legislation is currently in force.

19. Example form used to record usability issues

20. Now it’s your turn… • Watch this 5 min. narrated clip of a Trainee Solicitor using LexisNexis Butterworths to perform the updating task of ‘finding out whether a particular case or piece of legislation is still good law.’ • Note down any potential difficulties the user faces.

21. Clip shown with permission from LexisNexis Butterworths

22. Potential usability issues from clip • Unclear how to achieve updating task. • Usage of segmented fields unclear. • Search syntax unclear/inflexible? • ‘Next steps’ actions not as transparent as they could be. • Help page unhelpful, tutorial ‘cumbersome.’ • ‘Go’ button feedback (grey upon submission) unclear. • Unclear relationship between search buttons and associated search fields. • ‘View tagged’ action not immediately clear. • Unclear effect of selecting current document from ‘view’ combo.

23. Evaluation of the IB methods (1) • Developed iteratively and refined based on tests with ‘real users’ (i.e. potential evaluators). • One-day tutorial conducted with a small group of stakeholders working for LexisNexis Butterworths. • Participants taught information behaviour theory that underpins the methods, how to conduct the methods themselves and given chance to practice. • Hands-on and highly example-based tutorial.

24. Evaluation of the IB methods (2) • Participants conducted hour-long recorded practices using each method, acting as a formative evaluation. • Colleague conducted two 30 minute focus groups with questions focusing on how useful, usable and easy to learn they found the methods to be and how likely they would be to use the methods in future. • Brief Likert-scale summary questionnaires issued.

25. Comment on the IB functionality method P1: I think [the IB functionality method] is potentially useful. […] It is a structured way of facilitating a discussion about features instead of talking about features in general. So from that standpoint it was useful and out of that discussion came some new ideas like ‘how can we access our products from documents?’ We’d previously thought about it a little, but because of that structure it allowed us to really zoom in on it and talk about it in a structured way.

26. Comment on the IB usability method P10: I do a lot of work on user assistance and I’m always trying to find out what the problems are that users are trying to solve and [the IB usability method] is quite a thorough way of going about answering that question, because you ask users to solve a prescribed task – you watch it, you analyse it, and at the end the conclusions are quite obvious about the problems that they face. You don’t address those problems, you just try to find out what they are. It allows you to think about ‘What is it that they are trying to do?’ ‘What are the pitfallsthat may be consistent between different users?’ and address them and help the user. […] I think the greatest benefit is that you are looking at user behaviour and addressing user needs rather than taking a more system-centred approach, which is good. That’s the whole aim of usability.

28. Evaluation of the IB methods (3) • Generally encouraging feedback. • Suggestions for improving methods and tutorial materials included: • Considering functionality reduction at a summary level as opposed to for each behaviour/level. • Including practice in devising behaviour-focused tasks, not just analysing them. • Create electronic/spreadsheet versions of forms.

29. Summary and contributions • An empirical study into academic and practicing lawyers’ interactive information behaviour, leading to: • The identification of a number of information behaviours and • The validation, extension, broadening of scope and enhancement of the potential analytical detail of Ellis’s behavioural model of information-seeking. • The development and formative evaluation of the Information Behaviour (IB) methods

30. Future work • Studying information behaviour in non-legal domains. • ESRC-funded extension work on how Architectural students look for and use electronic information. • MSc project examining developers’ behaviour when finding and choosing web services. • Further examining lawyers’ information behaviour. • MSc project examining ‘monitoring’ behaviour in more detail. • Conducting further development and evaluation of the IB methods. • Well, 2 out of 3 ain’t bad!

31. Thank you for listening! • Key related articles listed below. • Also a handful of other related publications. • All downloadable from my website (just Google ‘Stephann’). • Makri, S., Blandford, A. & Cox, A. L. (2007). Investigating the information-seeking behaviour of academic lawyers: From Ellis's model to design. Information Processing and Management 44(2), pp. 613-634. • Makri, S., Blandford, A. & Cox, A. L. (2008). Using Information Behaviours to evaluate the functionality and usability of electronic resources: From Ellis’s model to evaluation. Journal of the American Society for Information Science and Technology 59(14), pp. 2244-2267.