Design and Evaluation of Health Care Information Systems

1. Design and Evaluation of Health Care Information Systems Focus on user involvement TDT4210 Helseinformatikk, 2 Nov. 2005, Gry Seland

2. Outline • Importance of user involvement Moen, A. (2003). A nursing perspective to design and implementation of electronic patient record systems. Journal of Biomedical Informatics, 36(4-5), 375-378. 2. Methods for user involvement Sørby, I. D., Melby, L., & Seland, G. (2005). Using scenarios and drama improvisation for identifying and analysing requirements for mobile electronic patient records. In J. L. Maté & A. Silva (Eds.), Requirement engineering for socio-technical systems. Hersley: Information Science Publishing. 3. Evaluation Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H.-U., Rigby, M., & Talmon, J. (2004). Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the his-eval workshop in innsbruck. International Journal of Medical Informatics, 73(6), 479.Examples of systems? Reference to standard for Human-centred design ISO13407. (1999). Iso 13407 human-centred design processes for interactive systems. London: British Standards Institution.

3. Why involve users? • Complex clinical work situation • Complex system • System must support clinical practice Design and evaluation of EPR systems

4. Challenges related to organizational factors • User behaviour, education and training • Legal and social issues • Cost-benefit of system? • Leadership • Enable development of new practice Design and evaluation of EPR systems

5. Anne Moen: Nursing perspective on EPR design and implementation • Technology, organization, informatics, nursing practice and nursing leadership Design and evaluation of EPR systems

6. Important questions • How can EPR support nurses/clinicians? • How can EPR lead to improved care quality and patient safety? • How can EPR contribute to building clinical knowledge? • Can EPR support collaboration between different health care professional groups? • Can EPR support administration and research? Design and evaluation of EPR systems

7. Examination of nursing/clinical practice • Characteristic of nurses’ work • Characteristic of context of nurses’ work ISO 13407 Standard Human-centred design processes for interactive systems Design and evaluation of EPR systems

8. Examination of nursing/clinical practice • Characteristic of nurses’ work • Characteristic of context of nurses’ work ISO 13407 Standard Human-centred design processes for interactive systems Design and evaluation of EPR systems

9. Nursing/clinical leadership • Responsible for developing nursing practice • Important with early and ongoing involvement in EPR project • Time allocation: Signify the importance of EPR Design and evaluation of EPR systems

10. Methods to understand and involve users of EPR system • Interviews and discussions • Observation studies • Role play • Usability testing Methods

11. Interviews and group discussions What do you need? • From formal to informal discussions • Useful information about practice and needs • Memory constraints • Demand characteristics: Tell what is expected • Don’t know about technological solutions • Not everything is easy to articulate (tacit knowledge) • What about mobility? Methods

12. Observation studies • Video, present in the same room, shadowing • Observe things people don’t talk about • Must wait with questions, cannot interrupt • Difficult to transform observations to design suggestions Methods

13. Role play • Some times it is easier to show than to explain • A role play can be “frozen” and replayed • Design of mobile technology needs methods that make visible the mobility of the situation Methods

14. Role play and prototyping • Technology change practice and vice versa Technology Task analysis, Use case (RUP) Interviews Role play and low-fi prototyping Future Observation studies Future workshops, BPR Present Practice Present Future Methods

15. Role play workshop • Part one: Focus on today’s situation • What is today’s situation? • What is concidered problematic today? • Part two: Tomorrow’s situation • How can technology be helpful in the future? • What cannot technology do? Methods

16. Design in action • Act out a scenario until someone identifies an information need • ”Freeze the scenario” • Choose a prototyping model. Sketch the functionality on paper. • Continue acting Methods

18. Design in action • Act out a scenario until someone identifies an information need • ”Freeze the scenario” • Choose a prototyping model. Sketch the functionality on paper. • Continue acting Methods

19. Usability testing • Early (paper prototypes) and last step between before implementation • Identifying ”breakdowns” • Usability laboratory at NSEP available for master students Methods

20. Evaluation (Ammenwerth et al.) Measuring or exploring properties of the health information systems to inform decisions to be made concerning that system in a specific context. • Summative • Formative (during system development) Evaluation

21. Why evaluate • Assess quality, value, effect and impact of IT in health care • Improve health information applications • Enable evidence-based health informatics profession and practise • Development and implementation is expensive  learn from experience

22. Evaluation questions • Usability (specific users and specific context) • User attitudes • Cost-effectiveness • Organizational and social consequences

23. Barriers to evaluation • Evaluation methods not adapted to health care context • Lack of support for formative evaluation • Evaluation studies do not always answer important questions • Limited value of evaluation reports to others • Evaluation is transdiciplinary, and all academic fields (medical informatics, psychology, health economics etc.) have their own methodologies

24. HIS-EVAL 2003 • Bring together experts from different fields • Problems and barrieres to evaluation • Visions and strategies with regard to evaluation of helth information systems • Long-term and short-term strategies to reach the goals in 2

25. Problems and barriers • Awareness: ”Evaluation is too academic, and have no relevance for developers, decision-makers, users and politicians” • Methodological issues: Not chosing the right methods, poor application • Practical issues: Conflicting interests • Dissemination: Evaluation results are not published

26. Visions and strategies • Awareness: Measurement of success and non-success is an integral part of IS design • Methodology: Methods chosen because of evaluation question, research on methods are valid research themes • Practical: Sufficient funding • Dissemination: Reporting studies for different audiences

27. Implementation of activities • Evaluation portal • Good evaluation practice and reporting • Networks • Awareness • Educate the evaluator

28. Conclusion • User involvement is important because Health Care Organizationss are complex, and health care work is complex • ISO 13407: Important to understand clinical work and context of use • Several methods available for user involvement, including interviews, discussions, observation studies and role play • Evaluation is important, and should be formative, not only summative

29. References Importance of user involvement Moen, A. (2003). A nursing perspective to design and implementation of electronic patient record systems. Journal of Biomedical Informatics, 36(4-5), 375-378. Methods for user involvement Sørby, I. D., Melby, L., & Seland, G. (2005). Using scenarios and drama improvisation for identifying and analysing requirements for mobile electronic patient records. In J. L. Maté & A. Silva (Eds.), Requirement engineering for socio-technical systems. Hersley: Information Science Publishing. Evaluation Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H.-U., Rigby, M., & Talmon, J. (2004). Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the his-eval workshop in innsbruck. International Journal of Medical Informatics, 73(6), 479.Examples of systems? Reference to standard for Human-centred design ISO13407. (1999). Iso 13407 human-centred design processes for interactive systems. London: British Standards Institution.