1 / 84

Evaluation of information systems

Evaluation of information systems. Nicolette de Keizer Dept Medical Informatics AMC - University of Amsterdam. Outline. Significance of evaluation Process of evaluation Evaluation questions Methods – study design, data collection Triangulation. IT in health care: possible benefits.

shani
Download Presentation

Evaluation of information systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of information systems Nicolette de Keizer Dept Medical Informatics AMC - University of Amsterdam

  2. Outline • Significance of evaluation • Process of evaluation • Evaluation questions • Methods – study design, data collection • Triangulation

  3. IT in health care: possible benefits • Reduction of medication error by CPOE • Better treatment / diagnostics by decision-support systems • Increase quality of documentation • Reduce costs by telemedical applications

  4. Unintended Consequences of Information Technologies • Aim • Determine the effect on mortality of introducing CPOE into Pittsburgh childrens hospital • Methods • Demography, clinical and mortality data collected on all children transported to a hospital where CPOE implemented institution-wide in 6 days. Trends for 13 months prior and 5 months after compared. • Results • Mortality rate increased from 2.80% (39 of 1394) to 6.57% (36 of 548) • After adjustment for other covariables, CPOE independently associated with increased odds of mortality (odds ratio 3.28, 95% C.I. 1.94 – 5.55)

  5. Unintended Consequences of Information Technologies • Conclusion • When implementing CPOE systems, institutions should continue to evaluate mortality effects, in addition to medication error rates • Importance • Received disproportionate media attention due to reactionary message • Follow-on study in Seattle, using same vendor system, also published in Pediatrics, showed no increase in mortality

  6. Negative effects of CPOE: Example 2 • Brigham and Womens' Hospital, Boston introduced a CPOE system, that allows physicians to order the medication online (and not on paper anymore). • After implementation, the rate of intercepted Adverse Drug Events (ADE) doubled! • Reason: The system allowed to easily order much too large dosages of potassium chloride without clear indicating that it be given in divided doses. • Bates et al The impact of computerized physician order entry on medication error prevention. JAMIA 1999, 6(4), 313-21.

  7. Unintended Consequences of Information Technologies • Reference • Linder et al., Arch Intern Med. 2007 Jul 9;167(13):1400-5. [Brigham & Women’s Hospital] • Aim • Assess effects of Electronic Health Records on quality of care delivered in ambulatory settings • Methods • Retrospective, cross-sectional analysis of 17 quality measures from 2003-2004 National Ambulatory Medical Care Survey, correlated with use of EHRs.

  8. Unintended Consequences of Information Technologies • Results • EHRs used in 18% of 1.8 billion visits • For 14 of 17 quality measures, fraction of visits where recommended best practice occurred was no different in EHR settings than manual records settings. • 2 better with EHR: avoiding benzodiazepines in depression, avoiding routine urinalysis • 1 worse with EHR: prescribing statins for hypercholesteremia (33% vs. 47%, p=0.01) • Conclusion • As implemented, EHRs not associated with better quality ambulatory care

  9. Unintended Consequences of Information Technologies • Reference • Linder et al., Arch Intern Med. 2007 Jul 9;167(13):1400-5. • Importance • Received disproportionate media attention due to reactionary message • Lost in the media hype: Less than 40% of EHR implementations have all elements important for effects on quality (e-prescribing, test ordering, results, clinical notes, decision support). • Best performance regardless of infrastructure was suboptimal (< 50% adherence to best practice).

  10. Other examples • LondonAmbulance Dispatch System collapsed due to inadequate testing. Thousands of emergency calls were answered not or too late. http://www.cs.ucl.ac.uk/staff/A.Finkelstein/las.html • The malfunction of Therac-25, a medical linear accelerator caused the death of three patients in the late 1980th. http://courses.cs.vt.edu/~cs3604/lib/Therac_25/Therac_1.html • University hospital stops introduction of Order Entry System due to user boycott. Costs: up to 34 MIO dollars • Ornstein C (2003) Los Angeles Times Jan 22nd, 2003

  11. Insufficiently designed, badly integrated or wrongly used IT systems can lead to user frustration and errors. „Bad health informatics can kill“ Ammenwerth E, Shaw NT. Bad health informatics can kill - is evaluation the answer? Methods of Information in Medicine 2005;44:1-3. Examples: http://iig.umit.at/efmi/ -> Bad Health Informatics can Kill

  12. Need for Evaluation IT systems can have large impact on quality of care IT systems are costly IT systems can fail or be sub-optimal designed Evaluation is a way to provide better IT Systems

  13. Systematic Evaluation of IT is essential • Formative (constructive): evaluation looking forward • Summative: evaluation looking backward

  14. Evaluation: Definition (1/2) • Evaluation is • the act of measuring or exploring properties • of a health information system (in planning, in development, in implementation, or in operation), • the result of which informs a decision to be made concerning that system in a specific context. • Ammenwerth E, Brender J, Nykänen P, Prokosch U, Rigby M, Talmon J. Visions and strategies to improve evaluations of health information systems - reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 2004 Jun 30;73(6):479-91.

  15. Evaluated types of information systems 1982- 2002 (n = 1.035)

  16. Annualy published papers in PubMed on IT evaluation in health care (PubMed analysis) 1.0% of all papers 0.6% of all papers Ammenwerth, de Keizer (2004)

  17. Evaluation in Informatics is Notoriously Difficult • We live in a pluralistic world. There will be many points of view on need, benefit, quality. • Real people using deployed technology: things can go wrong or right for very complicated reasons • Intersection of 3 domains where progress is very rapid: • Work in the domain (health care, bio-science) • Information technology • Evaluation methods • Sometimes they don’t really want to know...

  18. Outline • Significance of evaluation • Process of evaluation • Evaluation questions • Methods – study design, data collection • Triangulation • Publication bias

  19. Negotiation Questions Investigation Report "Contract" Decisions The General Process of Evaluation

  20. Questions Investigation Report Negotiation and Contract • Identify the primary audience(s) and interact with them Negotiation "Contract"

  21. Development Director's Superiors Resource Users and Their Clients Peers, Relations of Clients Roles in Evaluation: The Playing Field Evaluation Funder Public Interest Evaluation Team Groups and Director Professional Societies Staff Development Team Director Development Staff Funder Those Who Use Similar Resources

  22. Questions Investigation Report Negotiation and Contract • Identify the primary audience(s) and interact with them • Set general goals and purposes of the study • Identify, in general, the methods to be used • Identify permissions, accesses, confidentiality issues and other key administrative aspects of the study • Describe the result reporting process • Reflect this in a written agreement Negotiation "Contract"

  23. Negotiation Investigation Report "Contract" Questions • Specific questions derived from the general • Maximum 5-10 • They do not have to be stated as hypotheses • Depending on methods used, the questions can change over the course of the study Questions

  24. Investigation • Choose data collection methods • There are two major families of investigational approaches: objectivist and subjectivist • Although some studies use both families, typically you will choose one or the other Negotiation Questions Investigation Report "Contract"

  25. Report • Process of communicating findings: reporting is often done in stages • It doesn’t have to be a written document exclusively • Targeted at the audience(s), in language they can understand • Report must conform the evaluation agreement Negotiation Questions Investigation Report "Contract"

  26. Subjectivistic vs. objectivistic Subjectivistic (interpretative, explorative) approaches hypothesis Objectivistic (positivistic, explanative) approaches Generate hypothesis Open and broad Detect relationships Inductive Focus on qualitative methods Test hypothesis Focused and exact Prove relationships Deductive Focus on quantitative methods

  27. Group work • Judge abstracts 1 and 2: • Subjectivist or objectivist study?

  28. Outline • Significance of evaluation • Process of evaluation • Evaluation questions • Methods – study design, data collection • Triangulation

  29. Evaluation and IT life cycle Information needs Resource impact: effects, acceptance, costs, benefits, … Software Engineering: Verification, validation, Usability, stability, …

  30. Group work • What kind of aspects do you want to evaluate if your hospital implement a nursing documentation system or a DSS application? • Physician • Nurse • Manager • Developer

  31. Measures for IT evaluation studies 1. Static IT attributes (hardware and software quality) Static user attributes (computer knowledge) 2. Quality of interaction between IT and user (e.g. usage patterns, user satisfaction, data quality) 3. Effects of IT on process quality of care (efficiency, appropriatness, organisational aspects) 4. Effects of IT on outcome quality of care (quality of care, costs of care, patient satisfaction)

  32. Evaluted aspects in evaluation studies 1982- 2002 (n = 983) Ammenwerth, de Keizer (2004)

  33. Why formulate questions ? • Crystallize thinking of evaluators and key “stakeholders” • There is a need to focus and prioritize • It converts broad aims into specific questions that can potentially be answered

  34. Further benefits of identifying questions • Stakeholders can see where their concerns are being addressed • The choice of methods follows from the question • A list discourages evaluators from focusing only on questions amenable to their preferred methods

  35. Evaluation question: Recommendations • Evaluation is part of the whole IT life cycle • Any evaluation must have a clear evaluation question (there is no „global“ or „absolute“ evaluation). • The evaluation questions should be decided by the stakeholders. Take the time for elaborating clear and agreed evaluation questions!

  36. Outline • Significance of evaluation • Process of evaluation • Evaluation questions • Methods – data collection & study design • Triangulation

  37. Evaluation question Evaluation methods Data Answer to evaluation question

  38. Evaluation generates data to answer questions 85% of care plans are incomplete. Mean time for data entry: 3.5 min. „It does not work!“ Which types of data are represented? „I like it“ Costs of 3.500 Euro per workstation. Nurse tries to enter password several times. 4 medication errors per day. Physician curses at the computer. Mean user satisfaction: 1.9 (from 5 max.)

  39. Data Quantitative data: Numbers Qualitative data: Text, videos, … Count, measure, weight, … Describe, observe, … Positive attributes? Rich in content Needs less standardization Needs no large numbers Generates exact results Easy to work with Easier to aggregate

  40. Quantitative vs. qualitative methods in evaluation studies 1982- 2002 (n = 983) Ammenwerth, de Keizer (2004)

  41. Example 1a: Does documentation system reduce time? Introduction of nursing documentation system Reduce time efforts for documentation Hypothesis: causal relationship Quantitative RCT study

  42. Example 1b: What are effects of documentation system? Reduce time efforts for documentation Improve quality of documentation Improve communication in team Improve transparency of nursing care Improve IT skills Improve communication with physicians Improve quality of nursing care Qualitative ethnographic study

  43. Example 2a: Which factors determine user satisfaction with a nursing documentation system? Attitude towards nursing process Performance and Stability of the system Age of nurse Computer experience (in years) Quality of support Attitude towards computers in nursing Quality of training Qualitative interview study

  44. Example 2b: Does age or quality of training determine user satisfaction with a nursing documentation system? Quality of training User satisfaction Hypothesis: relationship Design: e.g. RCT, observational Age of nurse User satisfaction Hypothesis: relationship Quantitative study

  45. Kinds of evaluation study Evaluation studies Objectivist studies Subjectivist studies

  46. Kinds of evaluation study Evaluation studies Objectivist studies Subjectivist studies Demonstration studies Measurement studies Descriptive studies Reliability studies Correlational studies Validity studies Comparative studies

  47. Descriptive studies Aim: to describe something Example: how often do doctors use a CPOE? Methods: survey, log file, observation, case note audit… Variables: single variable of interest – the “dependent” variable (usage rate) Analysis: simple descriptive statistics – mean & SD; median & inter-quartile range…

  48. Correlational studies Aim: to correlate something with something else Example: is CPOE use associated with less calls from pharmacy to department ? Methods: survey, log file, observation, case note audit… Variables: “dependent” variable (calls) + independent variables (usage rate, age…) Analysis: univariate or multivariate regression

More Related