Evaluating Clinical Information Systems: What to Evaluate, Why and For Whom?

1. Evaluating Clinical Information Systems:What to Evaluate, Why and For Whom? February 2004

2. Reading List • Berridge E-J, Roudsari AV, diabCAL: evaluating computer-aided learning for diabetes patient education, ASLIB Proceedings, 2003; 55: 367-378 • Carson ER, Cramp DG, Morgan A, Roudsari AV,Clinical decision support, systems methodology and telemedicine: their role in the management of chronic disease, IEEE Trans IT Applications in Biomedicine, 1998; 2: 80-88 • Cohen B, Key IT Considerations for the modern healthcare enterprise, Br J Healthcare Comput & Info Mgt, 2002; 19:39-40 • Friedman CP, Wyatt JC, Evaluation Methods in Medical Informatics. New York: Springer Verlag 1997 • Leicester HJ, Roudsari AV, Lehmann ED, Carson ER, Methodological issues in validating decision support systems for insulin dosage adjustment, Artif Intell Med, 1994; 6: 161-173 • Skiadas M, Agroyiannis B, Carson ER, Cramp DG, Fourtounas C, Darnige A, Morgan A, Murley D, Tsavdaris H, Hassomeris C, Skouras C, Telematic home haemodialysis: system design, implementation and evaluation, J Telemed Telecare, 2002; 8: 157-164

3. Evaluation? • Evaluation • Validation • Verification

4. Stakeholders Patients Providers Purchasers Payers Public Health Professionals Politicians Settings Community Primary Healthcare Outpatient Clinics In-patient Settings Tertiary Care Regional Medical Centres Laboratories Pharmacies Evaluation - Wider Perspectives

5. Business Drivers • Shift of emphasis from administrative to clinical functionality, implying a wider range of clinical users • Need to avoid damaging the quality of clinician/patient interaction (importance of new computer input methods) • Need to cater for clinical interaction away from the normal workplace (hence importance of mobile devices) • Requirement for flexibility and configurability • Need for open and extensible systems • Capability for scalability with the growth of the organisation • Need for cost effectiveness (investment considerations)

6. Business Drivers • Value to all relevant healthcare professionals (e.g. in managing the longitudinal patient record • Allowing healthcare organisations to change their business processes quickly to meet market demands • Facilitating growth of the organisations • Effective management of costs

7. Technology Features • Performance • Ease of use • Availability • Configurability • Scalability • Affordability

8. Technology Requirements • Multi-tiered environment. Distinction between: - the user interface (the client tier) e.g. PCs, PDAs, WAP - the business logic (the application tiers) - the databases (the database tier) • Distributed architecture • Load balancing • Fault tolerance • Multi-platform • Object-orientated

9. Purpose of Evaluation(Friedman and Wyatt) • Promotional (to encourage the use of ICT) • Scholarly (to understand the principles of health informatics) • Pragmatic • Ethical • Medico-Legal

10. Stakeholders • Patients • Healthcare workers • Clinics, Hospitals • Regulatory bodies • Taxpayers • Industry, Insurers, HMOs

11. Perspectives • Patient - is it safe? - will it help me? • User - is it fast and accurate? - is it fun? • Purchaser - what is the cost benefit? - is it safe and reliable? • Developer - does it work? - will they use it?

12. Drug v Health Informatics Resource Healthcare worker Decision Action Patient data Patient Disease process Organ function Drug Health informatics resource Abstracted patient data Data advice etc. Healthcare worker Decision Action Patient Disease process Organ function Patient data Drug

13. Clinical Data Systems Clinical databases Communication systems (e.g. PACS) On-line signal processing (e.g. 24 h ECG analysis system) Alert/alarm generator (e.g. ICU monitor, drug interaction system) Laboratory data interpretation Medical image interpretation Clinical Knowledge Systems Computerised textbooks (on CD-ROM) Teaching systems (e.g. interactive multimedia anatomy tutor) Patient simulation programs (e.g. acid-base simulator) Passive knowledge systems (e.g. MEDLINE) Patient-specific advice generators (e.g. MYCIN) Medical robotics Range of Computer-Based Resources in Medicine

14. Questions about the resource Is there a clinical need for it? Does it work? Is it reliable? Is it accurate? Is it fast enough? Is data entry reliable? Are people likely to use it? Which parts cause the effects? How can it be maintained? Hoe can it be improved? Questions about the impact of the resource Do people use it? Do people like it? Does it improve users’ efficiency? Does it influence data collection? Does it influence users’ decisions? For how long do the observed effects last? Does it influence users’ knowledge or skills? Does it help patients? Does it change resource consumption? What might ensue from widespread use? Possible Questions Arising During Evaluation

15. Some Deeper Definitions • Evaluation is the systematic application of social research procedures to judge and improve the way information resources are designed and implemented (Rossi and Freeman) • Evaluation is the process of describing the implementation of an information resource and judging its merit and worth (Guba and Lincoln) • Evaluation leads to the settled opinion that something about an information resource is the case, usually but not always leading to a decision to act in a certain way (House)

16. The Evaluation Mindset • Tailor the study to the problem • Collect data useful for making decisions • Look for intended and unintended effects • Study the resource while it is under development and after it is installed (formative and summative evaluation) • Study the resource in the laboratory and in the field • Go beyond the developer’s point of view • Take the environment into account • Let the key issues emerge over time • Be methodologically catholic and eclectic

17. Philosophical Bases of Evaluation • Objectivist - gold standard - statistics • Subjectivist - dependence on the observer • Issues of measurement • Issues of value

18. Objectivist Approaches • Comparison-based approaches - using experiments to compare the informatics system to a control • Objectives-based approaches - determining whether the informatics system meets its designers’ objectives • Decision facilitation approaches - resolving issues important to developers and administrators so that they can make decisions about the future of the resource • Goal-free approaches - evaluators blinded to the intended effects of the IT resource

19. Subjectivist Approaches • Quasi-legal approach - based on a mock trial with a jury deciding on the merits of the IT resource • Art criticism approach - employing formal methods of criticism, the critic producing a critical report on the IT resource • Professional review approach - site visit by an experienced panel of peer reviewers • Responsive/illuminative approach - seeking to represent (and illuminate) the views of users of the IT resource

20. Two Methods for Measuring the Quality of Clinical Decisions Case abstracts Case simulator Test data Decision maker Decision proposed action • Actual decision • real action Patients Real data Decision maker

21. Changing Evaluation Issues During the Development Process Define Build the need prototypes Participatory design Disseminate, monitor Debugging, informal testing Best prototype Problems Study structure, function Problems Problems Best prototype Study impact on users Study impact on patients

22. Validity and Its Estimation:A Focus on Measurement • Content Validity - Do the observations appear to address the attribute that is the measurement target? • Criterion Related Validity - Do the results of a measurement process correlate with some external standard or predict an outcome of particular interest? • Construct Validity - Do the results of this measurement correlate highly with other measures (constructs) which theoretically would be expected to be closely related to this one?

23. Validity (in the context of modelling) • Theoretical validity • Empirical validity • Heuristic validity • Pragmatic validity

24. Rating Item with a Graphic Response Scale(Visual Analogue Scale) This patient was managed: Without any With multiple serious errors serious errors or or deficiencies deficiencies

25. An “Optimism “ Scale for Health Informatics Effect of computers on: Highly detrimental Highly beneficial Cost of healthcare 1 2 3 4 5 Clinician autonomy 1 2 3 4 5 Quality of healthcare 1 2 3 4 5 Interactions within the healthcare team 1 2 3 4 5 Role of govt. in healthcare 1 2 3 4 5 Access to healthcare in remote areas 1 2 3 4 5 Mgt. of medical ethical dilemmas 1 2 3 4 5 Enjoyment of practising medicine 1 2 3 4 5 Status of medicine as a profession 1 2 3 4 5 Continuing medical education 1 2 3 4 5 Rapport between clinician and patient 1 2 3 4 5 Access to up-to-date clinical knowledge 1 2 3 4 5 Patient satisfaction 1 2 3 4 5

26. Evaluation of Diabetic Systems System for Data Collection and Management of the Doctor/Patient Consultation (Rule-based) • Ingredients of the encounter • Assessing patient status • Test/investigation ordering • Drug prescribing • Scheduling the next appointment

27. Evaluation of Diabetic Systems System for Data Collection and Management of the Doctor/Patient Consultation (Rule-based) • Validation of expert system rule-set (more than 700 rules) • Technical evaluation • Impact on the diabetic clinic

28. Technical Evaluation • Speed of operation • Keystrokes

29. Impact on the Diabetic Clinic • Reliability of data collection system • Effects of data collection system on clinic dynamics • Reactions of clinicians using the system • Impact (stress) on patients • Impact on style of consultation • Accuracy and completeness of data recorded

30. Role of a Consensus Panel Clinician 1 System 1 Level 2: Peer review Level 1: Advice generation Clinician 2 System 2 Clinician 3 System 3 Consensus Panel Clinical Data Clinician 4 Clinician 5 Clinician 6

31. Example Questionnaire Questionnaire Patient 1Patient 2 Please answer with a value between 1 A clinician Com- A clinician Com- (negative answer) and 10 (positive answer) from level 1 puter from level 1 puter 1. Was enough patient data/ 7 8 8 8 information available to make an appropriate decision? 2. Do you feel that the advice 7 2 8 6 given will be effective in terms of Blood Glucose control? 3. Would you be satisfied if the 6 2 8 6 same advice was given in a real clinical situation? 4. Would your advice be similar? 7 2 8 5 5. Are you able to identify the 1 1 1 1 advice source?

32. Case Study 1 Telematic Home/Satellite Haemodialysis

33. Introduction: the Re-engineering of Healthcare Healthcare and social welfare systems are being re-engineered almost universally. Drivers for this include: • The need to contain costs • Technological advances in healthcare delivery, particularly with advances in information and communications technologies (ICT) • Demographic changes (aging populations in developed countries) • Increasing expectations on the part of patients an clients Amongst the radical change in healthcare delivery are: • Expanding roles for primary and community care sectors • Increase in day case/laporoscopic surgery • Call centres, e.g. NHS Direct, and similar triaging systems • Wider adoption of eHealth and telemedicine

34. ICT and the Need to Assess its Effectiveness and Efficiency The future of healthcare delivery will be predicated by two factors: • The provision of an infrastructure based on ICT (e.g. the planned total provision in the UK of electronic health records by 2006) • The availability of healthcare and other professionals able to utilise such infrastructure in order to deliver healthcare in the best possible way from the perspectives of effectiveness and efficiency To assess the effectiveness and efficiency of such ICT provision (e.g. to maximise the value of telecare as in the case of telematic home (or satellite) haemodialysis): • Appropriate schemes of measurement need to be in place • Such measurement schemes are best provided in the context of an overall systemic, model-based conceptual framework

35. A Systemic Model-Based Framework • The need for effective public health provision is complex and requires a systems approach to capture the essence of the multi-attribute, multivariable, multi-perspective nature of the public health problem • Whilst clinical and technical advances in healthcare delivery attract the most attention, it should be noted that the most difficult issues to deal with are usually the social, economic and political ones • A systems approach can encapsulate all the causal relations and influences in relation to the system of interest. In the context of end-stage renal disease, this would embrace both the on-going course of the disease process and its impact on patient state as much as the individual episodes of maintenance haemodialysis

36. Conceptual Modelling Approaches A range of modelling approaches has been adopted to express in conceptual terms the issue of modelling healthcare provision and its management (e.g. most recently the disease management equation approach of Albisser et al.) A simplified cybernetic paradigm includes: • Healthcare management (the decision makers) • Healthcare service provision (the effecting agent) • The target population • The information system providing feedback regarding health status At the simplest level, the population will often only be assessed in terms of their perceived health needs.

37. Simplified Cybernetic Conceptual Modelling Framework

38. Modified Generic Signed Digraph

39. Modified Signed Digraph Representation of the Public Health Setting The figure describes both causal connectivity amongst key variables, and also direction of change. Thus a positive sign indicates that a change in one direction (e.g. an increase in value) in a source variable results in a corresponding direction of change in the effected or sink variable. The upper part of the figure shows the population needs driven loops which catalyse changes in service organisation • All feedback loops are negative, implying inherent stability • Increased resource provision is treated as an exogenous variable in a way which facilitates scenario analysis to assess the impact of alternative strategies • Quality of care is assumed to be associated with positive feedback loops, suggesting that stability is dependent on nonlinear relationships

40. Modified Generic Signed Digraph

41. Modified Signed Digraph Representation of the Public Health Setting (2) The lower portion of the figure concentrates on population profile as a driver and determinant of population public health needs • The health outcome of the population treated within the healthcare system of interest influences the demographic characteristics • These, together with the other factors, include bringing about change within the population profile and hence the need for healthcare Provision is made for: • Demographic features • Factors affecting health over which the individual has no direct control (biological, genetic, behavioural, social , environmental, together with propensity for self control) • Lifestyle factors (involving personal choices)

42. Modified Generic Signed Digraph

44. An Example of ICT Driven Healthcare Delivery: Telematic Haemodialysis • Management of end-stage renal disease is a major clinical challenge • Home or satellite haemodialysis is convenient for the patient and efficient, but its take-up has diminished over the past 20 years in part due to lack of support for the patient in the home setting • ICT-enabled telematic home or satellite haemodialysis can overcome these problems through the provision of appropriate telematic links • Such a prototype system has been developed in the EU-funded HOMER-D project • The telematic system was designed adopting a model-based approach

45. Applying the Model to Telematic Home Haemodialysis • The conceptual modelling framework enables the overall value of the proposed ICT implementation of home/satellite haemodialysis to be assessed from a public health perspective • Table 1 shows how the generic variables, relating to population needs driven loops can be mapped to correspond to telematic home haemodialysis • Table 2 focuses on the population profile as a needs driver, defining the variables as they apply to telematic home haemodialysis

47. Population Needs Driven Loops: Telematic Haemodialysis Case Study __________________________________________________________________________________________________ GENERIC VARIABLES TELEMATIC HAEMODIALYSIS VARIABLES __________________________________________________________________________ Health problems Prevalence of end-stage renal disease Population (public health) needs Need for available and accessible maintenance haemodialysis Need for health service Need for telematic haemodialysis service organisation (provision) capability Resources Resources for telematic haemodialysis services Effectiveness of service organisation Effectiveness of provision of telematic haemodialysis service Level of service provision Level of provision of telematic haemodialysis Degree of accessibility Degree of accessibility of telematic services

48. Population Needs Driven Loops: Telematic Haemodialysis Case Study __________________________________________________________________________________________________ GENERIC VARIABLES TELEMATIC HAEMODIALYSIS VARIABLES __________________________________________________________________________ Health service management Performance of agencies responsible for telematic performance haemodialysis Quality of care Quality of provision of telematic care provided to patients with end-stage renal disease Service output Completion of haemodialysis session Service outcome Quality of health subsequent to telematic haemodialysis session Health outcome of treated Health outcome of patients treated by way of population telematic home/satellite haemodialysis

49. Population Profile as a Needs Driver: Telematic Haemodialysis Case Study ____________________________________________________________________________________ GENERIC VARIABLES TELEMATIC HAEMODIALYSIS VARIABLES __________________________________________________________________________ Population profile Prevalence of end-stage renal disease Demographic characteristics Demographic characteristics of patients with end-stage renal disease Treated population characteristics Characteristics of patient population treated with telematic home/satellite haemodialysis Factors affecting health Factors affecting renal health - Biological/Genetic - Biological/Genetic - Behavioural - Behavioural - Social/Environmental - Social/Environmental - Propensity for self-care - Propensity for self-care Lifestyle factors Lifestyle factors affecting renal status - Choice - Choice

50. Information Decisions Actions Outcomes Information Patient Medical Record Clinical Decision Maker Effector System Patient State Information Monitoring System