Approaches to Evaluating and Measuring Outcomes in Integrated Care Key Issues for Consideration

1. Approaches to Evaluating and Measuring Outcomes in Integrated CareKey Issues for Consideration Dr Nick Goodwin CEO, International Foundation for Integrated Care Senior Associate, The King’s Fund Paper to: Health Quality and Safety Commission New Zealand Workshop: Towards Integrated Care in New Zealand Wellington, New Zealand, November 14th2013

2. What is a ‘programme evaluation’? • A systematic method of collecting, analysing and using information to answer questions about projects, policies or programmes • In health care systems, they are particularly concerned with quality of care, patient safety, system efficiency and/or cost effectiveness • They also seek to examine whether programme goals are, of have been, appropriate and/or useful so can be used to change and adapt strategic directions • They tend to utilise both qualitative as well as quantitative methods • They can be ‘formative’ or ‘summative’

3. Typical components of a ‘programme evaluation’ • Assessment of the need for the programme • Needs assessment/gap analysis/population health planning • Assessment of the design and/or theory and logic of the programme in supporting its desired influence • Are the assumptions of the programme justified? • Assessment of how the programme is being implemented • Process evaluation - is it going to plan? • Assessment of the programmes outcome or impact; cost and efficiency • Outcome level, outcome change, programme’s effect Rossi, Lipsey and Freeman (2004) Evaluation: a systems approach, Sage

4. Understanding what to evaluatein an integrated care programme

5. What are you evaluating – some key questions • Who and what is the programme seeking to influence? Need to clarify aim and design of the integrated care intervention by looking at the needs of patients/users • What is the timeframe over which outcomes are expected to be achieved? Given this timeframe, which categories of outcomes have the potential to be improved? • Is there sufficient opportunity in a given population to achieve this targeted improvement in outcomes? • How can you measure the impact? How can you ensure attribution?

6. What are you evaluating – some key questions Before developing questions and/or survey instruments to examine the experience and impact of integrated care from a person’s perspective, there is a need to understand four things: • the programme theory of change – what are the assumptions that lie behind the programme (why?) • the (set of) problems to be addressed (where and who?) • the (set of) interventions best suited to address the problem (what and who?) • the strategy best suited to develop, implement, and evaluate the (set of) interventions (how, when and who?)

7. What are you evaluating – some key questions For integrated care to be successful, it needs to execute the following three functions: • accurate identification of individuals within target population (e.g. reliable predictive modelling, health risk assessment, medication list and/or laboratory values from EMRs); • individuals must be enrolled and actively participate in the program for a meaningful period of time (e.g. readiness to change, motivational interviewing, incentives); • the program must include a set of interventions that modify or close deficits in participant and provider behaviour (e.g. tailoring to needs).

8. Key Points to Consider • Baseline data • Define a comparison group • Define nature and structure of integrated care being implemented • Include measures of the professionals’ perspective where care is delivered through multidisciplinary teams • Identify what good looks like from a patients’ perspective and evaluate this through user feedback • Include analysis of utilisation and costs of care Experiences, care outcomes, utilisation & costs

9. Understanding what to measurein an integrated care programme

10. Measurement Types - 1 Care Outcome Measures • Patient outcomes • e.g. mortality, morbidity, functional status, quality of life • Cost and utilisation outcomes • e.g. hospital admissions, bed days, LOS, nursing home placements Care Process Measures • Occurrence of recommended care activities • e.g. presence of a care plan; patient follow-up – often processes that are set out in best practice guidance The measurement tools we have on outcomes and processes are mostly disease-specific - for people with multiple needs, the process of care is less well understood

11. Examples of outcome measures • Hospital utilisation • Emergency admissions; hospital readmissions; lengths of stay; number of bed days etc … • Disease-specific hospital admissions etc … • ‘Social care’ utilisation • Levels of home care support packages • Rates of long-term nursing home/residential home stays • Mortality and disease-specific mortality • Short-term clinical outcomes • e.g., glycated haemoglobin levels for diabetic patients • Functional status • e.g., for CHF patients • Quality of life • e.g., functional dependence • Other patient outcomes • e.g., missed school days for children due to illness; experiences with system • Treatment and service adherence • e.g., remaining in contact with services for mentally ill patients

12. Measurement Types - 2 Measures of care co-ordination • Information exchanges and transfers • Relational co-ordination between organisations and professionals • levels of awareness/interaction among participants • is there a common understanding of care activities and goals • shared culture and mission?

13. Co-ordination Mechanism Measures - Approaches • Direct observation • Interviews and staff surveys (self-report) • Medical record audits on information transfer • Measures of inter-professional collaboration within teams and organisations Examples: ITMA - Integrated Team Monitoring and Assessment Tool – see http://www.readiness-tools.com/tool-full.aspx?toolguid=0d6382ad-f017-4623-8d10-93f2f314e346 POET – Partnership Outcomes Evaluation Toolkit – see www.dhcarenetworks.org.uk/asset.cfm?aid=1479

14. Care Co-ordination Measures Atlas McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdf and http://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools

15. Domains for measuring care co-ordination Co-ordination activity: • Establish accountability/negotiate responsibility • Communication – informational and inter-personal • Facilitate transitions – e.g. across settings or as coordination needs change • Assess multiple needs and goals • Pro-active care planning • Monitor, follow-up, review • Support self-management • Link or refer to community resources • Align resources to meet individual or community needs Service delivery approaches: • Care management • Medicines management • Healthcare at Home • Multi-disciplinary teams • ICT-enabled integrated care (e.g. telehealth) Perspectives: Family/patient Professional System/organisation

16. Measurement Types - 3 Patient, carer, family-reported perceptions • PROMs • do you feel better? • have your symptoms improved? • PREMs • recommend to a friend? • how satisfied are you? • * PACIC patient assessment of chronic illness care http://www.improvingchroniccare.org/downloads/2004pacic.doc.pdf • * Care Transition Measures (CTM) – patient assessment of discharge from hospital http://www.caretransitions.org/documents/CTM-15.pdf The more ‘robust’ measures are those that ask about specifics of care co-ordination rather than generalities, and which link data sets together to understand association between responses. Two key things: measures that are specific to a particular group of people; measures that translate to actionable service improvements

17. Patient Reported Outcome Measures in England Since 2009, PROMs have been collected for four elective procedures: hip surgery, knee surgery, hernia repair and varicose veins – these costs the English NHS £800m/year Cataract surgery was to be included, but concerns on methods of data collection 250,000 patients a year invited to complete survey – all NHS providers (100%) collect data PROMS being extended to cover: anxiety and depression, cancer care, asthma, COPD, diabetes, epilepsy, heart failure, stroke It’s a careful process – identify the right PROMs instrument; pilot before roll-out; implement data collection; evaluate programme Uses EQ-5D – part 1 about mobility, self-care, usual activities, pain/discomfort, anxiety/depression; part 2 gives overall assessment of health on 0-100 scale NOTE: Not sensitive to looking at people with multiple conditions/needs

18. Some Conclusions on Measuring Integration • Many different tools available: • Need to define the client group • Need to understand the goal in terms of outcomes to patients and service users • Need to create ‘measurable’ outcomes and experiences • Measures need to mean something – i.e. that actions can follow • Patients and users tend to understand the term ‘care co-ordination’ or ‘continuity of care’ – e.g. to what extent they feel that care is co-ordinated around their needs • Baseline on measures required on which to base progress over time • Link measures to other data – e.g. on clinical outcomes, utilisation, costs • Where possible, benchmark performance with others or investigate with a matched ‘control’ • Use data in ‘real time’ to monitor progress and drive performance

19. Case Example Integrated Care Metrics NHS London (2012)

20. Activity metrics currently used in IC systems in London, across all settings of care

21. Process metrics currently used by IC systems in London

22. Outcome metrics currently used by IC systems in London

23. Patient and staff experience measures used by IC systems in London

24. Contact Dr Nick Goodwin CEO, International Foundation for Integrated Care nickgoodwin@integratedcarefoundation.org www.integratedcarefoundation.org @goodwin_nick @IFICinfo

25. Appendix 1: From a measure to an indicator that can be used for quality improvement In developing ‘indicators’ to judge comparative performance, the following criteria are important: • Statistical validity • Accurate – measures what is says it measures! • Reliable – can be tracked over time • Consistent – data collection robust and reproducable • Avoids bias • Data considerations • Data source – sample or full population; existing or new data source? • Unit of assessment – country, region, locality, practice etc • Client groups covered – what is the targeted population? account for bias/case mix? • Significance – smaller groups, reduced statistical power • Patient reported data is indirect – how cross-compare to other data (multi-methods) • Coverage – health, health & social care, health, social care and housing …. • Face validity • Meaningful to public; clinically credible; • Potential to support quality improvement; • Cost and value for money

26. Appendix 2: Key Resources on Care Co-ordination Measures McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010. http://www.ahrq.gov/qual/careatlas/careatlas.pdfttp://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools King et al - 1995/2004 - Measures in the process of care – MPOC http://www.canchild.ca/en/measures/mpoc56_mpoc20.asp Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract. 1997 Jul;45(1):64-74. Safran DG, Kosinski M, Tarlov AR, et al. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care. 1998 May;36(5):728-39. Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer experiences with primary care. Pediatrics. 2000 Apr;105(4 Pt 2):998-1003. Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract. 1998;46(3):216-26. Grimmer K, Moss J. The development, validity and application of a new instrument to assess the quality of discharge planning activities from the community perspective. Int J Qual Health Care. 2001 Apr;13(2):109-16

27. McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of health care. Int J Qual Health Care. 2003 Aug;15(4):309-18. Radwin L, Alster K, Rubin KM. Development and testing of the Oncology Patients' Perceptions of the Quality of Nursing Care Scale. Oncol Nurs Forum. 2003 Mar- Apr;30(2):283-90. Billings J, Coxon K, Alaszewski A. Empirical research methodology for ‘Procare’ version 3. University of Kent at Canterbury: Centre for Health Services Studies; 2003. Available from: http://www.kent.ac.uk/chss/docs/procare_version3.pdf. Morita T, Hirai K, Sakaguchi Y, Maeyama E, Tsuneto S, Shima Y. Measuring the quality of structure and process in end-of-life care from the bereaved family perspective. J Pain Symptom Manage. 2004 Jun;27(6):492-501. Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient's perspective: the care transitions measure. Med Care. 2005 Mar;43(3):246-55. (also CTM website) Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC).Med Care. 2005 May;43(5):436-44 Drewes et al (2012) PCIC+ at http://www.ijic.org/index.php/ijic/article/view/862/1896 Haggerty J, Roberge D, Freeman G, Beaulieu C, Breton M. Review. When patients encounter several providers: validation of a generic measure of continuity of care. Annals of Family Medicine (In press) and Breton et al at http://www.ijic.org/index.php/ijic/article/view/682/1534

28. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): Conceptualizing and Measuring Activation in patients and consumers. Health Services Research 2004: 39:1005-1026. Powell, R. Powell, H. Baker, L. & Greco, M. (2009) Patient Partnership in Care: A new instrument for measuring patient–professional partnership in the treatment of long-term conditions. Journal of Management & Marketing in Healthcare. Vol. 2 No. 4. PP 325–342. Long Term Conditions 6 (LTC 6) (part of the QIPP programme) Department of Health and also Living with your Long Term Condition (LWYLTC) Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al.Measuring Patients' Experiences with Individual Primary Care Physicians Results of a Statewide Demonstration Project. Journal of General Internal Medicine. 2006;21(1):13-21. Adair CE, McDougall GM, Mitton CR, Joyce AS, Wild TC, Gordon A, et al.Continuity of Care and Health Outcomes Among Persons With Severe Mental Illness. Psychiatric Services. 2005;56(9):1061-9. King M, Jones L, Nazareth I. Concern and continuity in the care of cancer patients and their carers: A multi-method approach to enlightened management. London: National Coordinating Centre for NHS Service Delivery and Organisation 2006. Dolovich LR, Nair KM, Ciliska DK, Lee HN, Birch S, Gafni A, et al.The Diabetes Continuity of Care Scale: the development and initial evaluation of a questionnaire that measures continuity of care from the patient perspective. Health Soc.Care Community. 2004;12(6):475-87

29. Kowalyk KM, Hadjistavropoulos HD, Biem HJ. Measuring Continuity of Care for Cardiac Patients: Development of a Patient Self-report Questionnaire. Canadian Journal of Cardiology. 2004;20(2):205-12. Hadjistavropoulos H, Biem H, Sharpe D, Bourgault-Fagnou M, Janzen J. Patient Perceptions of Hospital Discharge: Reliability and Validity of a Patient Continuity of Care Questionnaire. International Journal for Quality in Health Care : Journal of the International Society for Quality in Health Care / ISQua. 2008;20(5):314-23. Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based outpatient clinics: performance measures based on patient perceptions of care. Med Care. 2002;40(7):578-86.