Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear

1. Building a Complex Health Research Logic Model: Making Pathways to Impacts Clear American Evaluation Association Conference November 5, 2011 Gretchen B. Jordan Sandia National Laboratories gbjorda@sandia.gov Parts of work presented here was completed for the U.S. DOE by Sandia National Laboratories, Albuquerque, New Mexico. Opinions expressed are solely those of the author.

2. This talk will demonstrate • A complex, one page logic model for the return on investment for health sciences research • With several features to frame assessments that are similar enough to feed evaluation synthesis • Built on agreed upon theory and definitions to specify aspects of both the healthcare system and the environment that affects health and individual behaviors; • Categorized institutions and actors that are the pathways through which advances get to impact; and • Individual studies look at on pathways to impact, noting other parts of the logic as context. G. Jordan AEA 2011

3. Background and Charge • This logic model was part of a Major Assessment Panel led by the Canadian Academy of Health Sciences (CAHS) to develop a“Preferred Framework and Indicators to Measure Returns on Investment in Health Research.” • Twenty-three different organizationssponsored • Charge to the Expert Panel: Develop a framework or model at the outset of the assessment and demonstrate that the model fulfills the required criteria: • be useful to a range of funders/research types, • be compatible with what is already in place in Canada, • be transferable to international comparisons, and • be able to identify the full spectrum of potential impacts. G. Jordan AEA 2011

4. Model Design • For Canada’s four “pillars of health research” and cross cuts • Impacts at various levels — individual, institutional, provincial, national or international • Track health-research impacts in five main categories: • advancing knowledge • building capacity, • informing decision-making, • health impacts, and • broad socio-economic impacts. • Subcategories identify and partition evaluation methods • A menu of indicators in each sub category • Intermediate outcomes and (eventually) quantify the value(s) of impacts to end-users as a function of dollars invested G. Jordan AEA 2011

5. Model relies on evidence AND is potentially hypothesis generating • The model built on academic and practical studies of • the determinants of health: World Health Organization paper (2006); Evans and Stoddart (1990) and Dahlgren and Whitehead (1991) • diffusion of innovation theories: Everett Rogers (1995), Reed and Jordan (2007) for the five groups influenced by R&D • Synthesis across matched studies can test hypotheses and build theory • ease of synthesis enhanced if studies use similar terminology, good research design, and make clear the full context • a single study cannot be large enough in scope; differences in context are informative; points to features of an intervention that matter most • synthesis also shows where there are gaps in knowledge G. Jordan AEA 2011

6. CAHS Logic for Return on Investment to Health Research Initiation and Diffusion of HealthResearch Impacts Knowledge Pool Health Industry Research Results Global Research Health care Appropriateness, Access, etc. Prevention and Treatment Improvements in Health and Well-being • Canadian Health Research • Biomedical • Clinical • Health Services • Population and Public Health • Cross-Pillar Research Other Industries Interactions/Feedback Health Status, Function, Well-being, Economic Conditions Government Economic and Social Prosperity Research Agenda Determinants of Health Public Information, Groups Research Capacity Impacts feed back into inputs for future research • Topic Identification • Selection • Inputs • Process Final Outcomes Secondary Outputs Adoption PAYBACK FRAMEWORK Dissemination Primary Outputs G. Jordan AEA 2011

7. CAHS LOGIC in more detail That produces results That contribute to changing health, well -being and economic and social prosperity That affect healthcare, health risk factors, and other health determinants Research activity That influence decision making in… Health Industry -Products/drugs -Services, databases -Practitioners’ behaviour -Clinical/manager’s guidelines -Institutional policies -Social care practices Knowledge Pool Research Results Global Research • Health care • -Appropriateness • -Acceptability • -Accessibility • -Competence • -Continuity • Effectiveness • Safety • Occur through prevention and treatment • For disease, illness, injury, or progressive condition • Prevention • Diagnosis/prognosis • Treatment/palliation • Post-treatment • Canadian Health Research • Biomedical • Clinical • Health services • Population and public health • Cross-pillar research • Other • Industries • Products/services • Built infrastructure • Work environment Consultation/ Collaborations Improvements in health and well-being (disease prevalence and burden) Government (multiple levels) -Resource allocation -Regulation -Policy -Intervention programs -Taxes and subsidies Health status and function, well-being, economic conditions • Research Capacity • Increased understanding • Methodological advances • Larger, more comprehensive data sets • Human capital (absorptive capacity) • Student and faculty career paths • Reputation • Research revenues • Cross-fertilization of ideas/research • Education curriculum Research Decision Making -R&D agendas/investment (industry/gov’t/foundations) -Identify issues, gaps -Evidence problems are being addressed -Tackle harder problems Economic and social prosperity • Determinants of health • Personal behaviour • Social/cultural determinants • Environmental determinants • Living and working conditions External Influences: Interests, Traditions Technical limitations, Political dynamics • Public Information, Groups • - Advocacy groups • Media coverage • General knowledge • Confidence in data G. Jordan AEA 2011

8. Applying the model to different research pillars By cross-mapping impact categories for each pillar, we can create a specialized framework to show where the majority of impacts are likely to fall within that pillar. Intent is to show where most impacts are likely to be found and thus where indicators or metrics for those impacts should be concentrated. However, the areas of impact for any one research pillar are not exclusive of impacts in other domains. G. Jordan AEA 2011

9. A Biomedical Research Initiation and Diffusion of HealthResearch Impacts Knowledge Pool Health Industry Research Results Global Research Healthcare Appropriateness, Access, etc. Prevention and Treatment Improvements in Health and Well-being • Canadian Health Research • Biomedical • Clinical • Health Services • Population and Public Health • Cross-pillar Research Other Industries Interactions/Feedback Health Status, Function, Well-being, Economic Conditions Government Economic and Social Prosperity Research Agenda Determinants of Health Public Information, Groups Research Capacity Impacts feed back into inputs for future research Advancing Knowledge Informing Decision Making Economic Benefits Capacity Building Health Benefits G. Jordan AEA 2011

10. D Population and Public Health Research Initiation and Diffusion of HealthResearch Impacts Knowledge Pool Health Industry Research Results Global Research Healthcare Appropriateness, Access, etc. Prevention and Treatment Improvements in Health and Well-being • Canadian Health Research • Biomedical • Clinical • Health Services • Population and Public Health • Cross-pillar Research Other Industries Interactions/Feedback Health Status, Function, Well-being, Economic Conditions Government Economic and Social Prosperity Research Agenda Determinants of Health Public Information, Groups Research Capacity Impacts feed back into inputs for future research Advancing Knowledge Informing Decision Making Economic Benefits Capacity Building Health Benefits

11. Summary and Conclusion • The logic model was a definite asset for categorizing indicators and ensuring that important indicators were not overlooked. • The framework has been used for impact assessments by at least three organizations, • Alberta Innovates Health Solutions • Catalan Agency of Health Information, Assessment and Quality in Spain • Canadian Cancer Society Research Institute (CCS RI) • Some organizations continue to use the Payback Model • It remains to be seen if it is possible to copy this logic and menu of indicators in other areas of research • The groups to influence are similar • But the common language and indicators for the intermediate outcomes (“magic in the middle”) are less well defined outside of health area. G. Jordan AEA 2011

12. Thank you for your attention. • For a full copy of the assessment report see www.afmc.ca/pdf/ROI_FullReport.pdf • My contact information after December 15, 2011 gretchen.jordan@comcast.net G. Jordan AEA 2011