Assessing the impact of research and KTE activities

1. Assessing the impact of research and KTE activities Sandra Nutley

2. Agenda for this session • Key approaches to assessing the use and impact of research and knowledge mobilisation activities • Common challenges and methodological issues • Summary of findings from existing assessment activities

3. Why assess research impact? • Addressing accountability • Assuring value for money • Setting priorities • Assisting learning • Improving outcomes Summative or formative purposes?

4. Forward tracking Evaluation of initiatives Tracking back 1. Key approaches to assessing research impact Impact of research Single studies Research programmes Systematic reviews Policymakers Target KTE interventions Practitioners Media Centres promoting research use Organisational research use Research use in ‘user’ communities Impact of initiatives to increase research use

5. Forward tracking from research projects and programmes to use and impact Two common methods: • Describe and quantify impacts of research - E.g. Payback model 5 categories of possible impact: • Knowledge production • Research capacity building • Policy or product development • Sector benefits • Wider societal benefits • Describe and map networks and flows of knowledge and the effects of any interactions

6. Tracking back from decisions or behaviours to research influences on these • Bibliometric studies of policy documents and practice guidelines • User panels – surveys and workshops • Ethnographic studies of communities of practice • Economic impact evaluations – estimate economic impact of a policy and estimate extent of research influence on that policy

7. What methods are used in practice? Systematic review by Boaz et al 2008

8. Evaluations of initiatives to increase research impact • Standard programme evaluation strategies: • Qualitative investigation of processes and perceived effects • Action research • Experimental and quasi-experimental methods • Importance of strong theories of change & models of KTE to design and test interventions • Need shared taxonomy of KTE strategies and interventions

9. KTE models Ward et al 2009: • 63 different theories or models of KTE • Distilled down to 5 components of the KTE process • Connection between them is seen as interactive and multi-directional • Aim of framework is to provide a foundation for gathering evidence about knowledge into action processes

10. Attributes of knowledge (relative advantage, complexity and compatibility) How it is synthesised and adapted Organisational, individual, environmental and structural Knowledge/Research Recognition of a need for action Problem Interventions Utilisation Distribution-type interventions Linkage-type interventions Process of using intervention: cycles of selection, tailoring, implementation & evaluation Conceptual, instrumental, political, procedural Context Barriers/Supports

11. Taxonomies of KTE strategies E.g. Mitton et al 2007 – 8 key strategies • Face-to face exchange between researchers and ‘users’ • Education sessions for users • Networks and communities of practice • Facilitated meetings between researchers and users • Interactive, multidisciplinary workshops • Capacity building in user organisations • Web-based information, electronic communications • Steering committees for research projects/programmes

12. RURU taxonomy • Intervention types and underlying mechanisms, with focus on the latter • Five key mechanisms: • Dissemination • Interaction • Social influence • Facilitation • Incentives and reinforcement

13. 2. Common challenges and methodological issues • What types of use/impact are of interest? • When to assess impact? • Importance of context – assessing actual or potential impacts? • Dealing with attribution and additionality – constructing a convincing impact narrative • Getting away from linear models research use/impact

14. What uses/impacts are of interest? • Instrumental, conceptual or symbolic research use? • Reach/awareness, impact on behaviour & service delivery, outcomes for service users? • Positive and dysfunctional consequences?

15. When to assess impact? Impact often occurs far down the line, but impact trails grow fainter over time

16. Tools for analysing context • Accounting for the difference between actual and potential impacts • E.g. Conducive policy environments - research more likely to be used when information, ideology and interests coincide and institutions can support developments (Weiss)

17. Constructing a convincing impact narrative • Dealing with attribution – the potential of contribution analysis

18. Slide source: Wimbush 2010

19. Slide source: Wimbush 2010

20. Slide source: Wimbush 2010

21. Conclusions on methods • Case study methods to take account of differing types of research and contexts for impact • Combination of quantitative and qualitative methods and indicators within case study approach • Need for research impact theory/model to guide data collection and analysis • Limited benefits from studying the impacts of individual research projects • Dangers of generalising from case sampling – because distribution of research impact is likely to be uneven

22. 3. Summary of findings from the UK ESRC’s impact assessment activities • The most important drivers of impact are: • Established relationships and networks with user communities • Involving users at all stages with research • Well-planned user-engagement and KTE strategies • Portfolios of research activity that build reputations with research users • Good infrastructure and management support • The involvement of intermediaries and knowledge brokers as translators, amplifiers, network providers

23. RURU’s conclusions on generic features of effective KTE practices • Research must be translated - adaptation of findings to specific policy and practice contexts • Enthusiasm- of key individuals - personal contact is most effective • Contextual analysis - understanding and targeting specific barriers to, and enablers of, change • Credibility - strong evidence from trusted source, inc. endorsement from opinion leaders • Leadership - within research impact settings • Support - ongoing financial, technical & emotional support • Integration - of new activities with existing systems and activities

24. Implications for future research on research use and impact • Much that we still need to know • We need to move away from: • Poor documentation and under-evaluated KTE activities • Studies that focus only on the instrumental use of research • An assumption that research is used and applied mainly by individual practitioners • Studies that result only in a now familiar listing of barriers and enablers, especially where these are the barriers/enablers experienced by individual practitioners

25. Key messages • Need to understand more about how research-based knowledge enters and flows through diverse policy, organisational and practice settings • Need realistic assumptions about the nature and processes of research use and impact– these are many and complex • No single model of research use is likely to be sufficient for all situations when impact is to be assessed • Need to make choices about where and how to look for use and impact based on the purpose of the assessment

26. References Main reference • Davies HTO and Nutley SM (2008) ‘Learning More about How Research-Based Knowledge Gets Used Guidance in the Development of New Empirical Research’, Working Paper for the WT Grant Foundation, New York Other useful references • Boaz et al (2008) ‘Assessing the impact of research on policy: A review of the literature’, Kings College London/ PSI • ESRC (2009) Taking Stock: A summary of ESRC’s work to evaluate the impact of research on policy and practice, http://www.esrc.ac.uk/_images/Taking%20Stock_tcm8-4545.pdf • ESRC (2010) Branching out: New direction in impact evaluation from the ESRC’s Evaluation Committee http://www.esrc.ac.uk/_images/Branching%20Out_tcm8-14881.pdf • Meagher L, Lyall C and Nutley S (2008) ‘Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research’ Research Evaluation 17(3): 163-173 • Ward et al (2009) ‘Developing a framework for transferring knowledge into action: a thematic analysis of the literature’ Journal of Health Services Research and Policy 14(3): 156-164

27. For further information • The following three slides summarise the main research questions identified by Davies and Nutley (2008) when they were asked to define the emerging research agenda on research use and impact for the WT Grant Foundation. These questions are categorised under the three headings of: • Knowledge source, presentation and integration • Context and connections • Strategies and process

28. Knowledge source, presentation and integration • What models of research supply and synthesis might better support knowledge integration by potential users? • How do different kinds of messaging and messengers affect the use of research knowledge? • What is the role of the web in providing access to existing research? • To what extent are policymakers and service managers conducting their own in-house research using administrative and local data? • How is new knowledge integrated into current ways of thinking and models of practice?

29. Context and connections • What communities are involved in the field of interest? How are these connected? • Are policy and practice communities networked and does research-based knowledge flow across these networks? • How, where and under what circumstances do practitioners source new knowledge? • What models of research brokerage or intermediary activities have the best potential for fostering research use? • How can lay people, service users, and others contribute more fully to evidence-informed discussions? • How are education and CPD connected to and supportive of knowledge accumulation and integration?

30. Strategies and processes • Do different models of research use co-exist in different practice settings? • What knowledge management strategies are used across the sector? • What models of push, pull and linkage-exchange are in place? How are these evolving and connecting? • What strategies for increasing research use and impact have the best evidence in support of them, and how might they be tested to consolidate our knowledge base in this area? • How can we get new knowledge on the research-action process to influence the future actions of researchers, funders, intermediaries, policy/decision-makers, and practitioners/end users?