Evaluating impact

1. Evaluating impact Sharon Markless King’s College London & Information Management Associates

2. SLS Issues (from conversations) • Value of such core services as book loans needs to be seen (ROI?) • SLS appreciate the not quantify everything, but what to do without quants? Other options? • What is robust data (done surveys but low return rate)? Is there a robust way forward? Evaluation should be internal and external facing; level of robustness depends on audience, purpose etc • What sort of causal relationship can you talk about? Making ToC explicit.

3. What is impact? • Current thinking about impact evaluation in complexity • Stages in evaluating impact (systematic and robust) • Where next for SLS?

4. Little impact evaluation in library services: How have we got here? • Seen as at best difficult, at worst impossible to measure- e.g. perceptions, attitudes, complex interrelated behaviours, skill levels... • Changes in individuals often not manifest after a single activity or in the short-term; tend to emerge over time. Librarians rarely have resources to track over time. • Collecting ‘good enough’ data • Problem of cause and effect: complexity

5. 1 Starting point: what is impact? Any effect of your service/activities on individuals, groups, institutions, community. Impact is about change; making a difference Impact may be:positive or negative; intended or accidental

6. Reach of Impact? Impact on what? Changes in: • behaviour (doing things differently e.g. involve SLS early in project planning; read more widely) • competence (extend/develop skills) • knowledge • attitudes (e.g. more confident readers) • quality of life (more choices; save time; empowerment) Impact on whom:School librarians; teachers; students (specific groups; all students); parents; policy makers, etc

7. Levels of impact? Change is on a continuum:long/short-term; skills/life-changing; individual/collective; whole community/specific sub-group (eg ESL students) How deep seated/far reaching? Teachers know more about…; more headteachers engage in advocacy for SLS; students more able to ….; school feels more able to meet the needs of….; x department enabled to achieve y learning outcomes more effectively

8. Impact: a slippery concept Pin down the differences that you want and are able to make to individuals, to groups, to the institution... You need to be clear what you understand by impact, the different levels on which it operates, before you can begin to evaluate it.

9. 2 The changing world of impact evaluation: from attribution to contribution and beyond • Traditional focus on direct attribution (using experimental design/RCTs; in UK large surveys. All producing quants.). • Complexity is now the norm and complex situations and initiatives require flexible, agile evaluation approaches to deal with multiple factors, relationships and layers. • Now (especially in Europe) seek to gauge the impact of complex change programmes, and new services in complex settings using rigorous theory-based evaluations.

10. Theory-based evaluations… • make programme assumptions and implementation issues transparent (how and why things will work to achieve changes). • enable evaluators to recognise different levels and types of contributions to change • illuminate how things affect different groups/people, in what circumstances, through different mechanisms Pave the way towards credible causal claims, not relying on direct attribution.

11. A Common Factor: the Theory of Change Various theory-based approaches have been developed but one common factor is that these all start with articulation of a Theory of Change: Rigorous and systematic early articulation of the ways in which the programme/service expects to bring about clearly identified changes in individuals/communities at a range of different levels (programme theory). Makes options and priorities clear as well as nature of expected changes.

12. What makes a good Theory of Change? • Plausible - do evidence and common sense suggest that the activities will lead to the desired outcomes? Based on systematic use of relevant research literature from appropriate disciplines alongside stakeholder input. • Doable – are the resources available to carry out the initiative? • Testable – is the ToC specific and complete enough to track its progress in credible ways? “A good ToC is embedded in the context of the intervention and is developed incorporating the perspectives of key stakeholders, beneficiaries and the existing relevant research.” Mayne (2012)

13. Implications for the LIS field • Theory-based evaluation approaches offer an important change in focus for the LIS field, in which attribution is hard to prove. • Adopting these will make it easier to be seen as credible; better able to argue against making simplistic and unconvincing attribution claims for their services. • Using a ToC focuses data collection activities in a rigorous way; increasingly using mixed methods (Greene, J, 2008) • However, more credible LIS evaluation requires a significant and sustained investment of time and resources.

14. 3 Stages in impact evaluation BUILD a ToC: Focus – your goals/areas of impact leading to impact objectives Indicators – what will tell you that you are making a difference/change occurred? Key processes and assumptions Evidence –is change occurring? How? Mainly qualitative; but also appropriate quants (aggregated)

15. 2 key issues to accept at the outset • NOT about attribution; making credible causal claims of your CONTRIBUTION as part of a mix of factors (causal packages). Identifying how your intervention leads to consequences e.g. trigger? Stimulus? Sustain change? • Perceived/reported impact v ‘actual’ impact (surrogates). Balance needed.

16. Step 1: Clarifying your focus - areas of impact and impact objectives • The ‘reach’ of the service - be realistic about areas in which SLS can make a difference/ significant contribution. Use research evidence/what has been achieved elsewhere? • What do your stakeholders want/need? • Your professional judgement Don’t choose too many areas of impact!

17. Possible Areas of impact • Inclusion /empowerment /engagement • Student experience • Teacher experience (choice; time etc) • School library operational capacity Etc etc

18. From impact areas to impact objectives If you are effective what will be happening in each key area? • Choose objectives to reflect what is really important to you and align them with wider priorities (LEA; Govt). • Focus on what is achievable taking account of your resources (manageable/realistic). • Carefully worded (‘support’ and ‘provide’ will draw you down the process road!) . You cannot begin to think about evaluating impact until you have a set of clear impact objectives

19. Impact objectives: KLI (exert indirect influence) What will be different if we are successful? 1 Enhanced quality of the student learning experience and the learning environment 2 Increased recognition and status of teaching across College 3 Enhanced sector wide higher education scholarship. 4 Staff have re-examined and/or re-conceptualised teaching and learning and their educational role OR staff intentions, values and beliefs about teaching and learning are reinvigorated and/or changed (Hearts and minds objective - still under review)

20. Step Two: Impact Indicators Indicator – a statement around which you can collect evidence on a regular basis to show a trend • State the specific changes in attitude, behaviour, materials produced etc. that will tell you if you are making progress towards your impact objectives (what you are trying to achieve) • Directly linked to one particular impact objective but may give information about more than one

21. Forms of impact indicators • The proportion/percentage of teachers who consult SLS as a matter of course when writing new programmes of study (quants) • Teachers consult SLS as a matter of course when writing new programmes of study (statement) • Do teachers consult SLS as a matter of course when writing new programmes of study?(question)

22. KLI indicators for objective on enhanced quality of student learning experience etc • Students report that the assessment and feedback they receive is useful to their learning • Students report they are engaged in stimulating and challenging learning (TEF indicator) • Staff new to teaching report that they are secure and confident in their teaching role and activities • Staff implement academic student engagement activities/strategies • Course design is effective in stretching students to develop knowledge, skill and attributes that prepare them for future study and employment (TEF indicator)

23. Step 3 Completing the ToC identify key mechanisms and assumptions Processes: Identify the ways in which the programme/ service expects to bring about the changes (individuals/ groups/systems etc. stated in the impact objectives), and the assumptions being made (e.g. context; support mechanisms) Process indicators to monitor processes: see whether, how and in what context processes occurring

24. Step 4: What counts as evidence? This depends on: Why you are collecting it Who you will give it to How you will use it Using a ToC enables a systematic and coherent approach: why data is needed, what it relates to and what to do with it!

25. Evidence of Impact? Recognise that much impact evidence is qualitative, collected using social science research methods • Ask • observe • Infer BUT ALSO need to count (mixed methods)

26. What counts as impact evidence? How to collect it? • Statistics (relationships between quantitative measures that highlight change over time) • Case studies of the effects of implementation or innovation • Representative stories/ most significant change (catching the experience and authentic voices of people who may experience change) • Systematic observation data

27. What counts as impact evidence? • Self-assessments of change by those involved (e.g. through validated tools) • Aggregated critical incident analysis (based on specific encounters between people and the services provided) • Document analysis to infer changes in behaviour..

28. Some issues in gathering impact evidence • What standard of evidence do you need? • Short-, medium- or long-term? • Baseline data • Emergent impacts • Capturing evidence in busy libraries