Tracking the external impacts of academic research in long-term ways and for HEFCE purposes

1. Patrick Dunleavy London School of Economics and Political Science See our ‘Impact of Social Sciences’ blog at:http://blogs.lse.ac.uk/impactofsocialsciences/ Tracking the external impacts of academic researchin long-term ways and for HEFCE purposes

2. Structure of this talk • 1. Defining external research impacts • and exploring how they operate 2. Capturing useful evidence of external impacts (starting now for the longer term) 3.HEFCE’s maximalist view of external “impacts”, and its case study method

3. Defining research impactsin an evidence-based way A research impact is: “a recorded or otherwise auditable occasion of influence from university research upon another actor or organization”. a. Academic impacts from research are influences upon other actors in academia or universities, e.g. as measured by citations. b. External impacts are influences on actors outside higher education, that is, in business, government or civil society, e.g. as measured by references in the trade press or in government documents, or by coverage in mass media.

4. Academic work and external impacts: a simple view Primary impacts – observable occasions of influence Academic/ university outputs Academic impacts e.g. citations A c a d e mi c w o r k External impacts e.g. influence counts Economic/ public policy/ civil society outputs

5. Academic work and external impacts: allowing for time lags and cumulation/ delay effects Primary impacts – observable occasions of influence Academic/ university outputs Academic impacts e.g. citations A c a d e mi c w o r k Dynamic knowledge inventory External impacts e.g. influence counts Economic/ public policy/ civil society outputs

6. A naïve (science-based) view of howan academic discipline achieves external impacts Wider society Single discipline processes Media, cultural and civil society systems Economic and business systems Discovery Public policy systems Application

7. Howkey forms of scholarship within each academic discipline begin to achieve external impacts Impacts interface Wider society Single discipline processes Media, cultural and civil society systems Renewal Integration Economic and business systems Discovery Public policy systems Application

8. How cross-disciplinary influences add to and mediate external impacts Joined-up scholarship Impacts interface Wider society Single discipline processes Bridging Media, cultural and civil society systems Renewal University - local integration Integration Economic and business systems Discovery Public policy systems Application Academic Service

9. How the impacts interface shapes external impacts Joined-up scholarship Impacts interface Wider society Single discipline processes Bridging Media Media, cultural and civil society systems Specialist Media Renewal University - local integration Professions Integration Economic and business systems Corporations Entrepreneurs Discovery Consultants Public policy systems Think tanks Application Academic Service Policy communities NGOs

10. 2. Capturing useful evidence of external impacts (starting now for the longer term)

11. Six metrics of academic/university involvements with external impact • External funding linkages, especially business-to-sciences/ technology departments • Media and specialist media presence • Government website presence • Academic service in government • Reported ‘knowledge transfer’ activities • Academics’ perceptions of impacts

12. Universities’ key interactions with business in the UK

13. Proportion of research found in UK national press from different disciplines (2008)

14. Proportion of research found on UK government websites from different disciplines

15. Academic Service: University staff involved in UK central government quasi-government agencies Source: Griffiths, 2010

16. Academics’ reporting of ‘knowledge transfer’ activities - top 11 activities

17. Academics’ reporting of ‘knowledge transfer’ activities – next 10 activities

18. Area of external impacts Level of impact Social scientists only Academics spanning social sciences and humanities All respondents (including academics across all social science and humanities) Public policy Actual 4.6 3.6 3.4 Potential 6.0 5.5 5.1 Civil society Actual 4.5 4.1 4.1 Potential 5.6 5.3 5.1 Public debates and culture Actual 3.9 5.0 4.6 Potential 5.4 5.9 5.6 Economy and business Actual 3.6 3.1 3.0 Potential 4.7 4.0 3.8 Science and technology Actual 3.1 3.4 2.9 Potential 4.5 4.1 3.9 Social science and humanities academics’ perceptions of external impacts (2008)

19. Eight longer-term steps towards universities boosting their external impacts • Clean up ‘multiple branding’ problems • Reappraise events programmes • Develop an impacts file for all individual academics, and then for departments • Pull together databases and services to give improved ‘customer relationship management’ of research clients – e.g. bring in CRM systems • Start (and fill up) an online depository • For all research on the closed-web, publish a useful open-web version (see session D) • Improve professional communication – starting with multi-author blogs (see session D) • Work better in networks (see Panel 2)

20. 3. HEFCE’s expansive view of external impacts, and case study method

21. PRIMARY EXTERNAL IMPACTS OF RESEARCH Primary impacts – observable occasions of influence Academic/ university outputs Academic impacts e.g. citations A c a d e mi c w o r k Dynamic knowledge inventory External impacts e.g. influence counts Economic/ public policy/ civil society outputs

22. WHAT MORE DOES HEFCE NEED? – 1. EVIDENCE OF CHANGED OUTPUTS DEMONSTRATED CETERIS PARIBUS, CONTROLLING FOR ALL OTHER INFLUENCES Primary impacts – observable occasions of influence Changes in organizational activities or outputs Academic impacts e.g. citations A c a d e mi c w o r k Dynamic knowledge inventory Multiple other factors External impacts e.g. influence counts Economic/ public policy/ civil society outputs

23. WHAT MORE DOES HEFCE NEED? – 2. EVIDENCE OF CHANGED OUTCOMES AGAIN CETERIS PARIBUS, CONTROLLING FOR ALL OTHER INFLUENCES Primary impacts – observable occasions of influence Changes in organizational activities or outputs Changes in societal outcomes Academic impacts e.g. citations A c a d e mi c w o r k Dynamic knowledge inventory Multiple other factors External impacts e.g. influence counts Economic/ public policy/ civil society outputs Economic/ public policy/ civil society outputs

24. WHAT MORE DOES HEFCE NEED? – 3. PROOF OF POSITIVE SOCIAL BENEFITS Primary impacts – observable occasions of influence Changes in organizational activities or outputs Changes in societal outcomes Evaluations of outcome changes – against societal values Academic impacts e.g. citations A c a d e mi c w o r k Dynamic knowledge inventory Multiple other factors External impacts e.g. influence counts Economic/ public policy/ civil society outputs Economic/ public policy/ civil society outputs + ? -

25. HEFCE’S SCORING BANDS FOR IMPACT CASE STUDIES’ ‘SIGNIFICANCE’/ VALUE AND ‘REACH’/RELEVANCE

26. CHARTING ALL HEFCE’S CRITERIA high medium Significance (or value) low low medium high Reach (or relevance)

27. CHARTING ALL HEFCE’S CRITERIA 4* high exceptional 3* excellent medium Significance (or value) low low medium high Reach (or relevance)

28. CHARTING ALL HEFCE’S CRITERIA 4* high exceptional 3* excellent medium Significance (or value) 2* very good 1* low good low medium high Reach (or relevance)

29. CHARTING ALL HEFCE’S CRITERIA Additional criteria: ▪Impact occurring in the last 15 years ▪ Quality of Underlying Research ▪Distinctiveness of Research 4* high exceptional 3* excellent medium Significance (or value) 2* very good 1* low good U low medium high Reach (or relevance) unclassed

30. Short-run steps to prepare for HEFCE case studies Develop a large long list of possible case studies Collate grants, publications, contacts and testimonials evidence from (senior) academics on all potential cases, especially for 2005-11 For ‘probable’ cases - firm up outside evidence of outputs, activity and outcome changes; - push back pre-history to 1997 if needed Enlarge single-person cases wherever feasible Strengthen ‘sub-group’ case studies, by adding members (if feasible, without diluting ‘quality’), and re-branding for greater coherence by 2013 Monitor potential cases closely from now to 2013, and discuss with uninvolved senior case-writers

31. Thank you for listening See our ‘Impact of Social Sciences’ blog at: http://blogs.lse.ac.uk/impactofsocialsciences/ From there, download the free Handbook: Maximizing the Impacts of Social Science Research