Evaluation of Impact of e-Infrastructure Investments : a UK view point

1. Evaluation of Impact of e-Infrastructure Investments : a UK view point • Jane Nicholson, • Head of Research Infrastructure and International, • Engineering and Physical Sciences Research Council, UK

2. Over view • What is EPSRC and how do we define impact • Approaches to evaluating the impact of investments in research • Examples of impact evidence • Challenges for measuring impact in e-Infrastructures • Conclusions

3. What is EPSRC • One of the seven UK Research Councils that work together as RCUK • EPSRC is the main UK government agency for funding research and training in engineering and the physical sciences • EPSRC invests around £850m a year in world class research and training to promote future economic development and improved quality of life • EPSRC is key to tackling challenges such as energy security, climate change, our ageing population, crime and economic resilience • EPSRC is generating the fundamental knowledge and skilled people essential to business, government and other research organisations

4. How we see the future Our strategy has 3 clear goals • Delivering impact • Shaping capability • Developing leaders

5. Delivering impact • We will ensure excellent research and talented people deliver maximum impact for the health, prosperity and sustainability of the UK • We will build strong partnerships with organisations that can capitalise on our research and inform our direction • We will promote excellence and impact, and ensure it is visible to all

6. Definitions • What is impact ?

8. The facts and figures WE HAVE SUPPORTED 12,300 HIGHLY SKILLED POSTGRADUATES TO INDUSTRY AND THE PUBLIC SECTOR OVER THE PAST DECADE WE COLLABORATE WITH MORE THAN 2,300 COMPANIES AND MORE THAN 100 PUBLIC BODIES INCLUDING 12 GOVERNMENT DEPARTMENTS OUR £4BN PORTFOLIO HAS LEVERAGED AN ADDITIONAL £700m IN BUSINESS AND COLLABORATOR CONTRIBUTIONS WE SUPPORT AROUND 8,000 ACADEMIC RESEARCHERS SECTORS DEPENDENT ON ENGINEERING AND THE PHYSICAL SCIENCES ACCOUNT FOR 30% OF GDP AND 88% OF MANUFACTURING EXPORTS AROUND 43% OF OUR PORTFOLIO IS MULTIDISCIPLINARY

9. e-Infrastructures

10. EPSRC’S role in support of e Infrastructure • RCUK Managing agent for provision of shared national HPC systems • Ran core programme of RCUK e Science programme • Have previously joint funded with JISC the development of the National Grid Service (UK NGI in EGI) • Fund software development activities in Engineering and Physical Sciences • Fund research programmes of users of e-infrastructures

11. Support for e-Infrastrcuture : Hardware example UK National Supercomputing Service • HECToR – the UK National • HPC service. Based on a 366 TF • Cray XT6 system plus 114TF • Cray XT4 System. • Operated by EPCC, University • of Edinburgh with science support • provided by NAG Ltd. • Funded by EPSRC,NERC and BBSRC

12. Support for e-Infrastructure :Software ExampleSoftware Sustainability Institute • EPSRC funds the The Software Sustainability Institute (SSI). Run by researchers from the University of Edinburgh, Manchester and Southampton , it works in partnership with research communities to identify key software that needs to be sustained. • SSI’s vision is to make software useful for future generations of researchers by improving usability, maintainability and quality. This increases research productivity by enabling more people to use software, delivers added value by leveraging our expertise and connections, and ensures researchers can continue to use their chosen software as a cornerstone of their research. • SSI works with groups to facilitate the self-sustainability of research software within the communities that develop and use it, drawing on a skilled team based at the Universities of Edinburgh, Manchester and Southampton. This team includes talented individuals with a breadth of experience in software development, project and programme management, publicity and dissemination, and community engagement.

13. Approaches to Evaluation of Impact • Bibliometrics • Collection of Statistics • Feedback from Collaborators • Case Studies • Peer review e.g. Theme days, International Reviews

14. Bibliometrics : Supporting excellence • EPSRC sponsored researchers achievea higher citation rate of1.6compared to the UK averageof 1.4 and a world averageof 1.0 Figures from 2009/10 Annual Report

15. Approaches to Evaluation of Impact • Bibliometrics • Collection of Statistics • Impact on Collaborators • Case Studies • Peer review e.g. Theme days, International Reviews

16. Support of next generation skills • Around 12,300 of the highly-skilled postgraduates supportedby EPSRC have moved on into careers in industry and the public sector in the past decade • Career Path studies - comparision of average salaries vs. PhD salaries • First Destination statistics for PhD students – working with learned societies to track career paths • Number of research students who have used facilities – e.g, use of National Grid Service Figures from 2009/10 Annual Report

17. Impact from excellence • Nearly 40% of the research and trainingwe fund is directly involved in collaboration with industry • To date, 132 follow-on projects worth £11.5 million have been supported • Our Partnership with the Technology Strategy Board has led to over 270 research projects being sponsored and a joint portfolio of over £250 million • Our 28 Strategic Partners bring in £77 million of additional funding The prototype magnetometer. Figures from 2009/10 Annual Report

18. Statistics on use of e- Infrastrcuture • HECToR has 1400 UK academic registered users :40% of the users from the Engineering and Physical sciences area have industrial collaborators • Over 800 attendees on HECToR training courses to date , OMII has delivered 111 training events to 2000+ researchers • The Taverna workflow tool supported by the e-Science programme has had 65,000 downloads and has been used by 350 organisations and 23 companies in 35 countries. • The EPSRC funded OMII has supported software deployment on NGS sites plus on Lyceum, Legion,ECDF and Iridis campus infrastructures plus on TeraGrid, NAREGI, D-Grid and EGEE

19. Approaches to Evaluation of Impact • Bibliometrics • Collection of Statistics • Impact on Collaborators • Case Studies • Peer review e.g. Theme days, International Reviews

20. Collaboration Example with regional impact • Funding to the University of Newcastle by the e-Science Programme led to substantial regional impact through collaboration with Arjuna Technologies. Seven PhDs and thirty MScs from Newcastle University have worked for Arjuna, one PhD, many MScs and a Business Development Manager have returned from Arjuna to Newcastle University, and the institutionalised network of knowledge transfer to local companies has led to “cutting-edge technologies:” e.g. web services, grids and clouds and the regional involvement of Red Hat, Amazon and Microsoft. Newcastle University estimates that Arjuna/RedHat have contributed ~£16M GVA to the regional economy.

21. Collaboration example with spin out • Nationally, the Distributed Aircraft Maintenance Environment Project (DAME) partnered with Rolls-Royce, Data Systems and Solutions and Cybula Ltd to use e-Science to reduce engine maintenance times and to improve the interoperation of the maintenance team. The technologies developed are now used on Rolls-Royce Trent engines and the result was a spin-off company: Oxford Bio-Signals (OBS).

22. Approaches to Evaluation of Impact • Bibliometrics • Collection of Statistics • Impact on Collaborators • Case Studies • Peer review e.g. Theme days, International Reviews

23. Case Studies of Impact – Example 1 Two EPSRC funded EngD students from the University of Southampton developed the technology used in the building of Amy William’s gold medal winning skeleton sled Credit: Sarah Winterflood/UK Sport

24. Aircraft pressure relief valves are used to protect the fuel tanks of wide-body civil aircraft from over-pressurization. The relief valve outlet is typically in the shape of a cylindrical hole, or cavity, cut in the underside side of the wing skin. At typical approach speeds of around Mach 0.3, the flow past the cylindrical cavity can become unsteady. This produces an unwanted tonal contribution to the airframe landing noise. A numerical investigation of this phenomenon was performed in consultation with Airbus France, as part of the EU programme AeroTraNet. Dept of Engineering, University of Leicester, UK Case Study Example 2: Reducing Noise Pollution in Aircraft

25. Case Study Example 3 : Improving the modelling of Cardiac Arrythmia • Medical science is increasingly turning to computational models to study the possible effects of drugs and surgical interventions, before moving on to patient trials. One active area of research is in heart modelling. Researchers at the Oxford e-Research Centre (OeRC) and EPCC (University of Edinburgh), will be using HECToR to optimise heart-modelling software. If successful, this work will enable much greater integration of computer simulation with the operating theatre and could, ultimately, lead to personalised medicine.

26. Case study Example 4 : Software development benefits for basic science • NAG HPC experts, working under Research Council Funded NAG’s Computational Science and Engineering (CSE) support service for HECToR, the UK’s national academic supercomputing facility, have optimised a Quantum Monte‐Carlo application for multicore architectures, resulting in a performance increase of a factor of four, potentially saving £760k in computing resources on HECToR for a single one year research project, with several million pounds of savings when applied to future research on HECToR and other supercomputers used to run CASINO.

27. Case Study Example 5 : Software development benefits for climate research • Dr Andrew Coward, who is the manager of the Global Ocean Modelling Consortium on HECToR estimated that their group used around 6M AUs running NEMO last year. Reducing the wall clock time of NEMO by up to 25% will result in a saving in notional cost of AUs by as much as £95,000 per year (up to £400,000 for the remainder of the service), for only six months of person effort. • Other consortia using NEMO on HECToR have used around 40M AUs over the same period. If the code modifications are accepted into the main code base, then these other users could benefit too, leading to the possibility of multi‐million pound savings overall.

28. Approaches to Evaluation of Impact • Bibliometrics • Collection of Statistics • Feedback from Collaborators • Case Studies • Peer review e.g. Theme days, International Reviews

29. Peer Review Approach Example

30. An example : International Review of UK e Science programme • UK e Science Programme funding for projects ran from 2001- 2007 • Total of £250+M investment made by six UK Research Councils. • Review planned for after close of programme to be able to evaluate impact and longevity of actions. Review took place December 2009. • International review approach chosen given breadth and range of research supported plus desire for independent view of progress

31. Approach • International Panel formed of 16 academics and industrialists • Data and evidence of activity collected to produce a comprehensive data summary. • Panel spent a week in UK in December 2009 – interacting with researchers, stakeholders and produced a report of findings and recommendations. • RCUK grateful to Prof. Dan Atkins ( panel chair) and the Panel for the time and energy they put into the review and report preparation.

32. Terms of Reference for e-Science International Review Panel • The role of the review panel was to: • assess the impact of the programme on research areas nationally and internationally, on broader wealth creation and quality of life, and on e-Science itself; • assess and compare the quality of the UK research base in e-Science with the rest of the world via triangulation of data, panel and community perception; • comment on the added value of this programme; and • present findings and recommendations about the strength, weakness and opportunities for the future to the Research community and Councils.

33. Specific question on impact posed to panel ? • What has been the impact (accomplished and potential) of the UK e-Science Programme? • To what extent has the research undertaken through the e-Science Programme benefited the UK economy and our global competitiveness? • To what extent did the research undertaken through the UK e-Science Programme address key technological/societal challenges? • What evidence is there to show that the UK e-Science Programme supported the development of a creative and adventurous research base and portfolio?

34. 2009 International Review of e-Science

35. Findings of review • The Panel identified three ways in which projects had a substantial economic impact : • Direct involvement of industry from the inception of the project • University led linkages with industry during and after development of new technologies • Entreprennurial investigators The panel also assessed science quality, outcomes for human capital and made recommendation for the future.

36. Outcome of Review • Peer review identification of direct impacts : • Programme attracted £20 million in industrial collaboration and £7.1 M cash and in kind industry transfer • Programme resulted in 138 stakeholder collaborations • 30 licenses or patents • 14 spin off companies • 103 key results taken up by industry • Secondary impacts • Improvements in prediction of extratropical storms by numerical weather prediction using storm identification and tracking software led to a reduction of in the flood and wind related damage due to storms • Comment in other areas e.g.healthcare still to early to identify the full effect. • “

37. Challenges for assessing impact of e- infrastructures • Knowing who your “users” are. • Measuring the importance of the contribution of the use of e-infrastructure to the final science output. • Identifying the added value of the infrastructure investment – benefit of 1PT vs. 900TF computer etc. • Capturing the data over long time periods

38. Summary • For effective evaluation of impact need to use a range of approaches to be able to capture the benefit of investment in e infrastructures • For both quantitative and qualitative information need to start capturing this from the start of projects and programmes – trying to find information retrospectively is hard and extremely time consuming • Need to include both quantitative and qualitative metrics to be able to demonstrate the whole range of impacts • Need to collect this information often beyond the life of project so an on going activity • For e-infrastructure can exploit e-tools to capture data if plan for this in advance.

39. And finally • The more you know the user base of the infrastructure the easier identifying evidence of impact be it academic, social or economic will be. • Reports and Information on EPSRC available at http://www.epsrc.ac.uk and examples of case studies at http://www.impactworld.org.uk

40. Thank you for your attention