The importance of micro-data for the assessment of European research performance

1. European Parliament STOA Science and Technology Options Assessment Policy needs and opportunities Brussels, March 26 2013 The importance of micro-data for the assessment of European research performance EmanuelaReale CERIS CNR e.reale@ceris.cnr.it

2. Content • Importance of micro data • Definition of micro data • Opportunities and constraints • Recently developed tools: EUMIDA and JOREP • Conclusions

3. Changing context and requests for STI data and indicators • Differentiation of the demand side • Government (Ministries, Parliament, local government) • Intermediaries (Funding agencies, QA and evaluation agencies) • Research organizations (universities and PROs, companies) • Stakeholders • Differentiation on the supply side • Statistical offices • Indicator producers • Specialized institutes • Data should allow tracing the effects of the policies • on the beneficiaries and other agents • on the context, to assess the R&D performance of programmes and actors

4. New demands for new data and indicators • Demand for STI indicators become more differentiated and complex • Rise of evaluation • Importance of the evidence-based policy making (Sanderson, 2001) • Overcoming the input-output framework linked to the linear model of innovation • Emerging of new approaches (e.g. position indicators) for deepening the processes that allow getting a certain performance or result • Demand for micro data emerges related to the need of investigating: • Efficiency, effectiveness and impact of different policy options, • The performance of actors and systems in Europe, • The choice of the funding instruments, • The governance and steering approaches, • Framing conditions (enablers) for excellence and innovation, • Determinants and constraints of economic growth

5. What are micro data for • The “we cannot manage what we cannot measure” paradigm • We need to measure performance (Wagener, 2008): • Effectiveness: doing right things (set the right targets to achieve an overall goal) • Efficiency: doing things in the most economical way • Impact: the effect produced at a certain time and its social and economic value • Sustainability: meeting the needs of the present without compromising the ability of future generations to meet their own needs • Systematic and organic linkages between policy and indicators at European level under a comparative perspective • New policy requirements and changing organizational contexts

6. Data for STI investigations • Aggregated data • Based on large amount of input and output indicators • No distinguishing individual contributors and beneficiaries • Systematic accounting methodologies developed by international organizations in order to represent financial, human resources and output • Case studies • Qualitative investigations • Deepening effectiveness, adequacy or equity of a programme, impact on performers, social dynamics, processes • Administrative data • Information collected mainly for the management of administrative procedures. They have a significant potential as research resource, especially when linked with other sources (Jones and Elias, 2006)

7. Micro data • One can distinguish between (Everitt, 2006) • Census data: aimed to observe every member of a population • Survey: collect planned information for a sample of respondents and estimate characteristics, performance and effects • Data collected for statistical purposes are generally based on surveys • Survey of micro data devoted to investigate performance cannot be confused with standardized surveys for aggregated data (e.g. Frascati Manual) • Advantages • Possibility to look at range, variance, skewedness, coefficients of variation • Aggregate statistics often do not allow answering policy questions • Limitations and problems • Design and size of the sample • Questions and definitions • Non-responses • Attrition problems in longitudinal surveys • Problems of large countries

8. Micro data problems • Availability – are data available at national level? Who own the data (government, statistical offices, research performers, intermediaries)? • Comparability – how far are data comparable? What rules for data collection should be designed? What is the perimeter we can investigate using micro data? What is the underlying conceptual framework? • Confidentiality– How far confidentiality impedes/limits the data collection and dissemination? • Updating - Data become old: how can we update data collection? When data collected become a matter for NSI? • Maintenance and management (infrastructure) –Who is in charge for management of data and how the access to data should be regulated?

9. Two examples of micro data • Different layers in the European research policy system: performers, policy layers, research funding • We look at the mentioned problems using 2 key experiences of micro-data collection, on the performer and on the research funding • EUMIDA – Census data on Higher Education Institutions • It is the first attempt toward the foundations of a regular data collection (Register) by national statistical institutes on individual higher education institutions in the EU-27 Member States together with Norway and Switzerland • Large feasibility study on 27 European countries under the auspices of EC • JOREP – Database on characteristics of Joint and Open Research Programmes • Providing a quantitative basis for the monitoring of investments in joint and open research programmes in EU countries, as well as empirical evidence of the policy rationales and impacts of these programmes on the European Research Area. • Experimental data collection and impact analysis on 11 European countries

10. Availability • EUMIDA • Core set of micro data largely available • Research active HEIs data more problematic • Reasons of non-availability (FR, p. 141) • Legal issues (e.g. the statistical law explicitly forbids the publication of micro data) • Administrative barriers (e.g. it would be possible to publish micro data but NSI depend on administrative decisions from the Ministry) • Institutional settings (e.g. the publication of micro data requires additional workload for which the institution is not prepared, or it would be difficult to allocate the responsibility internally –no official sources to obtain output information on HEIs). • JOREP • Descriptors of joint programmes largely available • Openness of national programmes very difficult to understand • Funding data generally available but with some exceptions • Beneficiary data the most problematic issue • Reasons for non-availability • Funding agencies do not hold data based on the Frascati Manual performer sectors • Funding agencies record data on the basis of project decisions (trade off between funding volume referred to the commitment year and funding decision refereed to the implementation years) • Loan repayments from private performers

11. Comparability • For both the programmes comparability was the most challenging issue • Building a conceptual framework of what census and survey shall include (it means going inside the structure of HE systems and project funding systems of European countries) • Decide the right level of analysis (HEIs vs Departments, Research Programmes vs Research Projects) • Define the perimeter of the data collection • Choice common definitions, modes of data collection, calculations, currency, estimations, etc. • Disaggregation of data not always possible (e.g. beneficiaries in JOREP. Current and capital expenditures in EUMIDA) • Missing data • Comparability often implies a work of social construction

12. Confidentiality • Problem with the data disclosure of research performers: • Agreement of the Authority in charge of education and research • Agreement of the HEIs that data as regards the expenditures (R&D included) and income can be released • No problems (or very limited ones for the disclosure of data on project funding -organizations’ name not included)

13. Updating • Both data collections relate on country correspondents (experts) for the coordination of the contact with the relevant actors • Actors and roles in the data collection • Ministry of Education and Research • National Statistical Institutes • Funding Agencies (research performers acting as Agencies) • Other official sources (Web sites, Reports) • HEIs for descriptors • Need to enlarge the coverage • Other EU and non-EU countries • Other project funding schemes • Problems linked to the changes of the national HE systems and PF systems, which can require adaptation of the methodology • Important updating on a regular basis

14. Data infrastructure • Micro data collection often requires activities that go beyond the NSI functionalities • The definition of a suitable organizational form and of procedures for regular data collection is important • It must takes care of two characteristics: • The inclusion of both expert-based descriptive information and statistical data on funding flows; • Data are owned by different subjects and at different institutional levels • European agencies • European Commission, • National States • National funding agencies

15. Proposed organization for Joint programme data collection

16. Conclusions • Micro data are essential tools for the assessment of European performance • Overcoming the difficulties linked to heterogeneous populations and small numbers • Large access for different type of users • Supporting further investigation of the STI systems answering relevant policy questions • Several problems affect the possibility to collect and to maintain databases of micro data: data need investment • Availability, Comparability, Confidentiality, Updating, Infrastructure • The RISIS initiative under the EUFP framework toward the building of research infrastructures for STI studies shall supply an important step forward micro-data exploration and exploitation