I - Evidence-based Policy Making

1. Jean Monnet Chair I - Evidence-based Policy Making I.1 The role of statistics in supporting the decision making Luigi Biggeri 2017-2018

2. Statistical Reasoning for decision Steps of General Reasoning Steps of Statistical Reasoning • Problem to be solved Problem to be solved • Information needsDefinition of data need • Collection of information Collection of data (Surveys) • Analysis of information Statistical Methods for Analysis • Decision (choice of the action) Decision (support choices) Some points to be remembered

3. What to know about Statistics • Statistical Information • 1 methods of data production • 2sources of statistical data • 3 interpretation and comparison • Statistical Methods • 1 knowledge of the methods • 2 capability to implement Statistical Analysis • 3 capability to interpret and present the results of the analysis

4. What to know about statistical information -a- • 1 Data Production • General characteristics of the phenomenon (population) to measure • Crucial importance of the definitions (units of population and variables) to: • - conduct measures and estimations • - understand the content of the information • What is the relevant population? • What is(are) the variable(s) of interest? • Methods of producing data • - Complete or sample surveys Populations of units • - Data integration When it is not possible to have data on the population with only one survey or to obtain measure for a complex phenomenon

5. What to know about statistical information -b- • 2 Knowledge of data sources and their characteristics • EU and Contry levelEuropean Statistical System (ESS) for national accounts and system of surveys (Lectures of Daniela Ghio) • Local level Usually very few data is available and may be it is necessaryto use SAE to produce them (Monica Pratesi et al.) • 3 Interpretation and comparisons • InterpretationDefinitions used and Quality of data • ComparisonsNever plain: need for standardization

6. What to know about statistical methods • 1 Capability to implement Statistical Analysis • Knowledge of statistical methods • Capability to use the adequate methods for different aims • Knowledge of the intrinsic logic of the methods • 2 Capability to interpret and present the results of the analysis • What the data and method used allow • Interpretation of the results obtained for the specific analysis • Methods of presentation of the statistical analyses and results (tables and graphs, ect.)

7. Few References • McClave J.T., Benson P.G. and Terry T.T., (2014), Statistics for Business and Economics, Pearson (Chapter 1, What is Statistics?) • Biggeri L, Bini M., Coli A., Grassini L and Maltagliati M., (2012), Statistica per le DecisioniAziendali, Pearson (Chapter 1), in Italian Language

8. Jean Monnet Chair I - Evidence-based Policy Making I.2 The role of statistics to design policy intervention a framework Luigi Biggeri 2017-2018

9. Outline Introduction: importance of statistical information Policy definition and evaluation: a statistical theoretical approach Need for a pertinent Statistical Information System Statistical support for the implementation of policy interventions at a local level Different methods to get information Need for data of high quality: how to evaluate it

10. 1 Introduction: importance of statistical information • Nobody has ever doubted that statistical information and statistical methods are indispensable for taking rational decisions • However, for a long time even policy makers have rarely and badly used the statistical information available and, at the same time, producers of official statistics have not always supplied the necessary statistical data • Now, governments at different levels are more and more interested in formalizing their decision processes and in evaluating their programmes, activities and intervention policies in economic and social areas

11. … and of a statistical information system • Obviously, this implies adequate information and, above all, specific statistical information systemsand sets of indicators that official statistics should implement, for the different policies that the decision makers wish to implement • Therefore, the definition of a statistical theoretical approach for policy definition and evaluation is really important • This is important not only in the interest of the public decision-makers but also of the citizens, so that they can exercise a documented democratic control on the policy intervention

12. 2 Policy definition and evaluation: a framework The design of political intervention is clearly a strategic field at least for three reasons • SCIENTIFIC SUPPORT FOR DECISIONAL PROCESSES • Make more adequate choices • Rationalize interventions, programs and actions • On effective implementation of the programmed targets • On impact of the implemented programs • CONTROL (Monitoring) • transparency with references to users • safeguards of society’s interests • GUARANTEE FUNCTION • To specify how policy intervention design and the ensuing impact evaluation should be organized, we can refer to the simplified framework illustrated in the following sketch

13. SIMPLIFIED FRAMEWORK OF THE DESIGN, IMPLEMENTATION AND EVALUATION OF AN INTERVENTION POLICY FRAMEWORK (Contest, etc.) PROBLEM Model/s Design of Intervention Information System simulation of actions and evaluation Simulation of actions L E A R N I N G L E A R N I N G Possible Consequences Evaluation Choice among the alternative actions Implementation (resources and input characteristics) Evaluation Achievments(process characteristics) Evaluation Results (Output, Outcome, Impact) Evaluation Information on results Social Control

14. Framework analysis: characteristics • The sketch shows which are the steps to be implemented to define, carry out and evaluate intervention policies in a consistent framework • Framework analysis highlights some important and specific aspects that must be taken into consideration to organize policy design and evaluation. As a matter of fact, it is necessary: • To analyze the context and real situation (a good knowledge is required of how the phenomena works and how the involved units behave), and the problems that we have to face • To simulate the actions on which intervention is based with macro or micro models, to evaluate their possible consequences, and to choose among the various alternative actions

15. Framework analysis: characteristics • To evaluateeach phase of implementation of the actions and the obtained results • To use results and evaluation analyses for learning aims and, if necessary, to change the plan or to improve the information system already available • To disseminate evaluation results also as a means of social control by general public and by interested bodies • Need for a specific Statistical Information System (For European Statistical System see Daniela Ghio)

16. Framework analysis: programs and effects • To develop these statistical designs it is obviously essential to deeply know the nature and characteristics of the programme • For example, it is necessary to know which are the elements that influence programme results • Obviously, the effects of the programme must be measured using responsevariablesstrictly connected to the objectives (Which poverty indicators is important to use?) • The analysis must be done considering the real operative conditions and consequently the characteristics of the decisional process (as highlighted in pict. 1).

17. Framework analysis: situation and theories • Strategic: • to define a real reference framework of the situation in which is the subject of the study and/or a working model • To know the existing economical and sociological theories • If these theories do not exist or are not convincing, it is necessary to use empirical evidence, through a sociologic, economic, managerial study of the organizations and the processes involved in the programme (in this case, the importance of the interdisciplinary cooperation is evident)

18. 3 Need for a Statistical Information System –a- KEY CONDITIONS FOR THE APPLICATION OF THE SCHEME • adequate quantitative and qualitative statistical information • appropriate statistical information system for: • definition of intervention • management of intervention • evaluation of intervention • planning of quantitative and qualitative key indicators (on poverty, etc.) • analysis of characteristics of variables and indicators INF. METHODS: Impact evaluation MEASUREMENT OF NET BENEFIT OF THE INTERVENTION

19. 3 Need for a Statistical Information System –b- • To plan and evaluate the policy intervention in an objective way, it is necessary : • first “to measure”, having decided what to measure • then to have adequate measures of the variablesof interest and plan the appropriate quantitative and qualitative indicators • The use of different indicators is also an opportunity to check the quality and consistency of different data on the same phenomenon as well as data on different phenomena arising from different statistical sources • In the end, the analysis of data highlights the most important results obtained from different statistical surveys (thus increasing the value added in the presentation of the results of each single survey) and provides possible solutions to problems of economic and social policy. • Numbers produced by surveys are transformed into • “political relevant information”

20. 4 Statistical supportfor the implementation of policy interventions at a local level Figure 1 The multi-step procedure

21. Statistical approach for selecting and analyzing Indicators Statistical analysis helps the policy maker to identify issues, using various methods indicated in the sketch to give reply to various questions: How is the situationof the area? Where are the main problems? Why did the situation and problem arisen? in order to identify the most important aspects that need to be monitored and considered to design intervention policies But also to reply to other questions: Is the situation of the area improving or worsening over time? Is there a specific trend over the last few years? Do the sub-areas have the same characteristics and behaviour? in order to understand where intervention is primarily required

22. The implementation of a Statistical Information System at a local level: possible general effects • Answerto the new demand of information for programming and evaluating interventions • Enhancement of the existing territorial statistical information • Increase a permanent settlement of statistical data and indicators supply on regional and sub-regional basis • Use new methodologies and tools for producing territorial statistical estimations (SAE Methods) • Disseminate results achieved to different users and scientific society as a whole

23. Further effects of the implementation of a Statistical Information System at local level IMPACT OF THE DEMAND FOR TERRITORIAL AND SECTORIAL STATISTICS • Increasing of sensitiveness to the topics of territorial data and indicators • Implementation of exchange of territorial data • A RELATION IMPACT • Involvement of others administrations to provide and ask for territorial data, and to disseminate the Culture of Measurement and Evaluation • AN INSTITUTIONAL IMPACT • A direct reply to an increasing demand of territorial indicators coming from other actors not directly involved in policy making • AN EXTERNAL IMPACT • Need to increase investments and financial resources to implement • the local information statistical systems

24. 5Different methods to get information There are various methods to collect information about large population Disadvantages Advantages collection of data in a standardized methods over large number of units Not ever it is possible to standardized; possible errors • Statistical surveys • Administrative record systems (Big Data?) Sometimes offer very good data Little control over the measurements Yield deep understanding Small groups of informants • Qualitative investigation Are limited to a tiny fraction of behaviors (no quality control?) • Observation of the behaviors of units (Big Data?) Information on frequency of events Difficult applicability in real world Answer as stimuli cause behavior • Randomized experiments Not the only way, but the most important is Statistical Surveys Need for Small Area EstimationMethods (SAE), Monica Pratesi Lectures

25. 6Need for data of high quality: how to evaluate it • Confidence of decision makers and citizensin the quality of the statistical information is indispensable in order to use it without suspect • For this reason the international organization (UN and EUROSTAT) state that itis important that the statistical outputs meet some fixed quality standards (relevance, accuracy, timeliness, accessibility, interpretability, coherence and comparability)and actually they try to guarantee it • It is therefore necessary that the user of data know the characteristics of data and indicators and also their possible errors and interpretative limitations • The quality of data can be evaluated only from researchers that know “Methodology of Statistical Survey” • (Monica Pratesi Lectures)

26. References • Biggeri L. (2004), Official Statistics for Decision Making and Evaluation: territorial indicators, in Proceedings of OECD First Forum on: Statistics, Knowledge and Policy: Key indicators to Inform Decision Making, Palermo • Bedi T., Coudouel A. and Simler K., (2007), More than a Pretty Picture: Using Poverty Maps to Design Better Policies and Interventions, World Bank • Ray P., Greenhaigh T., Harvey G. and Walshe K., (2005), Realist Review – a new methods of systematic review designed for complex policy interventions, Journal Health Service Reseach Policy, Vol. 10, Suppl- 1 July 2005 • To get more information see also the Proceedings of the OECD Forums on Statistics, Knowledge and Policy hold in Instanbul, Turkey, (2007), Busan, Korea (2009), New Delhi, India (2012) and the Forum that has been hold in Mexico from 13 to 15 October 2015. The next will be held again in Korea.