Luiz Awazu Pereira da Silva Ministry of Budget and Planning (Brazil)

1. Building Statistical Capacity to monitor the Progress of SocietiesTechniques for Evaluating Public Policies in Developing Countries Luiz Awazu Pereira da Silva Ministry of Budget and Planning (Brazil) OECD World Forum on Statistics, Knowledge and Policy on Measuring and Fostering the Progress of Societies Istanbul, Turkey from 27 to 30 June 2007 Based on: “The Impact of of Economic Policies on Poverty and Distribution, Volume 1” by François Bourguignon, Luiz A. Pereira da Silva, eds. The World Bank, Oxford University Press (2003), and Volume 2 with Maurizio Bussolo (ed.) forthcoming (2007)

2. Measuring Progress of Societies changed from 50s (NA, aggregate data) to 90-00s specific economic policies and structural reforms in DCs (HHS, micro data) • Even macro-economic policies, monitoring post-stabilization requires specific surveys, not just NA aggregate data Fiscal monetary policy stance, exchange rate regime, public debt management strategy, etc. role of expectations (from market surveys) and of high frequency specific data (daily) • Evaluation of efficiency of public expenditures  design of social policies requires targeting thru micro data Tax policy reform, composition of public expenditure, design of social programs (CCTs) civil service reform, pension reform, decentralization • Ex-ante analysis of structural reforms too Trade liberalization, liberalization of specific markets, financial sector reforms, improving the investment climate, land reform, privatization etc.

3. Measuring Progress in Brazil: New Data and Surveys (Monthly, Daily to monitor macro policies and efficiency of public spending) • Credibility of Macro-Policies need to be monitored by high frequency macro data from specific surveys (daily, weekly) • Where is really going public spending (40% of GDP of about $1trillion), especially health, education, social security, requires HHS and micro data

4. Measuring progress with macro economic policies Macro data, aggregated, NA and specific surveys • G • public spending. • p inflation • w wages • L employt. • Y growth • Aggregate and first principle analysis used: assuming homogeneity of individuals and households, adds up into aggregate or “average” impact (e.g., higher growth, higher employment, lower inflation, higher social spending are all “good” on average for progress of societies) • Policy objectives essentially macro concepts (e.g., IMF, World Bank of the 1960-70s, “old” Governments, etc.) • Macro data bases (NA) available as the natural analytical environment for aggregate, growth/social analysis

5. Measuring progress with social policies • Scope and objective, challenges increase: evaluate the economic feasibility of public programs and policies and their overall ‘development” impact. • Aggregate and first principle analysis insufficient: heterogeneity of individuals and households, microeconomic behavior do not add up into aggregate nor “average”, specificities of economic structures and local political economy, transmission of shocks and policies • Policy objectives and social demand increasingly focusing on distributional effects and poverty reduction, essentially micro concepts (e.g., Post-WC IFIs, new types of Governments, etc.) • Micro data bases (household surveys HHS) increasingly available as the natural analytical environment for distributional and poverty analysis

6. Measuring progress in social policies Define impact for individual i as the difference in income yi with and without the program, denominated Dyi : yi : real income wi : wage rate Li : labor supply Ei : self-employment, non-wage income Ri : net transfer income Ai : socio-economic characteristics Ci : consumption characteristics  household-specific P price index p : general price index

7. Program or policy will shock one or more components that explain the individual income yi • A households character. • R transfers • p prices • w wage • L employt. Household Survey (HHS), i individual households Evaluation of Program and/or Economic Policy • Compare the distribution of y|P=1 with the distribution of y|P=0. • Calculate changes in inequality or poverty across the two distributions • Different tools/methods differ in how they construct the counterfactual distribution and the data that are needed • Rank results according to some agreed upon rule and/or objective

8. Several “Microeconomic techniques” 1. Average Incidence Analysis • Tax Incidence Analysis (Sahn & Younger) • Public Expenditure Incidence Analysis (Demery) 2. Marginal Incidence Analysis • Behavioral response to changes (Van de Walle) • Poverty mapping (Lanjouw) 3. Impact Evaluation (randomization, matching, double-dif) a) Ex-post (Ravallion) b) Ex-ante (Bourguignon & Ferreira) 4. Data and Measurement (not covered here) a) Multi-topic Household Surveys(Scott) b) Qualitative surveys (Rao & Woolcock) c) Performance in Service Delivery (Dehn, Reinikka & Svensson)

10. New “Macroeconomic techniques”, from robust to more speculative…. 1. Standard RHG approaches to macro-micro linkage: • "Micro-accounting"/RHG approach based on aggregate macro predictions (PovStat-SimSip-PAMS) • The disaggregated SAM-CGE/RHG approach (Adelman-Robinson, Bourguignon and al. in the "Maquette“, Loefgren, Robinson or Agenor and al. with IMMPA.) 2. Top-down "micro-simulation" approaches (micro-macro linkages) • "Micro-accounting modules" linked to disaggregated macromodels(Chen-Ravallion, McCulloch-Winters) • "Micro-simulation modules" linked to disaggregated macro models (Bourguignon-Robilliard-Robinson, Ferreira-Leite-Pereira-Picchetti, Cogneau-Robilliard-van der Mensbrugghe) 3.Other issues for research and applications: a) Fully integrated models(Townsend, Heckman, Browning-Hansen-Heckman) b) Accounting for general equilibrium effects of public expenditure programs c) Dynamic modeling and the proper treatment of growth

11. Evaluation of macro economic policies.Macro to micro linkages Macro framework, general/partial equilibrium • A households character. • p prices • w wage • L employt. • R transfers Instead of « exogenous and independent » shocks like before  use « endogenous and  dependent» shocks to 'microsimulate' the effect of policies on all individuals in the micro data sets, and the poor  some consistency constraints will be « binding » (e.g., budget envelope for social programs, real GDP growth, etc.) LAVs =Linkage Aggregate Variables Household Survey (HHS), i individual households

12. Top-down "micro-simulation" approach within a macro-micro linkageapproach Macro model Linkage AggregatedVariables (prices, wages, employment levels) Household income micro-simulation model

13. Top-down "micro-simulation" approach within a macro-micro linkageapproach LAVs from above Two distinct approaches to micro-module: - "micro-accounting" : no explicit change in behavior (envelope theorem argument), e.g., Chen-Ravallion - "micro-simulation" : change in behavior, possibly linked to (labor) market imperfections, e.g., Robillard, Bourguignon, Robinson and Ferreira, Leite, Pereira da Silva, Picchetti Household income micro-simulation model

14. Example of Brazil Results using this technique: Aggregate Poverty and Inequality Indices (on aggregate, good results) after financial crisis 1999

15. Example: Brazil, 1999 Financial Crisis, Results of Simulationnominal changes in per capita income after floating ER, across centiles of income

16. Top-down "micro-simulation" approach Final Remarks: introducing feedbacks Macro model Feedback, e.g., micro-transfers, minimum wage Linkage AggregatedVariables (prices, wages, employment levels) Household income micro-simulation model

17. Concluding Remarks • Statistical capacity is needed more than ever and evolves with changes in policy-making practices and objectives (from aggregate growth to local quality & efficiency of public spending, targeting, evaluation ex-post/ex-ante) • Data needed evolved from aggregate-macro (National Acc.) to sectoral and mostly micro (HHS); continuity of time series is paramount to measure policy results • Information technology for collecting data, lower costs, allows applying techniques to MICs and LICs; international coordination (OECD, WB, etc.) important for cross-country and panel comparisons with policy-implications • Statistical capacity, data collection needs to be built-in project and policy design, ex-ante and not only ex-post • Macro policies need continuous micro evaluation of results to measure progress/efficiency (e.g., Bolsa-Familia in Brazil)

18. Effectiveness of Public Spending, Redistribution Poverty Reduction Measuring Progress in Society? “Effective social policies” for growth and redistribution requires evaluation micro-data (HHS) & adequate policy design “Kuznets”type-curve Aggregate growth then redistribution Output Growth, quantitative indicators

19. ENDCaminante no hay camino, se hace camino al andar…(Antonio Machado)

20. Question? Usage of the techniques to meet DCs policy challenges, if, when, where and how • Are these techniques for evaluation of public policies used? Some of these techniques are 'simple', yet not all of them are widely used, why? Costs (training) and institutional implications for Ministries and agencies (Finance vs. Planning, political economy of budget process, etc…). Aid agencies (IDA, DFID, AFD, etc…) promoting evaluation, • When these techniques are used, are they useful for policy-makers? They capture only the 'between' (RHG) dimension of distributional changes, which empirically proves limitative. They are ill adapted to the distributional aspects of growth, but important in putting broader (poverty) perspective to decisions • Where are these techniques for evaluation of public policies used? Examples below

21. Usage of these “techniques” in connection with policy-making (with external assistance) Average Incidence Analysis (Tax Incidence Analysis and Public Expenditure Incidence Analysis) Most OECD countries. Also in many DCs particularly IDA [Ghana (ISSER), Madagascar (INSTAT & Cornell Univ.), Uganda (EPRC)] Marginal Incidence Analysis Many OECD countries and India, using NSS 1994; Indonesia, using SUSENAS 1981 & 1987; Vietnam using panel from VLSS 1993 & 1998; Argentina,using public spending and census data, Ministry of labor team; Brazil using PNADs Poverty mapping Many OECD countries Ecuador, Bolivia, Mexico, Panama, Nicaragua, Guatemala, South Africa, Madagascar, Kenya, Uganda, Malawi, Mozambique, Tanzania, Bulgaria, Albania, Thailand, Vietnam, Cambodia, Indonesia, China., Brazil, etc. Ex-post impact evaluation methods (randomization, PSM, double-dif) Many OECD countries. Also in many DCs, Argentina, Brazil, Kenya

22. "Micro-accounting"/RHG approach based on aggregate macro predictions SimSIP/PAMS Latin America, Burkina-Faso, Thailand, Indonesia The disaggregated SAM-CGE/RHG approach IFPRI (US) country models, IMMPA-Cameroon, Brazil, many countries have GTAP+LAVs+HHS "Micro-accounting modules" linked to disaggregated macromodelsChina, Colombia, Brazil, Many countries have GTAP+LAVs+HHS "Micro-simulation modules" linked to disaggregated macro modelsIndonesia, Brasil More sophisticated? Fully integrated models (Thailand, Madagascar) Accounting for general equilibrium effects of public expenditure programs (???) Dynamic modeling and the proper treatment of growth (???)