Aggregate Governance Indicators

1. Aggregate Governance Indicators Aart Kraay The World Bank Presentation at World Bank Conference “The Empirics of Governance” May 1-2 2008

2. Why Aggregate Indicators? • synthesize information about governance from a diversity of viewpoints • particularly useful for advocacy (look at success of TI-CPI) • achieve greater country coverage than individual indicators • enable comparisons across different sources • smooth out idiosyncracies of many individual sources • better measure of broad concepts of governance • but at the cost of specificity • generate explicit measures of the imprecision of aggregate – and individual – indicators they can (almost) always be disaggregated!

3. Plan for Talk • How to construct and use aggregate indicators • define what you want to measure • select sources and combine them • compute (and use!) margins of error • discuss only “multisource aggregates” • Details, details, details..... • balanced or unbalanced comparisons? • independence of errors? • relative or absolute changes? • complexity? • scaring the jello? • Summing up

4. Defining Topics for Aggregate Indicators • governance is hard to define sharply • but ... easy to overstate lack of definitional consensus around governance (see next slide) • some areas of governance have particularly clear and unambiguous definitions: • “Corruption is the use of public office for private gain” • lack of definitional agreement (at the margin?) should not paralyze measurement efforts • proponents of alternative definitions can feel free to construct their own indicators • are the resulting country rankings different? • what do we learn from the differences?

5. What do we mean by “Governance” ? • World Bank (1992): "Governance is the manner in which power is exercised in the management of a country's economic and social resources for development“ • World Bank (2007) definition: "...the manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide public goods and services" • WGI Definition (1999): "...the traditions and institutions by which authority in a country is exercised. This includes the process by which governments are selected, monitored and replaced; the capacity of the government to effectively formulate and implement sound policies; and the respect of citizens and the state for the institutions that govern economic and social interactions among them."

6. Six Dimensions of Governance in the WGI • The process by which those in authority are selected and replaced • VOICE AND ACCOUNTABILITY • POLITICAL STABILITY & ABSENCE OF VIOLENCE/TERRORISM • The capacity of government to formulate and implement policies • GOVERNMENT EFFECTIVENESS • REGULATORY QUALITY • The respect of citizens and state for institutions that govern interactions among them • RULE OF LAW • CONTROL OF CORRUPTION

7. Selecting Individual Indicators For Use As Ingredients of Aggregate Indicators • view individual indicators as imperfect or noisy proxies for broader concepts of governance, e.g.: • control of corruption: proxies include: • is corruption widespread? • percent of contract value demanded in bribes? • risk that value of FDI adversely affected by bribes? • Crucial observation: proxies do not need to be perfect to be useful! Indicator = Signal + Noise • ideally want indicators with low Noise/Signal... but Noise/Signal=0 is unattainable • as long as Noise/Signal>0, indicator is useful ingredient for aggregate indicator • aggregation can be used to downweight indicators with high Noise/Signal (to come....)

8. More Examples of Ingredients for Aggregate Governance Indicators • Rule of Law (WGI) • enforceability of private contracts (DRI) • fairness/speed of judicial process (EIU) • confidence in police (GWP) • property rights over rural land (IFD) • many more.... • Sustainable Economic Opportunity (Ibrahim Index of African Governance) • GDP/Capita, Growth, Inflation, Budget Deficit • Days to start a business • Contract-intensive money • Road density, computer and internet density

9. Placing Indicators in Common Units • Trivial to rescale data to 0-1 scale • More subtle issue: how do we compare a 7/10 score in a source that covers mostly developed countries with a 7/10 score in a source that covers mostly developing countries? • Option 1: percentile matching (TI-CPI) • Source 1: A > B > C • Source 2: C > D • Aggregate: A > B > C > D • Option 2: elaboration on unobserved components model (WGI). Details in KKZ (1999). • Useful byproduct of aggregate indicators is that it allows comparisons based on dissimilar sources (in example above you can now compare country A and country D)

10. Weight A Minute – All Aggregate Indicators Require Decisions on Weights! • Option 1: Arbitrarily assign weights • equal weights (e.g. TI-CPI, most others) • different weights based on views of what matters more (e.g. Ibrahim Index of African Governance) • decision to exclude a source implies setting a zero weight (e.g. TI-CPI excludes all household surveys) • Option 2: Let the data choose the weights (logic of unobserved components model underlying WGI) y1=g+e1 y2=g+e2 y3=g+e3 • if CORR(y1,y2)>>COR(y1,y3), y1 and y2 are more informative about g (if errors are independent) • Option 3: Regression-based weights to capture importance of each indicator for outcomes • not widely (ever?) used • in principle is appealing, in practice virtually impossible

11. Does Weighting Matter? • Depends crucially on the extent to which the underlying data sources are correlated with each other • if correlations are high, weighting matters little • if correlations are low, weighting matters a lot • Example: Two robustness checks on WGI weighting scheme for Control of Corruption • Option 1: equally-weighted • Option 2: aggregate 4 types of sources (commercial, NGO, public sector, and surveys) • Very highly correlated with baseline WGI indicator • Option1: correlation with baseline = 0.998 • Option 2: correlation with baseline = 0.959 • conclude that “ingredients” of WGI-CC are quite highly correlated – so details of weighting don’t matter much

12. Spot the Difference: Alternative Aggregations of WGI-Control of Corruption Indicators Using Different Weights

13. Margins of Error • margins of error summarize the degree of disagreement across sources in their assessment of governance • two ways to construct them: • standard deviation across sources • estimate based on a structural statistical model (e.g. WGI uses unobserved components model) • precision-weighting of sources in WGI (modestly) reduces margins of error • aggregation reduces measurement error about broad concepts (smooths out idiosyncracies of individual sources) • essential to use them to assess significance of cross-country differences or changes over time

14. Margins of Error Decline With the Number (and Quality) of Data Sources

15. Good Governance Control of Corruption Selected Countries, 2006 Margins of Error Governance Level Poor Governance DISCLAIMER: The data and research reported here do not reflect the official views of the World Bank, its Executive Directors, or the countries they represent. The WGI are not used by the World Bank Group to allocate resources or for any other official purpose. Source for data: 'Governance Matters VI: Governance Indicators for 1996-2006’, by D. Kaufmann, A. Kraay and M. Mastruzzi, June 2007, www.govindicators.org. Colors are assigned according to the following criteria: Dark Red: country is in the bottom 10th percentile rank (‘governance crisis’); Light Red: between 10th and 25th percentile rank; Orange: between 25th and 50th percentile rank; Yellow, between 50th and 75th; Light Green between 75th and 90th percentile rank; and Dark Green: between 90th and 100th percentile (exemplary governance). Estimates subject to margins of error.

16. Changes in Control of Corruption, 1996-2004

17. Margins of Error: A Little Perspective • do not confuse absence of explicit margins of error with absence of measurement error – present in all governance indicators • margins of error are not unique to subjective- or perceptions-based aggregate indicators • can infer them based on inter-correlations of any type of indicator • keep the baby, ditch the bathwater! • 2/3 of pairwise comparisons on WGI are significant (at 90% level) • 1/3 of countries show a significant (at the 90% level) change in at least one of the six WGI between 1996 and 2006

18. Details 1: Don’t Lose Your Balance! • comparisons of aggregate indicators across countries and over time are often “unbalanced” – different set of sources underlying the two comparators • the alternative (strictly balanced) is far too restrictive • balanced WGI-CC based on top five sources would cover just 117 countries, not 207 • much less diverse set of sources as well • “unbalancedness” is not so bad as you think! • 60% of pairwise comparisons in WGI involve 5 or more common sources • just 7% of variation in large changes due to changes in composition of sources • can always go back to the source data!

19. Details 2: I Think You Think I Think You Think I Think You Think Bangladesh is Corrupt • Correlated perception errors are potentially an important issue, as they could: • reduce the information content of aggregate indicators • distort weighting scheme • First-order issue: single- versus multiple-source aggregate indicators • single-source aggregates average responses of the same experts to many questions (CPIA, GII, DB, etc) • almost by definition have strongly correlated perception errors across components • multiple-source aggregates are less subject to this problem • unless perception errors perfectly correlated, still can get efficiency gains from aggregation

20. Evidence on Correlated Perception Errors? • easy to assert, but hard to test y1=g+e1 y2=g+e2 • all we observe is CORR(y1, y2) – is it because: • CORR(e1, e2) is high? • VAR(e1) and VAR(e1) are small? • need an identification strategy (not storytelling) • Example: expert assessments more likely to make correlated perceptions errors than survey respondents • are expert assessments more correlated with each other than with surveys? Not necessarily • average pairwise correlation of 5 expert assessments of corruption = 0.80 • average correlation of each with a firm survey = 0.82 • correlations among expert assessments don’t increase over time

21. Belarus

23. Details 3: Everything is Relative .... Or Is It? • All indicators require choice of units • 0-10 (TI-CPI), 1-6 (CPI), A-D (PEFA) • WGI has particularly nerdy choice (are you surprised?): • standard normal distribution • forces world average = 0 in each period • most other indicators also implicitly make choices about averages (e.g. CPIA grade inflation) • Does this confuse relative and absolute changes? Δy(j) = Δ(y(j)-average) + Δaverage • absolute changes and relative changes coincide if no changes in the world average • look for evidence in individual sources whether world averages change – answer is a resounding “no”!

24. Details 4: It’s Just Too Haaaaard.....! • common critique is that aggregate indicators are too complicated and non-transparent • same is true for all kinds of things (national income accounts, PPP adjustments, poverty measures, the NFL draft, the engine under the hood of my car) • better to be complicated (and a bit closer to right) than naive (and a bit further from right)

25. Details 5: Scaring the Plants (and the Jello) • how big are the risks of measurement ahead of theory? • risk of inaction until we all agree on a theory is worse • how do we verify indicators (and ingredients of indicators)? • are different indicators of a core dimension of governance correlated? • exactly what unobserved components model does • are they uncorrelated with other core dimensions of governance? • pretty much a hopeless question since core dimensions of governance are correlated • more or less “free entry” in the market for indicators • more interesting to show the quantitative relevance of critiques than to simply speculate

26. Summing Up • aggregate indicators can (for some purposes) serve as a useful summary of large numbers of indicators • but no reason to be wedded to any particular aggregate • we learn a lot from cross-referencing alternative related indicators as part of process of building aggregates • why are they similar, why are there outliers? • formally can construct margins of error • crucial for policy dialogue (and sensible use) • lots of potential for argument over the “nitty-gritty details” • less clear that these are first-order concerns

27. Bottom Line • differences between: • alternative aggregates, • aggregate versus individual, • subjective vs objective • ‘actionable’ versus ‘whatever the antonym is’ are minor compared to difference between having data and not having it at all