Musings on Governance Indicators, and some pathways forward…

1. Musings on Governance Indicators, and some pathways forward… Daniel Kaufmann The World Bank Institute Presentation at Seminar on The Empirics of Governance, May 1-2, 2008, World Bank, Washington DC

2. Overview • Preamble: Many extremely valuable efforts… • Measuring governance/institutional quality • some conceptual / definitional musings • what do we measure? rules vs. outcomes • whose views? experts vs. survey respondents • measurement error • disaggregation, ‘actionability’, & aggregation • Constructively Moving forward…

3. Do we know what governance is? • Ok: No Agreed standard or Unified Theory (AUT) of governance • Yet: Absence of AUT is also the case with most any field in social science/economics fields… [& physics..?] • And: Absence of AUT in many fields hasn’t deter measurement • In governance (as in other fields), in spite of no AUT: there is a scholarly body of theorizing on governance & institutions • Further: various definitions have been put forth over the years, mostly they are variants of each other – but we opt against either anorexic or tautologically ‘fat’ definitions – ‘broad’ is ok • Distinguish between pre-requisites for measuring governance, vs. hypotheses on whether, what, and extent of governance mattering for development outcomes (though iterative process) • Key question: do conceptual or definitional differences make a 1st or 2nd order difference to the measured indicators (or are other factors more important)?

4. Definitional Issues, in more detail (1) • Makes sense to avoid overly fat definitions • risks making links from governance to development tautological (think about North quote) • risks lack of focus in what we measure • e.g. Ibrahim Index of African Governance averages together things like corruption, inflation, and per capita GDP • Also makes sense to avoid overly narrow definitions • constitutions  institutions! • de facto rules matter more than their de jure counterparts • Don’t overstate degree of definitional disagreement, most definitions encompass capable states, operating under rule of law, and accountable to citizens

5. What do we mean by “Governance”, Cont’d (2)? • World Bank (1992): "Governance is the manner in which power is exercised in the management of a country's economic and social resources for development“ • World Bank (2007) definition: "...the manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide public goods and services" • WGI Definition (1999): "...the traditions and institutions by which authority in a country is exercised. This includes the process by which governments are selected, monitored and replaced; the capacity of the government to effectively formulate and implement sound policies; and the respect of citizens and the state for the institutions that govern economic and social interactions among them."

6. A Fundamental Challenge in Measuring Governance • The ‘true’ level of governance is inherently unobservable (or, at least, in some dimensions, extremely hard to measure) • Unavoidably: any measure of governance and investment climate are proxies of the ‘true’ level • Cats, mousetraps and bathwater filters: the above means that we need to take very seriously: i) triangulation; ii) margins of error; iii) moving from objective-subjective distinction to more meaningful ones… • Further, different objectives means different tools are more appropriate

7. A Taxonomy of Governance Indicators: moving away from the subjective-objective divide to rules vs outcomes-based

8. Legend for Taxonomy

9. What To Measure: Rules or Outcomes? • Rules-based de jure measures capture formal rules, institutions, and processes • presidential vs. parliamentary system • first-past-the-post vs. proportional representation • does a statutorily-independent election monitoring agency or anticorruption agency exist? • Outcome-based de facto measures capture how these rules are implemented and their effects in practice • is election monitor politicized? • do firms perceive corruption is a problem? • do teachers show up at school? No “bright line” separating the two, think of as a continuum

10. Rules-Based Indicators • Main virtue is clarity: does a rule exist? has legislation been passed? • obvious appeal to gov’ts / donors who want to monitor progress on governance • reflected in language of “actionable” indicators • Three main drawbacks • easy to overstate clarity: measurement of all but most basic rules requires careful judgment – virtually no truly “objective” indicators exist • links from (highly detailed) rules to outcomes not always clear • which “actionable” indicators are “actionworthy”? • risk of “reform illusion” or “teaching to the test” • obvious potential for gaps between de jure rules and de facto outcomes

11. Outcome-Based Indicators • Majority of governance indicators focus on measuring de facto outcomes (governance outcomes, that is…) • budget outcomes relative to budget plans? (PEFA, OBI) • fraction of contract value paid as bribe? (GCS) • is corruption in government widespread? (GWP) • Main virtue is that these measure relevant governance outcomes as seen by stakeholders, including informality • Drawbacks: • (broad) outcome measures hard to link back to specific policy interventions (mirror image of previous problem) • (poorly-designed) questions can be vague and open to interpretation • avoid generic questions, focus on experiential ones • provide benchmarks for respondents (e.g. vignettes)

12. Subjective and Objective Measures of Ease of Business Entry: OECD/NIC Sample High r = 0.51 Low

13. Subjective and Objective Measures of Ease of Business Entry: Developing Country Sample High r = 0.24 Low

14. Are Elections Free and Fair? Global Integrityvs. Global Corruption Barometer Survey Source: Center of Public Integrity, 2006 - http://www.globalintegrity.org/, and Gallup International.

15. Whose Views To Rely On -- Experts or Survey Respondents? • Expert assessments produced by wide variety of organizations, e.g. • Multilateral organizations (World Bank CPIA) • Commercial risk rating agencies (EIU, DRI, PRS, WMO) • NGOs (Freedom House, Global Integrity) • Several large cross-country surveys • of firms (GCS, WEF, BEEPS) • of households (Afrobarometer, Latinobarometer, Gallup World Poll)

16. Asking the Experts: • experts natural respondents for certain questions • details of budget or regulatory procedures not “common knowledge” • are experts “biased”? ideological biases: do “right-wing experts” like “right wing governments”?: difference between expert assessments and surveys not highly correlated with political orientation of country’s government • business-elite biases: do businesspeople just want low taxes and no regulation?: business and non-business views on governance often correlate well across countries • do experts engage in “group-think”? -- in GM V: experts no more correlated among themselves than w/ surveys untainted by group-think, so little evidence of correlated errors

17. Asking Households and Firms • Main advantage is we get information from the ultimate beneficiaries (victims) of good (bad) governance • their perceptions matter because they act on them • Main disadvantages • costly to do regular cross-national surveys • concerns about representativeness (but far less than for single-expert assessments) • badly-designed questions areopen to interpretation • but replace with good questions • generic problem for all types of indicators • respondent reticence • use random response methods to alleviate -- type of political regime may affect responses (enterprises)

18. -- Thus, triangulation is paramount – looking at various rules and outcome-based indicators, drawn from experts and surveys-- Further, studying what matters, what is action-worthy is important

19. Anti-Corruption Agencies (GII rating) associated w/ extent of Corruption (as reported by individuals)? Source: Center of Public Integrity - http://www.globalintegrity.org/, and transparency International.

20. Anti-Corruption Agencies (GII) associated with extent of Corruption (as reported by enterprises)? Source: Center of Public Integrity - http://www.globalintegrity.org/, and World Economic Forum..

21. Disaggregate and Aggregate Indicators • Aggregate indicators have clear drawbacks (but not alone…) • Many individual measures of governance provide imperfect signals about broader concepts • major advantage of aggregation is that allows reduction in and transparent quantification of margins of error • Aggregate indicators are more informative about broad concepts, but at a cost of specificity • average information from distinct sources (WGI, TI) • contrast with “single-source” aggregates averaging answers from same respondent (e.g. CPIA, GII, DB) • here correlated errors are a major concern • Not an “either/or” situation: aggregate indicators can be disaggregated

22. Precision and Number of Sources:Control of Corruption, 2006 Source for data: 'Governance Matters VI: Governance Indicators for 1996-2006’, D. Kaufmann, A. Kraay and M. Mastruzzi, (www.govindicators.org);

23. A list of 10 suggestions on ways forward... • Many valuable existing ‘baths’, with clean (‘signal’) & some dirty (‘noise’) water. Use & Improve • Since we deal w/ proxies: recognize & disclose margins of error in all indicators. • ‘Trapping’ Informality (incl. state capture…):let us not think that we solely rely on formal rules • Continue to develop “action-worthy” indicators, complementing them with “outcome” indicators • Yet, more ‘policy circumspection’: ‘cats & mousetraps’ vs. ‘injections’ -- what Matters is not always ‘WB actionable’ • Triangulate: do not rely exclusively on any oneindicator or source w/out checks/balances w/ others Cont…

24. On Moving forward, cont. • Match indicator(s)/tool with desired objective: in-depth country diagnostic, vs. broad country coverage, and exploit complementarities (aggregate & diagnostics; rules & outcomes) • Distinguish between 1st vs 2nd order problems and prioritize -- Improving questions, sampling, testing & correcting for biases important (vs. ideology, correlated errors, ‘halo’ effects…?) 9. Perceptions do Matter, & likewise accounting for reports from citizens and entrepreneurs – though margins of error, yet not unique (& judgments galore) 10. Fully disclose data sets (& methods, comparability across countries, periodicity, etc), & subject to rigorous referee review -- & avoid censorship from publication of data

25. Expressions... Aristotle, Lord Kelvin, Einstein… already covered, so onto the simple bumper sticker nowadays… “if you think there are problems with the data, try without it…” Lets move forward together…, with some circumspection, cats and mousetraps…