Actionable Governance Indicators (AGI s )

1. Application and Development Stephanie Trapnell, PRMPS April 29, 2010 AGI Initiative http://agi Actionable Governance Indicators (AGIs)

2. Overview • What is, and is not, an AGI? • AGIs on different governance dimensions • Application and development of AGIs

3. What are AGIs? Exogenous factors

4. Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials • Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement. • Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students • Goods and services produced by the project. • Examples: Surveys and Trainings conducted, Laws revised, Agencies established The Missing Middle

5. What are AGIs? • AGIs provide information on the discrete elements of governance reforms. • Specific • Narrowly-defined • AGIs reflect actionability, i.e., they establish the link between actions and indicator values.

6. What AGIs cannot do…. • AGIs do not provide a comprehensive picture of governance in a country. • AGIs do not provide a general picture of the state of a particular governance system, i.e., business regulation, freedom of information, etc. • Individual AGIs are not used for ranking countries. • Except in cases of aggregation!

7. Revising targets into AGIs

8. Clearly defined? • Improved participatory budgeting in 30 poor municipalities. • Capacity of authorities in transport management and project implementation strengthened • Greater use of flexibility in bankruptcy law

9. 4. AGIs on governance dimensions

10. Governance Dimensions Citizens/Firms • Political Accountability • Political competition, broad-based political parties • Transparency & regulation of party financing • Disclosure of parliamentary votes • Effective Public Sector Management • Ethical leadership: asset declaration, conflict of interest rules • Cross-cutting public management systems: meritocracy, public finance, procurement • Service delivery and regulatory agencies in sectors • Checks & Balances • Independent, effective judiciary • Legislative oversight (PACs, PECs) • Independent oversight institutions (SAI) • Global initiatives: UN, OECD Convention, anti-money laundering • Civil Society & Media • Freedom of press, FOI • Civil society watchdogs • Report cards, client surveys • Private Sector Interface • Streamlined regulation • Public-private dialogue • Extractive Industry Transparency • Corporate governance • Collective business associations Citizens/Firms Citizens/Firms • Decentralization and Local Participation • Decentralization with accountability • Community Driven Development (CDD) • Oversight by parent-teacher associations & user groups • Beneficiary participation in projects Citizens/Firms

11. Different Dimensions of Good Governance Are Measured Differently (examples) Citizens/Firms • Political Accountability • Political competition, broad-based political parties • Transparency & regulation of party financing • Disclosure of parliamentary votes Public Accountability Mechanisms Global Integrity Freedom House Afro/Latino/AsiaBarometer

12. Different Dimensions of Good Governance Are Measured Differently (examples) • Civil Society & Media • Freedom of press, FOI • Civil society watchdogs • Report cards, client surveys • Private Sector Interface • Streamlined regulation • Public-private dialogue • Extractive Industry Transparency • Corporate governance • Collective business associations Public Accountability Mechanisms (PAM) Human rights database Reporters Without Borders Media sustainability index Citizens/Firms Doing Business sub-indicators Business Environment and Enterprise Performance Surveys (BEEPS) Investment Climate Assessment (ICA) Business Risk Service Transparency International (TI) index Global Competitiveness index

13. Different Dimensions of Good Governance Are Measured Differently (examples) Public Expenditure Review (PER) Public Expenditure Tracking Surveys (PETS) Public Expenditure and Financial Accountability (PEFA) Quality of Service Delivery Surveys (QSDS) Score cards • Decentralization and Local Participation • Decentralization with accountability • Community Driven Development (CDD) • Oversight by parent-teacher associations & user groups • Beneficiary participation in projects Citizens/Firms

14. Different Dimensions of Good Governance Are Measured Differently (examples) Judicial/Rule of Law indicators PEFA TI index Global Integrity indicators Open Budget Initiative • Checks & Balances • Independent, effective judiciary • Legislative oversight (PACs, PECs) • Independent oversight institutions (SAI) • Global initiatives: UN, OECD Convention, anti-money laundering Citizens/Firms

15. Different Dimensions of Good Governance Are Measured Differently (examples) Public Accountability Mechanisms (PAM) PSG country-specific projects Score cards PEFA Global Integrity index OECD procurement indicators • Effective Public Sector Management • Ethical leadership: asset declaration, conflict of interest rules • Cross-cutting public management systems: meritocracy, public finance, procurement • Service delivery and regulatory agencies in sectors

16. AGI Initiative http://agi http://agidata [WB internal] https://www.agidata.org [external] 5.Application and Development of AGIs

17. AGI Product Lines

18. Incorporating AGIs into projects

19. Established Data Sources http://agidata [WB internal] https://www.agidata.org [external]

20. AGI Sources • Content Summary: • 6 thematic areas, 1182 indicators • 13 sources Note: The total includes the main indicators & sub-indicators.

21. Data Sources on AGI Data Portal • AfroBarometer – Governance in Social Sectors (AFROBAR) • Business Environment & Enterprise Performance Surveys (BEEPS) • Country Policy and Institutional Assessments (CPIA) • Doing Business (DB) • Enterprise Surveys (ES) • Global Integrity Index (GI) • Human Resource Management (HRM) • International Budget Practices and Procedures Survey (IBPP) • Open Budget Index (OB) • Press Freedom Index by Reporters without Borders (RSF) • Public Accountability Mechanisms (PAM) • Public Expenditure & Financial Accountability (PEFA) • Statistical Capacity (SC)

22. Thematic groups of AGIs • Public Sector Management • Human Resource Management • Public Finance Management • Procurement • Service Delivery • Statistical Capacity • Political Accountability • Electoral Competition • Branches of Government • Integrity Systems • Budget Oversight • Checks and Balances • Branches of Government • Oversight Institutions • Redress Institutions • Other • Civil Society and Media • Civil Society Organizations • Media • Press Freedom Index • Public Access to Information • Open Budget • Public Access to Key Fiscal Information • Private Sector Interface • Permits and licenses • Regulation of Entry • Property Rights • Corruption in Business-Government Transactions • Infrastructure • Trade • Taxes • Decentralization and Local Participation • Transparency of inter-governmental fiscal relations • Oversight of aggregate fiscal risk from other public sector entities

23. AGI Dataset Snapshots

24. Guidance on Indicator Selection

25. Guidance on Indicator Selection

26. Guidance on Indicator Selection

27. Guidance on Indicator Selection

28. Established Data Collection Instruments • Expansion of datasets in established AGI sources • Human Resource Management • Public Accountability Mechanisms • Public Expenditure Tracking Surveys/ Quality of Service Delivery (PETS/QSDS) • Teacher Policies around the World

29. Considerations when using established instruments Limitations • Quality of data • Reliability of data Consistency • Sampling methodology • Survey instruments

30. Developing new AGIs: Considerations

31. Conceptual Clarity • Narrowly-defined indicators that reflect empirical research • Making Services Work: Service Delivery in the Human Development Sectors • Forestry Outcomes: An Analytical Framework for Governance Reforms • Public Accountability Mechanisms (PAM) Methodology Paper

32. Relevance and Validity • Relevance • Is the indicator capturing data that is relevant to the goals of the project? • Does the data being captured help to understand the design, capacity or performance of a system under study? • Validity (Construct) • Is the indicator measuring what it claims to be measuring, or is it measuring something else?

33. Reliability • Inter-rater or Inter-observer • Two or more independent experts provide data • Parallel forms & Internal consistency • Different questions assess the same construct • Test-Retest • Used for items that are stable over time, such as personality • Statistical tests for reliability • Average Inter-item Correlation, Average Item-total Correlation, Split-Half Reliability, Cronbach's Alpha

34. Other considerations • Availability of data • Can the data be captured, given the resources of the project and the context? • Sustainability • Should data collection be conducted regularly for the data to be useful? • Exogenous factors • Cannot always be accounted for in the design of the indicator

35. Red flags on Aggregation • Aggregation requires an understanding of the relative importance of each AGI to the concept being measured in the aggregated indicator. • What are the underlying weights of the AGIs? Equal weighting (averages) or unequal weighting (assigning more importance to some AGIs but not others). This understanding should be rooted in empirical research.

36. Thank you!