Indicators: Levels, Types, Existing and New

1. Indicators: Levels, Types, Existing and New Ken Mease, University of Florida Cairo, June 2009

2. What is an Indicator? • An indicator is a device for providing specific information on the state or condition of something. An indicator is also a measure, gauge, barometer, index, mark, sign, signal, guide to, standard, touchstone, yardstick, benchmark, criterion and point of reference. (Source: Oxford Dictionary)

3. The differences between the macro and micro levels should be examined closely. At the macro level we find political institutions At the micro level we find the experiences, attitudes and beliefs of typical citizens. In particular the extent to which the political institutions are accepted as legitimate Macro, Micro or Both Levels of Governance

4. Many claim that to have democratic governance, there must be a fit between these two levels. At the macro level there should be political institutions that conform to democratic procedures and the rule of law. At the micro level, an acceptance by the mass public, who in a democracy have the power to sanction or remove their leaders. A careful examination of the political institutions and how they are perceived by the public is needed in a comprehensive governance assessment. Macro, Micro or Both

5. The de facto form of data refers to what happens in practice. An example here might be to ask citizens the degree to which there is freedom of the press in a country The de jure form refers to the existence of formal rules. These formal written rules are often found in laws, regulations and/or the constitution. For instance, a law or constitutional provision protecting freedom of the press De facto or De jure?

6. Some projects assess governance, democracy and human rights with only one form of data – either de jure or de facto. While other projects assess both the de jure and de facto states of governance, democracy and/or human rights. This is because sometimes rights or laws may exist on paper (de jure), but not in practice (de facto). De facto, De jure or Both?

7. Triangulation • Governance operates on two levels, the macro level of political institutions and the micro level of the citizens • The ideal situation may be to combine a professional desk study that captures the de jure state of governance with de facto survey of typical citizens and other available data to provide both qualitative and quantitative data respectively • Quantitative data provide numbers and statistics, while qualitative data can offer deeper context

8. An index or a scale? •  Many of the existing governance assessment report their results as an index. • An index is often made up of many types of data – surveys, archival data, desk studies, etc. - collected in different ways from different stakeholders . • Itisa numerical tool that represents a combination of indicators and information that can be compared over time.

9. Can be developed from archives or other data sources. De jure examples include the existence of an integrity commission, existence of a law protecting human rights, freedom of speech or some other civil liberty. De facto (practice) examples include the number of corruption cases prosecuted or the number of human rights violations reported. Objective indicators

10. These indicators are often used in surveys of typical citizens They are also found from a variety of sources including government statistics, such as number of arrests, convictions, etc. Household surveys conducted by Central Statistical Offices often ask citizens if they have ever been asked to pay a bribe to a public official (de facto). Reported behavior or eventsbased

11. Often found in surveys of typical citizens or smaller surveys of key government and no-government stakeholders They rely on opinions or perceptions of how things are (de facto). Perception based data have proven to be very reliable over the years in many different contexts and cultures. Moreover, in many cases they are the only data available. Subjective or PerceptionBased

12. Measure the subject of interest indirectly, rather that directly. Sometimes issues of time and money influence the need to use proxy indicators. Other times, researchers use proxies for subjects that are difficult to measure directly. Proxy indicators

13. Theory - when possible, broadly based theory should guide how we operationalize these concepts into testable hypotheses Validity - does the indicator measure what it purports to measure? Reliability - can the indicator be produced by different people using the same sampling, coding rules and source material? Clarity – make sure the questions can be understood by the population Good Measurement: An Overview

14. Lack of Transparency - in the production of the indicator Representativeness - for survey data, what is the nature of the sample of individuals? Variance Truncation - the degree to which scales force observations into indistinguishable groupings Aggregation Problems - for combined scores, to what degree are aggregation rules logically inconsistent or overcomplicated. Information Bias - what kinds of sources of information are being used? Measurement Issues

15. New or Existing Indicator Checklist • What sources of data are available for this indicator? • What type of data will this indicator produce? • Can the results be disaggregated? • At what level does it assess governance: macro or micro? • What form of governance is examined: de facto or de jure?

16. New or Existing Indicator Checklist • Will it be an objective, perception-based or a proxy indicator? • How should it be measured: by text (qualitatively), numerically (quantitatively) or by both? • Will it capture the local context and/or vulnerable groups?

17. Participatory approaches to developing indicators • Participation in indicator identification by an expanded group of stakeholders will likely increase the amount of ownership people feel in the process and increase interest in the results. • However, the larger the group, the longer and more costly the process is likely to become. When it comes to the more technical side of developing the indicators, a smaller skilled group is best.

18. Developing New Indicators or Not • Developing new indicators requires skill and cost $$$. • Many previously used indicators have been tested for reliability and validity, which can save time and money. • Only develop new indicators where gaps exist. • Brainstorming in a meeting of stakeholders is the easy part; meeting high professional standards for this type of work is much harder. • Like most factors, it may come down to how much time, money and human resources are available.

19. Conclusions, costs and benefits concerning indicators •  Best to try and strike a balance between “core” (existing) and “satellite” (new country- or culturally-specific) indicators • In the recent Mongolian assessment only 11 percent of indicators used fell into the “satellite” category. • It also is recommended that both de jure and de facto forms of governance be assessed, and the macro and micro levels of governance be measured as directly as possible. 

20. Exercise • Pick a governance issue in your country • Develop approximately five indicators • Identify the level – macro or micro • Identify the type of indicator • Objective • Reported events • Perceptions • Proxy • Identify the source of the data