Results Measurement Models: How Can We Measure Enforcement Quality and Reform Success?

1. Results Measurement Models: How Can We Measure Enforcement Quality and Reform Success? GiedriusKadziauskas, Consultant, Inspection Reform and Better Regulation Lars Grava, Consultant, Regulatory Governance

2. Why do Results Measurements matter? (for governments and IFC) Often, need to overcome the lack of experience and motivation by the inspectorates themselves for reform in public sector Results measurement in inspections reforms poses a special challenge: negative incidents mean failure, but lack of incidents rarely is seen as success! Results may not show up soon enough. Remember: to measure results later, you need a baseline first It is very difficult to attribute specific results in terms of safety (or other public goods) to the action of one or several regulators or inspectorates. • Does what you are doing matter? Will there be a positive impact? (Or was there a positive impact?) • Are you on the right track? • Can this generate reform support? Too often, M&E is an afterthought

3. How does IFC count reforms in business inspections? IFC will count a Reform when changes to primary and secondary legislation, changes to administrative procedures, or institutional changes have led to: • 10% or more reduction in the percentage of businesses subject to inspections and meet any of the positive criteria specified in the M&E matrices; or • 10% or more reduction in the number of inspections to which specific categories of companies are subject, and meet any of the positive criteria specified in the M&E matrices ; or • 10% or more reduction in average duration of inspections to which specific categories of companies are subject, and meet any of the positive criteria specified in the M&E matrices. • Number of firms that benefit from reformed inspection requirements Desired Impact: does it lead to Direct Compliance Cost Savings?

4. But to be fair… • WBG’s Investment Climate Department has the following mission statement: “The Investment Climate Department of the World Bank Group helps governments implement reforms to improve their business environments and encourage and retain investment, thus fostering competitive markets, growth, and job creation.” • There is a long history to the time-and-cost savings approach • Now there is interest in reviewing and revising the indicators

5. Besides lower administrative burdens for businesses, what else matters? (Better results with fewer resources!)

6. What else matters for businesses especially in low-income and low-capacity countries • Free from hassle of multiple official and unofficial inspections (Tajikistan – some businesses reported being inspected daily) • Free from public officials, politicians using inspections as a tool to distort money, influence from businesses • Free from unfair competitors using inspections as a tool to remove competitors • Ensuring level playing field with semi legal/illegal competitors - paying taxes, keeping up with environmental, work safety standards • Ensuring clear interpretation rules – honest businesses end up applying highest standard   • Reduce litigation risks with the inspectors that might involve future ramifications for businesses.  Difficult to measure. But essential to communicate to businesses in order to get support for reform and to show adequate assessment of the picture.

7. What do we monitor and evaluate as the result of the reform • Businesses – perception of burden, relationship with inspectorates, compliance with requirements • Inspectorates – positive outcomes, activities • Inspectors - activities

8. Reaching businesses • Surveys – questionnaires to businesses, significant or representative sample • Focus groups – discuss with group of businesses about experiences. • Post-inspection feedback – collect information after contact with the inspector. • ------- • There are advantages and disadvantages – but feedback is a necessity.

9. All of the forms of gathering of information from businesses • Ensure anonymity • “Businesses know and care little about the specific inspections, they are all nuisance for businesses” – be short and clear. • Retain critical approach to results especially when done for the first (second) time. • Build trust – independent surveyor, show how the results were used. • Build capacities in the public sector to carry out surveys, to interpret results, to communicate to the government and public

10. Surveys – there is experience • World Bank Group has for 10+ years been supporting surveys of inspections in countries where it assists with reform work • Surveys provide clear picture on how many businesses are inspected, how often, how long etc. – aggregated across country – by sector – by agency etc. • Of course margin of error – and some answers may be wrong (misunderstandings, bad memory etc.) – but mostly this gets “averaged out” by large numbers of respondents • Very useful information because (among other issues): • Consolidated data was simply missing – Government, even inspectorates, often did not know situation • Discrepancy between practices and “official numbers”

11. Some formulations might show the transition, not only static picture 68%of businesses interviewed this year say inspectors’ attitudes have improved during 2012 in Lithuania (2012)

12. Indicators for inspectorates • More of the good. • Less of the burden. • More results with less resources.

13. Assessing inspectorates • Against the goal – goal setting Fines and violations vs. positive outcomes for safety, better response from businesses • Set of unified indicators (legally binding and allowing flexibility) Set of common indicators for all inspectorates to ensure comparability and support for implementation • Who is watching the inspectorates and monitors the indicators? Government, designated authority, reform group – political power, capacities to “read numbers” from specialized inspectorates.

14. Indicators to measure outcomes • For some sectorial inspectorates same safety indicators are used worldwide (unified methodology, comparability) • Cases of severe injuries and/or deaths at the work place (for Occupational safety and labor inspectorates) • Number of outbreaks of food-borne diseases (for Food safety)

15. Labor inspectorate of Estonia publishes comparison of indicators with other countries

16. CANNOT HIDE - FATAL CAN HIDE

17. Less of the burden – Kyrgyzstan experience

18. In the initial stages of the reform +/- approach • If sophisticated data collection is not available. • And there is a need for pressure on the inspectorates • opt for +/- approach

19. Assessing teams and individual inspector If done correctly the process of assessment is as influential as the measures taken

20. Additional potential challenges to M&E in low-income and low-capacity countries = Similar challenges as in more advanced economies • Poor data (including information sources) • Measuring the extent of the informal economy • “Time and cost savings” is not really the issue • Degree of “compliance” is not really the issue • No practices (like risk-based assessment) to establish benchmark • Measuring corruption • … +

21. So innovative thinking is needed… Survey of outbound airplane tourists from Nepal: How well is the government’s - GREEN - AMBER - RED sticker scheme working?

22. How can the M&E system be improved? • EFFECTIVENESS • Evidence Based Enforcement • Coordination of REI agencies • Information integration • Compliance promotion • Safety and performance indicators • EFFICIENCY • Selectivity • Risk focus and proportionality • Responsive regulation • Coordination and consolidation • Burden and costs • TRANSPARENCY AND GOVERNANCE • Long term vision • Transparent governance • Clear and fair process • Professionalism • Perceptions and stakeholders' relations • “Inspections Scorecard” based on OECD principles, focus on quality of the system • Allows experts to rate a system on a number of different indicators, qualitative and quantitative

23. How can the M&E system be improved? • In addition to assessing the quality of the system, enlarge the scope of measuring regulatory costs and burden for businesses, identify other indicators • Sectoralapproach • Use causal pathways • … Source: UK’s LBRO/BRDO

24. In closing