The challenges of obtaining reliable data

1. The challenges of obtaining reliable data World Bank Group Investment Climate Advisory Services Measuring reform success: business surveys, "client satisfaction" and administrative burden Florentin BLANCVilnius, 22 March 2012

2. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK The need for measurement – knowing where you stand, and where you are going • Reforms are undertaken – but do the regulated/inspected businesses feel them? Do they see inspectors as a positive factor, or not? Do they complain about burden, lack of transparency etc. – or less? • Essential to gather positive results in dealing with the regulated “clients” – while also maintaining or improving outcomes for the public – so the two sides should get measured • Crucial for the Government as a whole – which has to show improvements to the citizens – and for the inspecting agencies, which have to report to the Government, and need success in improving effectiveness and efficiency in a difficult economic / fiscal context

3. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Range of instruments – all with specific upsides and downsides • Business surveys: ask all, or a sample of, inspected businesses how their experience with inspectorate(s) was • Focus-group discussions: gather several businesses with a certain profile and ask about experiences with range of structured questions • Both of these methods can be implemented directly by inspectorate or (preferred) through a specialized company or researchers • Open consultations with stakeholders (e.g. business associations)

4. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Focus of our measurement efforts – not just “what” but “how well”… • Relatively easy to find out from businesses whether something took place or not (“Were you inspected?” – “Did you receive advice?”) • More difficult to get reliable answers on questions of quality – which are inherently more subjective (and thus where answers can vary…) • Specific focus in our case is to try and find out “satisfaction” with “quality of consultations” • If possible, we should try and measure the results rather than subjectively expressed satisfaction

5. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Business consultations – what are we trying to achieve • Immediate aim of consultations is to help business comply: knowing and understanding what they have to do means less violations • Longer-term goal is to facilitate business creation, growth and diversification by providing more certainty for investment (if one operates within parameters explained in consultation, there is no risk of the business being closed down) • Using indicators such as “compliance levels” and “business growth” to measure success of consultations is nonetheless problematic because they can be impacted by many other influences…

6. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK “Client-satisfaction surveys” – an instrument that can be misleading • Frequent approach: measure reform success by asking businesses whether they see improvements • Several risks with such surveys: • Insincere answers – whenever the business fears about confidentiality (e.g. survey given just after inspection…) • Subjective answers – correlated with external events or “latest impressions” and not reflecting actual “average situation” with inspections • May over-estimate “success” – or hide it… • Such “subjective” questions are better asked by independent researchers, in focus-groups or in-depth interviews – where the reasons for the answers can be asked and discussed

7. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK An example of qualitative, independent investigation of business perceptions • From the Business End of the Telescope • Commissioned by UK LBRO • Focus-groups, in-depth interviews (with individual businesses, business associations etc.) to understand how businesses perceive inspections, inspectors • Gives real insights on negatives and positives, what businesses resent and what they appreciate, expect etc. • Trust in independence of interviewers – professional competence to investigate

8. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Designing surveys – asking the “right” questions • First crucial step: ensure trust of interviewees in independence of survey, confidentiality • Ask questions that will give more reliable answers: • Objective quantitative indicators: how often, how long, etc. – allowing to track inspections patterns • Qualitative but not “like” or “feel” questions: • Did you get advice that was clear? • Was the inspector proficient in technical issues? • Develop questions that are clear, precise, yield meaningful answers – questionnaire testing is needed

9. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Examples of “quantitative surveys” • World Bank Group has for 10+ years been supporting surveys of inspections in countries where it assists with reform work • Provide clear picture on how many businesses inspected, how often, how long etc. – aggregated across country – by sector – by agency etc. • Of course margin of error – and some answers may be wrong (misunderstandings, bad memory etc.) – but mostly this gets “averaged out” by large numbers of respondents • Very useful information because (among other issues): • Consolidated data was simply missing – Government, even inspectorates, often did not know situation • Discrepancy between practices and “official numbers”

10. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK The cost challenge – and the different technical solutions • Business surveys have a cost – particularly if you don’t use the “questionnaire filled and collected at the end of the inspection” approach… • Phone, mail, internet surveys have different costs – but also different levels of reliability • Probably makes sense to combine “qualitative” instruments conducted by independent researchers, and “quantitative” ones – resources can be pooled by inspectorates, no need to always have a separate study for each • Key is to properly conceive what we want to know: what are the important questions, and the target groups (all businesses, some specific sectors, sizes etc.?)

11. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Trust and credibility • Businesses (and citizens – as some surveys could target inspected subjects that are not entrepreneurial ones) need to have full confidence that survey results will not harm them – so not only individual confidentiality, but even “group” confidentiality / results should not be traceable • “Survey fatigue” will soon kick in if there are too many studies, and if they are not seen as having consequences • Essential for Government, inspecting agencies to show that surveys / studies results are being acted upon and used to design / drive further changes • Such link to policy actions can also be advertised before a new survey / study starts – to increase response rates / quality

12. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Some recommendations (1) – combine approaches • Business surveys can be used for quantitative data (“did you receive advice?”, “were your questions answered?”) – particularly as long as most inspectorates do not have an information system to consolidate data on their activities • Using (trusted) independent researchers to conduct qualitative research through focus-groups etc. is likely to provide more insights into satisfaction and remaining problems • Direct discussions with business representatives are also essential to increase trust between the two sides • Finally, trying to measure and track outcomes would be a very important step…

13. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Some recommendations (2) – measure outcomes • Far more difficult because of “attribution” issues and of measurement quality issues (for compliance – since it is measured through inspections => conflict of interest…) • Areas of focus should be determined depending on baseline – if businesses particularly complain in some areas, these are the ones where to check if changes have taken place • More business creations or expansions, more jobs, in a sector that had problems, can be indicators that consultations (and other changes) are working • Also very important: through surveys (possibly internet) or independent research, try and identify why some were discouraged from creating or expanding business activity – this may show the remaining gaps

14. World Bank GroupMultilateral Investment Guarantee Agency THE WORLD BANK Thanks for your attention fblanc@ifc.org www.wbginvestmentclimate.org