Some Elements of Guidance for the Design and Implementation of PETS/QSDS

1. Some Elements of Guidance for the Design and Implementation of PETS/QSDS Bernard Gauthier World Bank and HEC Montréal Following the Money: Participatory PETS and Absenteeism Methods for Civil Society Kampala, Uganda July 29 2009

2. Contents I.Introduction II. Design and Implementation Steps • Preparation: consultation, time frame, budget, team composition • Scope of the study, research questions • Institutional analysis • Choice of tracking flows • Rapid data assessment • Sampling strategies • Instrument design • Survey pre-pilot • Training • Survey pilot • Survey Implementation • Data entry, cleaning • Report/analysis • Results, dissemination and follow up III. Conclusion

3. I. Introduction • In the last decade, PETS and QSDS have proven to be powerful tools for identifying bottlenecks, inefficiencies and waste in service delivery. • They have been successfully used in some cases to promote reforms leading to significant improvements in resource allocation (e.g. education sector in Uganda). • However, PETS/QSDS are resource intensive tools and to be efficient instruments to diagnose and analyze service delivery, they need to be well designed and implemented, and results well disseminated to contribute to better accountability, transparency and ultimately improved outcomes. • We examine steps that need to be taken to design and implement PETS/QSDS studies.

4. II. Design and Implementation Steps • Preparation • Time frame and budget: Sufficient time and resources are necessary to plan, design and implement a survey - Adequate time should be further allotted for findings dissemination and policy reform discussions. • Team composition: skilled technical expertise in budget execution and sector-specific knowledge; team prior experience in surveys • Consultation: central and local governments, sector cooperation, various stakeholders, civil society and local community

5. Design and Implementation Steps 2. Scope of the study, research questions • Identify important issues relating to efficiency and equity problems in service delivery system • resource leakage, delays, absenteeism, inefficiencies, inequities in the actual resource allocation, quality of services, etc. • Determine whether these issues are amenable to survey work. • Overall scope of study: case study, geographical coverage, facility ownership types (public, private, not for profit) • General research objectives should be translated into specific objectives and questions that will drive the data collection strategy. • Hypotheses should be formulated in order to determine what kind of data should be collected.

6. Design and Implementation Steps 3. Analysis of institutional arrangements and budget execution processes • Identify the structure, roles and responsibilities of various administrative units in the supply chain and budget execution processes • All sources of potential transfers to frontline facilities • allocation rules used by different resources at the various levels, and the nature of information flows (including accounting, reporting and monitoring procedures). • Official rules versus in practice • Understanding the environment of public providers • Types of schools or heath facilities operating in the region, the mixture of public, private, religious and community facilities. • In itself, institutional analysis allows better understanding of service delivery system, favoring local community participation

7. Design and Implementation Steps 4. Choice of tracking flows • Any tracking survey requires determination of the specific flows on which financial and quantitative information will be collected and at which administrative levels. • In each of the various branches or resource flows of the allocation procedure, there are possibilities of leakage: • Funding, supplies, drugs, equipment or materials could leak or be stolen through the procurement process at various levels in the service provision supply chain. • Similarly, salary expenditures could leak through the creation of fictitious (ghost) workers, for instance. • However, not all flows are amenable to tracking. • Nonexistent records or data inconsistencies make certain flows untraceable or data non informative. • Also, the complexity of tracking whole categories of expenditures requires PETS to restrict the tracking exercise on a subset of the service provider environment

8. Choice of tracking flow • Common trap: Too wide coverage • Several surveys have attempted to gather information on line ministries entire recurrent expenditures. • Trade-off between wide coverage and feasibility • Restriction of the tracking domain: To gather quantitative information on a line ministry entire budget is very hazardous. • Surveys that attempt to track entire sector flows run the risk of not being able to collect consistent and high quality data • Given data limitations, it is often indicated to focus on specific funding or flows for which records or accounts of good enough quality on at least two levels could be identified.

9. Choice of tracking flow • What should determine the choice of flows or program to track in the supply chain? • Several factors are at play. In some cases, the choice set can be determined directly from the research question or survey objectives, which, in certain cases, could call for a specific flow to be tracked. • For instance, if the objective of the survey is to identify the availability of specific basic material (such as school books or medication), then the focus of the tracking exercise could naturally be limited to these specific items • If the purpose of the survey is to evaluate the importance of ghost workers, then the domain of financial flows to track could potentially be restricted to salary flows.

10. Choice of tracking flow Cases of 3 successful tracking A. 1996 Uganda PETS in education • The initial intentions were to track all public spending in education through the entire delivery system. • However, a pilot survey revealed important data availability problems: - Was discovered that at both the central government and district levels, official records (for both wage and non-wage expenditures) were very poor, if not simply non-existent. Quality of information at the district level in particular, both on transfers from the MOE and disbursement to schools, was very poor. • Decision was made to exclude the district level from the tracking exercise and to limit data collection to the central government and service provider levels

11. Choice of tracking flow • Furthermore, the pilot survey revealed that at the central government level, data were not available on salaries paid to primary school teachers, either by districts or schools. • The only systematic information on primary education found to be available and of good quality at the central level was capitation grants for non-wage spending. • Fortunately, financial records were also relatively comprehensive at the school level,

12. Choice of tracking flow • The tracking exercise then ultimately focused on a single flow: capitation grant. • The survey objective thus became: to determine how much of the capitation grant allocated by the MOE actually reached primary schools. • Data (financial and in-kind) were collected at the school level on the reception of this single program. • To estimate leakage (between the entitled budgetary allocations and actual reception at the school level), the only other information required was enrolment data at the school level, which was also available.

13. Choice of tracking flow B. Zambia PETS/QSDS education 2002. • A very thorough analysis of the education sector and in-depth analysis of the administrative process was done (Das et al 2004). • Allowed identifying characteristics of resource flows in the public education system. • Six main flows were identified and categorized in terms of sources, types and administrative levels’ discretionary power in fund allocation (See Table)

14. Source Gauthier 2006, p.62

15. Choice of tracking flow • Choice was made to track non-wage financial flows from the MOE and donors at the provincial, district and facility levels. • In-kind transfers, as well as salary transfers, were excluded from the tracking exercise. • The specific objective of the tracking exercise then became to determine whether: a) Schools received the fixed-rule component of the MOH budget (lump-sum payment per school); b) Provinces and districts supported schools further through discretionary expenditures c) Decentralization had an effect on fund allocation behavior

16. Choice of tracking flow Lessons • Having restricted the tracking exercise to cash flows has facilitated data collection in Zambia, as good quality financial records were available. • Very thorough field work of the survey team, in particular at the pre-survey and instrument design stages, was key factor in allowing a clear understanding of the institutional ramification of the education sector and the crafting of a very sound survey methodology.

17. Choice of tracking flow Ghana 2007 Education • The PETS survey was designed to capture quantitatively the resource flows (materials or financial) at each administrative office where the public expenditures are handled • Four expenditures were tracked: • Text books and Stationary • Capitation Grants • Investment Expenditures in Basic Education. • Service Activities (Item 3 Expenditures)

18. Choice of tracking flow • Important tool used: flow charts (see Figures) (and the corresponding questionnaires). • Thorough institutional analysis was realized • Note that expenditures go through different channels before reaching schools, even for the same expenditures. • Ex: while the schools books are distributed by District Education Office (DEO) to basic schools, the text books are directly distributed by the Ghana Education Service (GES) and the Ghana Education Trust Fund (GETFund) to the second cycle schools, respectively.

19. Example of Tracking Flows Ghana Education 2007

22. Choice of tracking flow Cases of 2 lesssuccessfulsurveys A. Uganda 1996 PETS in Health • Contrary to the education sector where, after a pilot survey, the focus was specifically restricted to a particular program (capitation grants), in the health sector the focus remained on all public resources and administrative levels. • However, lack of almost any financial information at the facility level and the heavy reliance on in-kind measures were not anticipated at design and pilot stage of the survey. • Consistent quantitative data could not be collected. No systematic facility level information on financing, inputs or outputs were found. • Ultimately, survey did not produce any reliable quantitative measures of expenditure flows or leakage assessment.

23. Choice of tracking flow B. Mozambique PETS/QSDS health 2002 • Suffered from important data problems. • Primary health care facilities in Mozambique are not allocated funding from higher administrative levels, but only receive in-kind transfers: complicates tracking. • Could have chosen between focusing: i) on specific resource flow or program in order to collect detailed data to permit reliable assessments of leakage ii) take a broader focus and attempt to measure most health expenditures, but incur the risk that the data would not allow firm conclusions on leakage.

24. Choice of tracking flow • The second option was chosen: tried to track all non-wage recurrent expenditures, drugs and other supplies, and human resources, at three levels: provinces, districts and facilities. • However, data quality turned out to be a serious concern at both the provincial and district levels. • Large gaps in information were observed in about 75% of the districts between District Health Offices’ financial information records and those provided by the Provincial Health Offices. • Furthermore, complete district level financial data could be collected for only about 40% of districts. Similar discrepancies between provincial and district records were also found in the case of medication transfers and health worker data.

25. Choice of tracking flow • Lesson: Successful past surveys (e.g. Uganda and Zambia) have restricted the tracking domain on flows for which good quality and consistent data were available. • Another lesson from the first PETS: flow tracked does not have to be a dominant share in the service provider budget to convey important results. • Capitation grants in Uganda represented on average only about 6% of total government transfers to schools during the period under study (1991-95) • Furthermore, the bypassing of some levels are sometimes indicated given the limited and potentially inconsistent information available at these levels: Ex: Mali, Mozambique • Data quality assessment needs to be accomplished before the launching of the full scale survey: during a data and institutional assessment phase

26. Design and Implementation Steps 5. Rapid data assessment • Following the identification of the data required to analyze service delivery performance, a rapid data assessment should be performed • Data problems are frequent (availability, quality, consistency) • The objective is to verify that the data required to test the hypothesis are available, and if not, to adapt the empirical strategy to available data. • Seek to determine the availability and quality of records at various levels (as well as among various types of providers), and avoid risk of gathering inconsistent data. • This small scale study should determine the survey’s feasibility and usefulness.

27. Rapid data assessment • A simple questionnaire administered at various administrative levels is usually sufficient for such a purpose • For instance, if information is of poor quality at the local government level (region or district), this level could be bypassed and information could be collected only at the facility level and at the central ministry level (in order to know how much was officially sent). • This rapid assessment of data availability should lead to redefinition of research questions and to the final choice of tracking flows in light of the available data.

28. Design and Implementation Steps 6. Sampling strategy • Fundamental for tracking survey quality • Standard PETS/QSDS: should always be representative of the population under study. • The sample has to be randomly selected at every stage (regions, districts, facilities). • Small scale PETS/QSDS: could also be realized as a case study, in a few districts or regions • If results are required for different categories of facilities (ownership types, location, etc.), then the sample needs to be explicitly stratified.

29. Sampling strategy • The sample size of each category must be sufficiently large to yield reliable statistical results. • Sample frame based on list of facilities • An example of provincial sample: • in Congo a Health PETS/QSDS to be implemented has randomly selected 100 public health facilities from a list ordered geographically and stratified by urban and rural areas. • This will provide province-representative estimates • Could also seek to have a district representative sample etc.

30. Sampling strategy Allocation rules and sampling strategy • The sampling strategy is conditioned by the tracking flow selected and the allocation rule in the sector • Case A: If the flows tracked have fixed allocation rules (e.g. capitation grant/financing per student, number of books by school or by students) then a random sampling strategy is in order. • A random sample of schools provides a representative picture of the situation in the country in the context of fixed rule allocation.

31. Sampling strategy • If the resource tracked is allocated through a rule-based formula (fixed or hard allocation rule) such as in Uganda for capitation grant (i.e. specific amount per student), then the measurement of leakage is done using the following standard formula: • EX: In Uganda, in presence of capitation grant, to measure leakage one needs only to compare how much the school is entitled to based on the number of students using the government formula and compare it with amount received by the school. • The leakage rate is then defined as the ratio between how much the facility actually received and how much it should have received.

32. Sampling strategy • Case B: However, if the resources tracked are allocated without fixed rules (that is through discretion of officials at the various levels), then the tracking exercised is more challenging and the sampling strategy is affected. • The following leakage formula applies: • In the absence of fixed-rule, leakage rate is then defined as the ratio between how much the facility actually received and how much the central level (or other hierarchical level) has sent to the facility. • Need to collect data not only on the resources sent by central, regional and district levels but on all resources received by the providers from the various levels. • Data has to be collected on the amount of financial transfers and value of all resources sent to a region (unit) during the fiscal year.

33. Sampling strategy • With respect to the facility sampling strategy, in the context of administrative discretion over resource allocation, all facilities should be surveyed on a censusbasis within the area of analysis chosen (e.g. district), in order to collect complete information on the reception of resources. • A random sample of facilities would not allow measuring the leakage level of resources but only the probability of receiving public resources at the provider level, a concept different then leakage. • Indeed, with only a few facilities visited per district in the sampling strategy, it is not possible to say anything about resource use in a specific district for instance, in terms of reception of materials, financing, drugs, user fees, etc relative to other districts (or aggregated at the provincial level).

34. Sampling strategy Material resources tracking • Valuation of in-kind material received at the frontline could be made difficult because of sheer number of such resources potentially received by facilities during a year. In several countries, in kind items constitute the only source of transfers (other than staff) to frontline facilities. • Except if complete electronic records are available from the higher levels and facilities, tracking should be based on a sample of in-kind items and equipments. • To identify the items to be sampled, records at the central government depot and value of invoices should be used. • The list of the main invoices could generally be obtained from the division of material resources from line Ministry

35. Sampling strategy • Based on that list of material, for instance, the 5 most frequent materials received at the facility level (based on their shipment frequency) in the line ministry invoice list should be identified. • Note that if on the contrary one was choosing a rare but high-value material (e.g. a car), this would introduce the risk of not finding that material in the visited facilities simply because not all of them were able to receive it. • In contrast, by choosing frequently-shipped materials of small value (e.g. books or soap), it is likely that a maximum number of facilities would report receiving them. • This strategy hence provides upwardly biased percentage of facilities receiving materials from the authorities.

36. Sampling strategy • For instance in Chad, data on financial transfers were available from the national budget (final). • For the material, data from the Direction administrative was collected for 2003. • All the slips of invoices were available in a single binder with dates of shipments, the recipient of the material, list of material sent with quantities and prices, signature of recipient and comments. • All records were photocopied for further analysis. • To measure the material received at the various administrative level, one strategy could have been to photocopy all forms found. • In Chad for instance, this was not feasible given that there were no photocopy machines in especially in rural areas. Information was obtained through structured questionnaires.

37. Sampling strategy • Given the very large number of various items sent to lower administrative levels, the arrival of material in Chad was done on a sample basis. • Focus was on 8 frequently sent items, to maximize the probability of finding the item and that the data could be triangulated. • The region questionnaire asks what was the quantity of a specific item sent to districts and facilities in the sample. • Then at the district level, the questionnaire asked how much received from the region and how many sent to the facilities. • Finally the facility questionnaire asks how many of these items were received. • All numbers have to come from the records.

38. Design and Implementation Steps 7- Instrument design • PETS and QSDS are generally composed of various survey instruments intended to collect information at the different organizational levels and among stakeholders involved in service delivery, on both the supply and demand sides. • The design of survey instruments depends on survey objectives and the choice of tracking flows. • Past surveys have typically included modules on the following units (Table 3): • Central government’s Ministry of Finance • Central government’s line ministry (e.g. Health or Education) • Provincial (or regional) administration • District (or local) administration • Service provider (e.g. school or health centers) • Staff • Clients (e.g. patients or students) • In some cases, households information has also been collected • Data collected have covered information on seven main categories of variables (Table 4):

41. Instrument design Records vs recall: For quantitative data, records should be used as possible to minimize measurement errors When no other sources of data are available and recall data are collected, clear indications in that respect should be reported. Length of data tracking: Data collection should ideally involve annual data and cover a period of a maximum of two financial years to maximize data collection quality. If monthly data are collected, seasonality issues need to be examined beforehand. Recording procedures: survey instruments should be adapted to the specific recording procedures in practice in the administrative units and facilities under study. Parsimony of data collected is recommended to reduce costs, but also to increase the quality of data collected. The questionnaire should be focused, and contain a reduced number of questions.

42. Instrument design Valuation of in-kind items received • Sometimes, the official price of in-kind transfers is not known by frontline facility officials. In such case, valuation at market price is required. • Note that even when official prices are available, prices could be inflated by the recipient or the supplier. • It is thus recommended to verify the prices with market prices for equivalent goods. • Such verification was done for instance in Mali part of an education PETS in 2005. Official prices were compared relative to market prices for the same items and important overvaluation was discovered. • The Madagascar 2003 survey also compared official prices of in-kind items relative to market prices.

43. Instrument design Exemples of instruments Zambia PETS/QSDS (9 questionnaires) Chad PETS/QSDS (10 questionnaires) Uganda QSDS (3 questionnaires) Instruments available at: http://econ.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTRESEARCH/EXTPROGRAMS/EXTPUBSERV/0,,contentMDK:20292627~menuPK:545282~pagePK:64168182~piPK:64168060~theSitePK:477916,00.html#PETS Could be useful to design PETS in the perspective of a baseline for benchmarking for the purpose of monitoring and evaluate future intervention in the sector. In such case, indicators are required. Various indicators could be built. Table 5 provides some potential PETS/QSDS indicators

44. Table 5: Potential PETS/QSDS Indicators

45. Table 5: Potential PETS/QSDS Indicators (Continued)

46. Table 5: Potential PETS/QSDS Indicators (Continued)

47. Design and Implementation Steps 8- Preliminary pilot phase: • All survey instruments should be tested through a small scale pilot phase on a specific number of units and ownership types or regions. • The choice of tracking flows should be assessed, as well as the quality and consistency of data. • In particular, quantitative data questions, financial data, inputs and outputs, which have been customized to the sector’s administrative system, should be carefully examined. • Questions’ wording, ambiguous responses, answer codes, etc. should be revised at this stage.

48. Design and Implementation Steps 9- Training • Adequate time for training • A minimum of one week (and possibly two) are required for enumerator and supervisors training. • The training by the survey core team involves acquainting enumerators and supervisors with the instruments and techniques used in data collection. • Training should include the testing of instruments by agents in the filed. • Enumerators or supervisors should have access to a survey manual detailing questions’ objectives and interpretation. • Following the training, questionnaires may be revised.

49. Design and Implementation Steps 10- Full pilot phase • Following (or during) the training, all questionnaires should be tested. • Field testing of instruments is crucial for increasing the likelihood of obtaining good quality survey information. • A test on about 5% of the sample including all types of respondents should be adequate. • Following the pilot, a final revision of instruments should be carried out.

50. Design and Implementation Steps 11- Survey implementation • Field work must be closely monitored • Core team provides technical assistance and monitors the implementation of the survey • Random visits to enumerators to ensure quality control and coherence in the interpretation of questionnaires. • Random checks of questionnaires and data quality during the survey implementation. • Survey Timing: In the pre-design phase, an essential decision is to determine when to field the survey. • The fielding of a survey should ideally be done two or three months after the end of a fiscal year, in order for accounting books to be closed. • In any respect, the tracking should always be done on the preceding fiscal year, never on the current one.