Some Elements of Guidance for the Design and Implementation of PETS/QSDS Bernard Gauthier World Bank and HEC Montr�al Following the Money: Participatory PETS and Absenteeism Methods for Civil Society Kampala, Uganda July 29 2009
Slide 2:Contents
I.Introduction II. Design and Implementation Steps Preparation: consultation, time frame, budget, team composition Scope of the study, research questions Institutional analysis Choice of tracking flows Rapid data assessment Sampling strategies Instrument design Survey pre-pilot Training Survey pilot Survey Implementation Data entry, cleaning Report/analysis Results, dissemination and follow up III. Conclusion
Slide 3:I. Introduction
In the last decade, PETS and QSDS have proven to be powerful tools for identifying bottlenecks, inefficiencies and waste in service delivery. They have been successfully used in some cases to promote reforms leading to significant improvements in resource allocation (e.g. education sector in Uganda). However, PETS/QSDS are resource intensive tools and to be efficient instruments to diagnose and analyze service delivery, they need to be well designed and implemented, and results well disseminated to contribute to better accountability, transparency and ultimately improved outcomes. We examine steps that need to be taken to design and implement PETS/QSDS studies.
Slide 4:II. Design and Implementation Steps
Preparation Time frame and budget: Sufficient time and resources are necessary to plan, design and implement a survey - Adequate time should be further allotted for findings dissemination and policy reform discussions. Team composition: skilled technical expertise in budget execution and sector-specific knowledge; team prior experience in surveys Consultation: central and local governments, sector cooperation, various stakeholders, civil society and local community
Slide 5:Design and Implementation Steps
2. Scope of the study, research questions Identify important issues relating to efficiency and equity problems in service delivery system resource leakage, delays, absenteeism, inefficiencies, inequities in the actual resource allocation, quality of services, etc. Determine whether these issues are amenable to survey work. Overall scope of study: case study, geographical coverage, facility ownership types (public, private, not for profit) General research objectives should be translated into specific objectives and questions that will drive the data collection strategy. Hypotheses should be formulated in order to determine what kind of data should be collected.
3. Analysis of institutional arrangements and budget execution processes Identify the structure, roles and responsibilities of various administrative units in the supply chain and budget execution processes All sources of potential transfers to frontline facilities allocation rules used by different resources at the various levels, and the nature of information flows (including accounting, reporting and monitoring procedures). Official rules versus in practice Understanding the environment of public providers Types of schools or heath facilities operating in the region, the mixture of public, private, religious and community facilities. In itself, institutional analysis allows better understanding of service delivery system, favoring local community participation
4. Choice of tracking flows Any tracking survey requires determination of the specific flows on which financial and quantitative information will be collected and at which administrative levels. In each of the various branches or resource flows of the allocation procedure, there are possibilities of leakage: Funding, supplies, drugs, equipment or materials could leak or be stolen through the procurement process at various levels in the service provision supply chain. Similarly, salary expenditures could leak through the creation of fictitious (ghost) workers, for instance. However, not all flows are amenable to tracking. Nonexistent records or data inconsistencies make certain flows untraceable or data non informative. Also, the complexity of tracking whole categories of expenditures requires PETS to restrict the tracking exercise on a subset of the service provider environment
Slide 8:Choice of tracking flow
Common trap: Too wide coverage Several surveys have attempted to gather information on line ministries entire recurrent expenditures. Trade-off between wide coverage and feasibility Restriction of the tracking domain: To gather quantitative information on a line ministry entire budget is very hazardous. Surveys that attempt to track entire sector flows run the risk of not being able to collect consistent and high quality data Given data limitations, it is often indicated to focus on specific funding or flows for which records or accounts of good enough quality on at least two levels could be identified.
What should determine the choice of flows or program to track in the supply chain? Several factors are at play. In some cases, the choice set can be determined directly from the research question or survey objectives, which, in certain cases, could call for a specific flow to be tracked. For instance, if the objective of the survey is to identify the availability of specific basic material (such as school books or medication), then the focus of the tracking exercise could naturally be limited to these specific items If the purpose of the survey is to evaluate the importance of ghost workers, then the domain of financial flows to track could potentially be restricted to salary flows.
Cases of 3 successful tracking A. 1996 Uganda PETS in education The initial intentions were to track all public spending in education through the entire delivery system. However, a pilot survey revealed important data availability problems: - Was discovered that at both the central government and district levels, official records (for both wage and non-wage expenditures) were very poor, if not simply non-existent. Quality of information at the district level in particular, both on transfers from the MOE and disbursement to schools, was very poor. Decision was made to exclude the district level from the tracking exercise and to limit data collection to the central government and service provider levels
Furthermore, the pilot survey revealed that at the central government level, data were not available on salaries paid to primary school teachers, either by districts or schools. The only systematic information on primary education found to be available and of good quality at the central level was capitation grants for non-wage spending. Fortunately, financial records were also relatively comprehensive at the school level,
The tracking exercise then ultimately focused on a single flow: capitation grant. The survey objective thus became: to determine how much of the capitation grant allocated by the MOE actually reached primary schools. Data (financial and in-kind) were collected at the school level on the reception of this single program. To estimate leakage (between the entitled budgetary allocations and actual reception at the school level), the only other information required was enrolment data at the school level, which was also available.
B. Zambia PETS/QSDS education 2002. A very thorough analysis of the education sector and in-depth analysis of the administrative process was done (Das et al 2004). Allowed identifying characteristics of resource flows in the public education system. Six main flows were identified and categorized in terms of sources, types and administrative levels� discretionary power in fund allocation (See Table)
Source Gauthier 2006, p.62
Choice was made to track non-wage financial flows from the MOE and donors at the provincial, district and facility levels. In-kind transfers, as well as salary transfers, were excluded from the tracking exercise. The specific objective of the tracking exercise then became to determine whether: a) Schools received the fixed-rule component of the MOH budget (lump-sum payment per school); b) Provinces and districts supported schools further through discretionary expenditures c) Decentralization had an effect on fund allocation behavior
Lessons Having restricted the tracking exercise to cash flows has facilitated data collection in Zambia, as good quality financial records were available. Very thorough field work of the survey team, in particular at the pre-survey and instrument design stages, was key factor in allowing a clear understanding of the institutional ramification of the education sector and the crafting of a very sound survey methodology.
Ghana 2007 Education The PETS survey was designed to capture quantitatively the resource flows (materials or financial) at each administrative office where the public expenditures are handled Four expenditures were tracked: Text books and Stationary Capitation Grants Investment Expenditures in Basic Education. Service Activities (Item 3 Expenditures)
Important tool used: flow charts (see Figures) (and the corresponding questionnaires). Thorough institutional analysis was realized Note that expenditures go through different channels before reaching schools, even for the same expenditures. Ex: while the schools books are distributed by District Education Office (DEO) to basic schools, the text books are directly distributed by the Ghana Education Service (GES) and the Ghana Education Trust Fund (GETFund) to the second cycle schools, respectively.
Example of Tracking Flows Ghana Education 2007
Cases of 2 less successful surveys A. Uganda 1996 PETS in Health Contrary to the education sector where, after a pilot survey, the focus was specifically restricted to a particular program (capitation grants), in the health sector the focus remained on all public resources and administrative levels. However, lack of almost any financial information at the facility level and the heavy reliance on in-kind measures were not anticipated at design and pilot stage of the survey. Consistent quantitative data could not be collected. No systematic facility level information on financing, inputs or outputs were found. Ultimately, survey did not produce any reliable quantitative measures of expenditure flows or leakage assessment.
B. Mozambique PETS/QSDS health 2002 Suffered from important data problems. Primary health care facilities in Mozambique are not allocated funding from higher administrative levels, but only receive in-kind transfers: complicates tracking. Could have chosen between focusing: i) on specific resource flow or program in order to collect detailed data to permit reliable assessments of leakage ii) take a broader focus and attempt to measure most health expenditures, but incur the risk that the data would not allow firm conclusions on leakage.
The second option was chosen: tried to track all non-wage recurrent expenditures, drugs and other supplies, and human resources, at three levels: provinces, districts and facilities. However, data quality turned out to be a serious concern at both the provincial and district levels. Large gaps in information were observed in about 75% of the districts between District Health Offices� financial information records and those provided by the Provincial Health Offices. Furthermore, complete district level financial data could be collected for only about 40% of districts. Similar discrepancies between provincial and district records were also found in the case of medication transfers and health worker data.
Lesson: Successful past surveys (e.g. Uganda and Zambia) have restricted the tracking domain on flows for which good quality and consistent data were available. Another lesson from the first PETS: flow tracked does not have to be a dominant share in the service provider budget to convey important results. Capitation grants in Uganda represented on average only about 6% of total government transfers to schools during the period under study (1991-95) Furthermore, the bypassing of some levels are sometimes indicated given the limited and potentially inconsistent information available at these levels: Ex: Mali, Mozambique Data quality assessment needs to be accomplished before the launching of the full scale survey: during a data and institutional assessment phase
5. Rapid data assessment Following the identification of the data required to analyze service delivery performance, a rapid data assessment should be performed Data problems are frequent (availability, quality, consistency) The objective is to verify that the data required to test the hypothesis are available, and if not, to adapt the empirical strategy to available data. Seek to determine the availability and quality of records at various levels (as well as among various types of providers), and avoid risk of gathering inconsistent data. This small scale study should determine the survey�s feasibility and usefulness.
Slide 27:Rapid data assessment
A simple questionnaire administered at various administrative levels is usually sufficient for such a purpose For instance, if information is of poor quality at the local government level (region or district), this level could be bypassed and information could be collected only at the facility level and at the central ministry level (in order to know how much was officially sent). This rapid assessment of data availability should lead to redefinition of research questions and to the final choice of tracking flows in light of the available data.
6. Sampling strategy Fundamental for tracking survey quality Standard PETS/QSDS: should always be representative of the population under study. The sample has to be randomly selected at every stage (regions, districts, facilities). Small scale PETS/QSDS: could also be realized as a case study, in a few districts or regions If results are required for different categories of facilities (ownership types, location, etc.), then the sample needs to be explicitly stratified.
Slide 29:Sampling strategy
The sample size of each category must be sufficiently large to yield reliable statistical results. Sample frame based on list of facilities An example of provincial sample: in Congo a Health PETS/QSDS to be implemented has randomly selected 100 public health facilities from a list ordered geographically and stratified by urban and rural areas. This will provide province-representative estimates Could also seek to have a district representative sample etc.
Allocation rules and sampling strategy The sampling strategy is conditioned by the tracking flow selected and the allocation rule in the sector Case A: If the flows tracked have fixed allocation rules (e.g. capitation grant/financing per student, number of books by school or by students) then a random sampling strategy is in order. A random sample of schools provides a representative picture of the situation in the country in the context of fixed rule allocation.
If the resource tracked is allocated through a rule-based formula (fixed or hard allocation rule) such as in Uganda for capitation grant (i.e. specific amount per student), then the measurement of leakage is done using the following standard formula: EX: In Uganda, in presence of capitation grant, to measure leakage one needs only to compare how much the school is entitled to based on the number of students using the government formula and compare it with amount received by the school. The leakage rate is then defined as the ratio between how much the facility actually received and how much it should have received.
Case B: However, if the resources tracked are allocated without fixed rules (that is through discretion of officials at the various levels), then the tracking exercised is more challenging and the sampling strategy is affected. The following leakage formula applies: In the absence of fixed-rule, leakage rate is then defined as the ratio between how much the facility actually received and how much the central level (or other hierarchical level) has sent to the facility. Need to collect data not only on the resources sent by central, regional and district levels but on all resources received by the providers from the various levels. Data has to be collected on the amount of financial transfers and value of all resources sent to a region (unit) during the fiscal year.
With respect to the facility sampling strategy, in the context of administrative discretion over resource allocation, all facilities should be surveyed on a census basis within the area of analysis chosen (e.g. district), in order to collect complete information on the reception of resources. A random sample of facilities would not allow measuring the leakage level of resources but only the probability of receiving public resources at the provider level, a concept different then leakage. Indeed, with only a few facilities visited per district in the sampling strategy, it is not possible to say anything about resource use in a specific district for instance, in terms of reception of materials, financing, drugs, user fees, etc relative to other districts (or aggregated at the provincial level).
Material resources tracking Valuation of in-kind material received at the frontline could be made difficult because of sheer number of such resources potentially received by facilities during a year. In several countries, in kind items constitute the only source of transfers (other than staff) to frontline facilities. Except if complete electronic records are available from the higher levels and facilities, tracking should be based on a sample of in-kind items and equipments. To identify the items to be sampled, records at the central government depot and value of invoices should be used. The list of the main invoices could generally be obtained from the division of material resources from line Ministry
Based on that list of material, for instance, the 5 most frequent materials received at the facility level (based on their shipment frequency) in the line ministry invoice list should be identified. Note that if on the contrary one was choosing a rare but high-value material (e.g. a car), this would introduce the risk of not finding that material in the visited facilities simply because not all of them were able to receive it. In contrast, by choosing frequently-shipped materials of small value (e.g. books or soap), it is likely that a maximum number of facilities would report receiving them. This strategy hence provides upwardly biased percentage of facilities receiving materials from the authorities.
For instance in Chad, data on financial transfers were available from the national budget (final). For the material, data from the Direction administrative was collected for 2003. All the slips of invoices were available in a single binder with dates of shipments, the recipient of the material, list of material sent with quantities and prices, signature of recipient and comments. All records were photocopied for further analysis. To measure the material received at the various administrative level, one strategy could have been to photocopy all forms found. In Chad for instance, this was not feasible given that there were no photocopy machines in especially in rural areas. Information was obtained through structured questionnaires.
Given the very large number of various items sent to lower administrative levels, the arrival of material in Chad was done on a sample basis. Focus was on 8 frequently sent items, to maximize the probability of finding the item and that the data could be triangulated. The region questionnaire asks what was the quantity of a specific item sent to districts and facilities in the sample. Then at the district level, the questionnaire asked how much received from the region and how many sent to the facilities. Finally the facility questionnaire asks how many of these items were received. All numbers have to come from the records.
7- Instrument design PETS and QSDS are generally composed of various survey instruments intended to collect information at the different organizational levels and among stakeholders involved in service delivery, on both the supply and demand sides. The design of survey instruments depends on survey objectives and the choice of tracking flows. Past surveys have typically included modules on the following units (Table 3): Central government�s Ministry of Finance Central government�s line ministry (e.g. Health or Education) Provincial (or regional) administration District (or local) administration Service provider (e.g. school or health centers) Staff Clients (e.g. patients or students) In some cases, households information has also been collected Data collected have covered information on seven main categories of variables (Table 4):
Slide 41:Instrument design
Records vs recall: For quantitative data, records should be used as possible to minimize measurement errors When no other sources of data are available and recall data are collected, clear indications in that respect should be reported. Length of data tracking: Data collection should ideally involve annual data and cover a period of a maximum of two financial years to maximize data collection quality. If monthly data are collected, seasonality issues need to be examined beforehand. Recording procedures: survey instruments should be adapted to the specific recording procedures in practice in the administrative units and facilities under study. Parsimony of data collected is recommended to reduce costs, but also to increase the quality of data collected. The questionnaire should be focused, and contain a reduced number of questions.
Valuation of in-kind items received Sometimes, the official price of in-kind transfers is not known by frontline facility officials. In such case, valuation at market price is required. Note that even when official prices are available, prices could be inflated by the recipient or the supplier. It is thus recommended to verify the prices with market prices for equivalent goods. Such verification was done for instance in Mali part of an education PETS in 2005. Official prices were compared relative to market prices for the same items and important overvaluation was discovered. The Madagascar 2003 survey also compared official prices of in-kind items relative to market prices.
Exemples of instruments Zambia PETS/QSDS (9 questionnaires) Chad PETS/QSDS (10 questionnaires) Uganda QSDS (3 questionnaires) Instruments available at: http://econ.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTRESEARCH/EXTPROGRAMS/EXTPUBSERV/0,,contentMDK:20292627~menuPK:545282~pagePK:64168182~piPK:64168060~theSitePK:477916,00.html#PETS Could be useful to design PETS in the perspective of a baseline for benchmarking for the purpose of monitoring and evaluate future intervention in the sector. In such case, indicators are required. Various indicators could be built. Table 5 provides some potential PETS/QSDS indicators
Slide 44:Table 5: Potential PETS/QSDS Indicators
Slide 45:Table 5: Potential PETS/QSDS Indicators (Continued)
Slide 46:Table 5: Potential PETS/QSDS Indicators (Continued)
8- Preliminary pilot phase: All survey instruments should be tested through a small scale pilot phase on a specific number of units and ownership types or regions. The choice of tracking flows should be assessed, as well as the quality and consistency of data. In particular, quantitative data questions, financial data, inputs and outputs, which have been customized to the sector�s administrative system, should be carefully examined. Questions� wording, ambiguous responses, answer codes, etc. should be revised at this stage.
9- Training Adequate time for training A minimum of one week (and possibly two) are required for enumerator and supervisors training. The training by the survey core team involves acquainting enumerators and supervisors with the instruments and techniques used in data collection. Training should include the testing of instruments by agents in the filed. Enumerators or supervisors should have access to a survey manual detailing questions� objectives and interpretation. Following the training, questionnaires may be revised.
10- Full pilot phase Following (or during) the training, all questionnaires should be tested. Field testing of instruments is crucial for increasing the likelihood of obtaining good quality survey information. A test on about 5% of the sample including all types of respondents should be adequate. Following the pilot, a final revision of instruments should be carried out.
11- Survey implementation Field work must be closely monitored Core team provides technical assistance and monitors the implementation of the survey Random visits to enumerators to ensure quality control and coherence in the interpretation of questionnaires. Random checks of questionnaires and data quality during the survey implementation. Survey Timing: In the pre-design phase, an essential decision is to determine when to field the survey. The fielding of a survey should ideally be done two or three months after the end of a fiscal year, in order for accounting books to be closed. In any respect, the tracking should always be done on the preceding fiscal year, never on the current one.
Slide 51:Survey implementation
Some past survey did not do that and faced data quality problems. For instance, in Madagascar, a PETS tracked two main funding programs to schools and examined delays in salary payments to teachers. However, the timing of the field research (April-May 2003) did not capture clear information about leakage given that the 2002-03 school year was in progress when the survey was carried out, and some of the data were collected for the incomplete school year. The study could not distinguish between direct leakage and delays in budget execution of the main schooling programs studied due to the data collected (Francken, 2003, p.22). Cameroon PETS 2005 had same type of problem of collecting data on a year in progress.
12- Data entry and cleaning Data entry programs should be written following the completion of the questionnaires Should be tested during the survey pilot phase. Inconsistencies and potential errors in data detected should be verified while the survey is still being fielded Data cleaning and analysis should be done shortly after the end of data collection.
13- Reports / Analysis Policy reports and research documents should clearly identify and communicate the specific findings and enounce specific policy recommendations Study findings: focus on governance, public expenditure effectiveness and public service delivery performance.
14- Dissemination/Follow up Dissemination of the results Follow up: seminars/workshops to present , discuss and interpret the findings and implications for policy should involve government, civil society, local community and other national and international stakeholders Final report disseminated and available on the web Specific service delivery inefficiencies, including measures of leakage at each level Recommendations to be implemented, etc. Follow up of policy reforms
Slide 55:III. Conclusion
PETS/QSDS have been useful in identifying inefficiencies, capture of funds and problems of incentives in the service delivery supply chain. Important to diagnose the service delivery system but also identify ways to improve it, which requires evaluating the impact of different interventions Ultimately, tracking surveys are a means to an end: Need to develop policy dialogue and dissemination of results to ensure the transfer of information about problems identified in the service delivery system to correct the various governance problems identified in service delivery.
Slide 56:References
Ablo, Emmanuel and Ritva Reinikka (1998) �Do Budgets Really Matter? Evidence from Public Spending on Education and Health in Uganda� Policy Research Working Paper 1926, The World Bank, Washington D.C. Das, Jishnu, Stefan Dercon, James Habyarimana and Pramila Krishnan (2004) � Public and Private Funding Basic Education in Zambia: Implications of Budgetary Allocations for Service Delivery� Africa Region Human Development Working Paper Series No. 62, The World Bank, Washington, DC. Francken, Nathalie (2003) Service Delivery in Public Primary Schools in Madagascar: Results of a Budget Tracking Survey, The World Bank, Madagascar Country Office, Antananarivo, September, processed. Gauthier, Bernard (2006) �PETS and QSDS in Sub-Saharan Africa: A Stocktaking Study�, HEC Montr�al and World Bank, Washington DC. August, mimeo. --------- (2008) �Harmonizing and Improving the Efficiency of PETS/QSDS� The World Bank, mimeo, march --------- (2008) �Measuring Public Expenditure and Service Delivery Performance Across-Countries: Micro-level Governance Indicators Using PETS/QSDS� � The World Bank, mimeo, September Gauthier, Bernard and Waly Wane (2009) �Leakage of Public Resources in the Health Sector: An Empirical Investigation of Chad� Journal of African Economies (18) 52-83. Gauthier, Bernard and Ritva Reinikka (2007) �Methodological Approaches to the Study of Institutions and Service Delivery: A Review of PETS, QSDS and CRCS�, African Economic Research Consortium framework paper, The World Bank, mimeo, December. Reinikka, Ritva and Jakob Svensson (2004) �Local Capture: Evidence from a Central Government Transfer Program in Uganda�, Quarterly Journal of Economics, 119 (2): 1-28. Wane, Waly (2007) �PETS in a difficult environment: The case of Chad�, The World Bank World Bank (2004) World Development Report 2004: Making Services Work for Poor People, World Bank and Oxford University Press, Washington DC. World Bank�s Survey Tools: Human Development and Public Services Research http://econ.worldbank.org/external/default/main?theSitePK=477916&contentMDK=20292627&menuPK=546432&pagePK=64168182&piPK=64168060#PETS

Contents

Contents

Presentation Transcript

Contents

Contents

Contents

Contents

Contents

Contents

Contents

CONTENTS

Contents

Contents