Measuring the Quality of Education and Health Services : The Use of Perception Data from Indonesia

1. Measuring the Quality of Education and Health Services:The Use of Perception Data from Indonesia Basab Dasgupta Ambar Narayan Emmanuel Skoufias PRMPR- World Bank April 21, 2009

2. Motivation & Scope-1 • Increasing trend of decentralization of service delivery to local governments • Decentralization  increases accountability, increases citizen participation and political engagement, improves public service delivery, allocative efficiency and fiscal administration • Good measures for local government performance necessary for evaluating the impact of decentralization • Many of these tools include subjective instruments that gauge citizen perceptions (citizen report cards, community scorecards, facility exit polls, and citizen satisfaction surveys) , (Amin, Das, and Goldstein, 2007). • Can satisfaction-related questions be valuable in measuring the quality of public services, specifically in health and education?

3. Motivation & Scope-2 • satisfaction surveys are appealing • A quick and easy way for policymakers to measure the impact of governance reforms on government performance, particularly for sectors where measurement of service quality is not easy, provided citizen satisfaction is closely correlated with the actual quality of services. • Less time and labor intensive than facility surveys and public expenditure tracking surveys) • BUT…. there is little consensus on whether citizens’ satisfaction reflects the actual quality of services satisfaction surveys • Whether useful • and if so How to use it

4. Motivation & Scope-3 • Need better understanding of what factors influence satisfaction • What household and community level factors (besides quality) play a role? Decentralization  improved outcomes is based on premises (i) increased accountability of service delivery increases performance and (ii) citizens are able to discern between good and bad gov’t and then influence local authorities.

5. Concerns w/ Satisfaction Surveyse.g. Indonesia-GDS2 survey • May get High Reported Satisfaction due to cultural norms or social pressure rather than the superior quality of services. • Absence of a common baseline makes it • Difficult to interpret the data • Tricky comparing data-points across regions and countries

7. When most users claims to be “satisfied” with services... • Average satisfaction with public health and education facilities surprisingly high in GDS-2 • Among five options (scale of 1 to 5), option 1 or 2 (somewhat satisfied or better) chosen by around 90 and 80% of hholds for health and education respectively • Inconsistent with the poor reputation of health and education services in Indonesia, also supported by more objective measures of quality from surveys like GDS-2 • Problem not unique to Indonesia, but not universal in surveys either • High variation in satisfaction with education and health services in a number of countries where CWIQ-type surveys have included questions on satisfaction • 2006-07 survey in Pakistan showed 35% satisfaction rate for public health facilities; 2007 survey in Sierra Leone showed satisfaction rate of ~40% for public schools • High reported satisfaction in Indonesia probably attributable mainly to cultural norms • Key question: is there useful information content in the Indonesia satisfaction data?

8. GDS-2 survey--1 • Survey in May-Sept 06 to assess governance and local public service delivery in Indonesia • collecting data on quality and satisfaction from households, communities, and facilities • Household sample (8544 hholds) nationally representative – 1068 PSUs (hamlets or dusun), in 89 districts (kabupaten/kota), 267 sub-districts (kecamatan) and 534 villages • Facility data collected from a sample of health and education facilities – guided by which facilities were reported as most frequently used by households • Health facility data from the 6 community health centers (puskesmas) most frequently mentioned by households within each sub-district (kecamatan) • School facility data collected from the most frequently used public elementary school in a village and public junior high school in a sub-district

9. GDS-2 survey--2 • Thus data on both household satisfaction and objective measures of facility quality are available for 52% of households using public health facilities and 57% of those using public education facilities. • Sources of possible selection bias • Households choosing public facilities may systematically differ from those choosing other types of facilities • Restricting the facility sample to the most frequently used facilities instead of a random sample of all available facilities can potentially add to the bias 9

10. Model--1 • Simple expectancy disconfirmation model • S= f (Quality – Expectations) Where expectations=g(hh char & experience) • A modified expectations disconfirmation model • Expectations play a role in the choice of the type of facility; expectations proxied by household and community level factors • Conditional on the choice of a service provider, reported satisfaction with the service facility is a function of the actual quality of the service and governance (service is an “experience good” experience it only after choosing it). • Caveats: quality of service measured in terms of inputs in the production of services and NOT outcomes (e.g., achievement scores, z-scores).

11. Model--2Two-stage Heckman model to correct for selection bias • 1st stage selection equation predicts the propensity of households to use a facility for which objective data on quality are available • 2nd stage equation examines how satisfaction (S) correlates with indicators of quality and governance, conditional on the choice of a service provider • Models estimated separately for samples from poor and rich districts, and in a pooled sample • Accounts for difference in expectations between the residents of rich and poor areas • Rich districts = GRDP pc > median GRDP pc in sample of 88 districts

12. Defining the satisfaction variable in GDS-2 • Useful information can be extracted by focusing on variations in satisfaction level • To exploit the variation, we define our dep. variable as a binary S, which is =1 for all those who chose option 1, and =0 otherwise • For health services, S=1 for 58% of hholds; for education services, S=1 for 50% of households Analyzing the determinants of S using two models • Simple heuristic model: correlating household characteristics with satisfaction, separately for health and education • Illustrates the range of factors – that service providers have no control over – influencing satisfaction with services among households • 2-stage Heckman selection model: corrects for selection bias in facility choice

13. HEALTH

14. Health facility choice—1st stageExplanatory vars • S=1 if the household uses a public health facility (puskesmas) for which facility data is available, =0 otherwise • Explanatory variables • Expectations proxied by household and community characteristics • Other factors likely to affect choice of facility: location of facility (in village or not), availability of information on corruption and the sources of that information • Wealth, rural-urban designation, and provincial location of district

15. Health facility choice—1st stageResults Demographic/socio-economic characteristics play a statistically significant role in influencing whether or not a puskesmas is the facility of choice for a household These characteristics are far more important for hh in poor districts than rich districts Geographic location: relative to households in Java, households belonging to other regions are more likely to use puskesmas, with some differences between rich and poor districts Household’s relative position in society (leadership position) or access to information do not seem to matter for choice of health facility Caveat: these factors explain the choice of a certain type of public health facility for which facility data is available 15

16. Satisfaction with health-- 2nd stageExplanatory vars • Objective indicators of quality and governance environment • Quality : (a) coverage area of puskesmas; (b) types of medical support available from ancillary facilities; (c) quality of services (human resources and medical supplies); (d) infrastructure • For each category, multiple indicators combined into a single index created using Principal Component • Institutional /governance environment : (a) willingness to complain (voice), (b) govt responsiveness to complaints (accountability), and (c) an index of participation in the administration of health services

18. Satisfaction with health-- 2nd stageResults • Strong correlation between satisfaction and certain dimensions of service quality • Namely, support available to the main puskesmas from ancillary facilities, the quality of service care in terms of human and medical resources (for poor districts); quality of infrastructure has insignificant impact • Thus citizen satisfaction with health facilities, particularly in poor districts, correlated more with the availability of ancillary facilities, human personnel and medicines, rather than facility infrastructure • Institutional and governance indicators correlate with satisfaction, but with rich-poor differences • Hholds in poor districts more likely to be satisfied when they participate more in the administration of health services • Hholds that complained about health services less likely to be satisfied with health services; hholds in rich districts more satisfied when health services are responsive to complaints • All the governance indicators may be endogenous with perceptions of quality and satisfaction 18

19. EDUCATION

20. Education facility choice—1st stageExplanatory vars Sample restricted to households that have at least one child of school age Dep variable =1 if the household sends a child to a public school for which facility data is available, 0 otherwise; explanatory variables similar to 1st stage model for health

21. Education facility choice—1st stageResults • As expected, a household’s “selection” of a school is the result of a very different decision process from its choice of medical care • Demographic and socio-economic characteristics matter less for the choice of schools than puskesmas • Access to information and social status of households seem to matter for schooling choice and not for the choice of health facilities • Regional location of the household again plays a role: relative to households in Java, households belonging to other regions are more likely to send their children to public schools • Again, the model explains the choice of a certain type of school (public) for which facility data is available 21

22. Satisfaction with education--2nd stage Explanatory vars • Indicators of school quality : (a) quality of infrastructure in school; (b) the quality of teaching staff; ; (c) student performance; and (d) coverage of students (enrollments, attendance) by the school • Institutional and governance environment: • (a) information available to households about bribery and corruption in education, • (b) willingness to complain against service providers, • (c) provider’s responsiveness to complaints, • (d) extent of participatory decision-making in school • (e) coverage and implementation of BOS

24. Satisfaction with education--2nd stage Results • Strong correlation between satisfaction and certain dimensions of school quality in the pooled sample, but important differences between rich and poor districts. Satisfaction levels have significant correlation with: • Better infrastructure facilities in schools (e.g. condition of classrooms, availability of books, library, computers) – for poorer districts only • Participatory decision-making for school’s mission and vision (jointly by school principal, teachers and the community) – for rich districts only • The coverage of a school by the BOS program and the progress in implementation of BOS – for rich districts only 24

25. Satisfaction with education--2nd stage Results Thus satisfaction in poor districts more influenced by the basic features of a school (e.g. condition of building facilities), whereas in richer districts influenced more by “second-generation” factors (e.g. participatory mode of mgmt) ? Knowledge of bribery and corruption in education and complaints against schools have significantly negative effects on satisfaction, but again with differences between rich and poor districts 25

26. Comparing results: health vs. education--1 • Key differences in what factors matter for satisfaction with health and education services • While quality of infrastructure has no influence on satisfaction with health facilities, quality of school infrastructure has significant influence on satisfaction with schools in poor districts • Indicators of quality (availability of personnel and medicinal inputs) are key determinants of satisfaction in health services; but teacher quality does not seem to correlate at all with satisfaction in schools

27. Comparing results:health vs. education--2 • For both education and health facilities, a more participatory management of the facility induces higher satisfaction. However, differences in how the results vary across rich and poor areas • For health facilities, households in poor districts are more likely to be satisfied with higher participation in the administration of health services • For schools, households in rich districts are more likely to be satisfied when management of schools is more participatory or the implementation of a school-based management system is more advanced • Responsiveness to complaints about facilities improves satisfaction with health facilities in rich districts only and satisfaction with schools in poor districts only 27

28. Comparing results:health vs. education--3 • Important difference in how the regional location of a household influences satisfaction • Matters only marginally for satisfaction with health facilities (and has no effect for poor districts) • But satisfaction with schools likely to be much lower in the poorer areas of the Kalimantan, Sulawesi and Sumatra regions compared to the poor areas of Java region 28

29. Implications of our analysis “Satisfaction” data has important information content • But requires finding meaningful variation in responses and models to account for selection bias and role of expectations in facility choice • For health and education, satisfaction with facilities significantly correlated (in the right direction) with measures of quality, governance and institutional environment of the facility • In many cases, collecting satisfaction data matched with facility level data is not practical • Common practices: user surveys/citizen report cards, hhold surveys • What does our analysis suggest for such “2nd-best” scenarios • The design and use of instruments that just measure satisfaction from hhold surveys, as a proxy for quality of services?

30. Implications for surveys collecting satisfaction data--1 • Even if a linked facility survey is not possible, clear benefits in having a satisfaction survey collect as much information on the characteristics of households and communities as possible, including proxies for governance/institutional environment • Random sample administered at household level is likely to yield more representative results in most cases than a survey of a sample of users of a particular type of facility • Better for correcting the selection bias arising from facility choice, in contrast to a survey limited to the users of a particular type of facility • Incorporating questions on satisfaction with basic services in household surveys is becoming more common (E.g. CWIQ surveys combining questions on satisfaction with basic services with hhold and community characteristics) • In some cases where user surveys are the only practical option (e.g. when a service is used by a tiny % of the population), collecting hhold/community level information from users is recommended

31. Implications for surveys collecting satisfaction data--2 • Even if most respondents appear to be satisfied (or not), useful information can still be extracted by using the variation in responses, rather than the strict meaning of the responses • Butthe survey design must allow for that, for example through: Multiple choice responses – variation in response more likely to occur when surveys phrase satisfaction-related questions with multiple options, as opposed to a simple “yes/no” or “satisfied/dissatisfied” Being specific – More variation in responses likely if questions are specific to different aspects or features of a school or health facility, as opposed to a single “are you satisfied with school/health center” type question 31

32. Implications for education and health services in Indonesia--1 Which aspects of health and education services in Indonesia matter the most for user satisfaction and how these differ across richer and poorer districts • Infrastructure • Quality of personnel and inputs • Participation in decision-making, governance and accountability

33. Implications for education and health services in Indonesia--2 • Infrastructure • Satisfaction with health facilities related more with access to ancillary facilities supporting the main puskesmas, rather than the physical infrastructure of the puskesmas • But for schools in poor areas, infrastructure appears to be high priority among parents of students 33

34. Implications for education and health services in Indonesia--3 • Quality of personnel and inputs • Quality of human resource and medicinal inputs in health facilities is a major concern among users in poor districts • Indicators of teacher quality and student performance do not seem to matter much for user satisfaction • Does not necessarily imply that households are ambivalent about education quality– rather that these indicators may not reflect the aspects of “quality” households care most about 34

35. Implications for education and health services in Indonesia--3 • Participation in decision-making, governance and accountability • Greater degree of community participation in decision-making for facilities and better responsiveness of service providers to complaints appear to improve satisfaction • In education, satisfaction positively correlated with the extent of implementation of BOS (decentralization of service – e.g. school-based management, allocating funds to schools, participatory planning) • Indicators of participation and decentralized service delivery may proxy the “quality” valued by users; may also reflect a special value users may attach to being involved in the management of the facilities • Key questions: why these indicators matter for satisfaction, what explains the variations between rich and poor areas, and what that implies for the priorities of a government ? 35

36. Thank you

37. Simple heuristic model of household satisfaction • OLS of S on key characteristics of households and communities, using the full household sample, separately for health and education • Household variables include education, gender, age, ethnicity, religion; district level variables such as urban/rural and rich/poor districts • Restricted to characteristics that service providers have little control over; variables related to objective quality of services omitted • Certain hhold characteristics have statistically significant correlation with reported satisfaction • E.g. gender of hhold head, religion, ethnicity, education level, household composition • Rural respondents less likely to be satisfied with public facilities they frequent • Information on governance seems to matter • (i) households less likely to be satisfied when they know about complaints, but more when there was a follow-up in response; (ii) information about corruption/bribery in education strongly associated with lower satisfaction, but not so for health services • But these types of information highly likely to be endogenous

38. Sample Selection Strategy 1 • Kabupaten/Kota (district) Sampling frame: 434 districts (–) following 26 districts =408 districts (21 (Aceh) – 2 (Nias) – 3 (pre-test locations: Kabu- Maros (South Sulawesi), Kabu- Pontianak (West Kalimantan), and Kabu- Muaro Jambi (Jambi)) • Procedure 1. 89 districts are selected randomly using SRS • Rest of the districts are top-offs from ILGR districts, USDRP, SPADA, and SPADA-Justice (WBOJ); • And 2training districts : Kota Salatiga and Kabu- Boyolali (Central Java) • Total districts selected = 135 (with 1 dropped)

39. Sample selection strategy 2 • Kecamatan (sub-district) Sampling frame: all kecamatans within respected districts (data source: PODES 2005) • Procedure 1. Random districts: 3 sub-districts using PPS with # HH as weight • Total sub-districts selected = 134 * 3 = 402 • Desa/Kelurahan (village) Sampling frame: all villages within respected kecamatans with 50+ HHs; 2. 16 households within the village • Procedure: 1. Pick 2 villages using PPS • Total villages selected = 402 * 2 = 804

40. Sample selection strategy 3 • Dusun (hamlet) • Sampling frame: all hamlets within respected village • Procedure: sort hamlet names alphabetically; select 2 hamlets from the sorted list: (a) the first and (b) the middle of the list • Total hamlets selected = 804 * 2 = 1,608 • Household • Get the most recent household list from head of hamlet. Randomly select 8 households from the list. • Total household = 1,608 * 8 = 12,864 • Note: We use 88 randomly selected districts

41. Sampling criteria: Schools & Puskesmases Sub District (3) District Education Unit=1 SD-3: Junior High (1) Puskesmas (2) SD-2: Junior High (1) Puskesmas (2) SD-1: Junior High (1) Puskesmas (2) Village(6) V-1: Primary (1) Pvt.Paramed (3) Pvt. Doctor (1) V-2: Primary (1) Pvt. Paramedic (3) Pvt. Doctor (1) V-3: Primary (1) Pvt. Paramed (3) Pvt. Doctor (1) V-4: Primary (1) Pvt.Paramed (3) Pvt. Doctor (1) V-5: Primary (1) Pvt. Paramedic (3) Pvt. Doctor (1) V-6: Primary (1) Pvt. Paramed (3) Pvt. Doctor (1)

42. Selection of facility respondents