Caitlin Biedron ASPH M&E Fellow CDC-Rwanda January 15, 2009

1. Assessment of Malaria Outcome Indicators Using Lot Quality Assurance Sampling (LQAS):Estimation of Bed Net Coverage in Mozambique Caitlin Biedron ASPH M&E Fellow CDC-Rwanda January 15, 2009

2. Lot Quality Assurance Sampling (LQAS) • Classification method used in industrial sampling to identify batches of products (or lots) with an unacceptable number of defective items. • Typically implemented as part of a stratified random sampling design: small samples are selected from all lots in a given area. • In each lot, the sample determines whether coverage by a health intervention exceeds a specific target by using a statistically determined decision rule.

3. LQAS Terminology • A lottypically consists of a supervision area, such as a health district. • Lots classified as being acceptable/unacceptable vis-a-vis the target. • In this analysis lots are enumeration areas (what-if analysis). • The decision rule is the minimal number of individuals in the sample that should have the intervention. • Selected to determine whether a population coverage target for an intervention has been reached, such as 80% vaccination coverage. • If this coverage target is deemed to have been reached, the lot is classified as acceptable.

4. LQAS Methodology The decision rule used when applying the LQAS method can be represented with the composite hypothesis test: The null hypothesis will be rejected if it is determined that the coverage proportion (p) of the lot is below an adequate level of coverage (pc). The decision rule is defined by pc , the threshold value of coverage below which program managers deem services to be unacceptable.

5. LQAS Methodology The α error is also commonly referred to as type I error, termed the provider risk in the LQAS context. A type I error is made if one rejects the null hypothesis when it is true. It is labeled as provider risk because the provider is at risk when the health area is identified as inadequately covered by an intervention when it has in fact met the target coverage level. The classification of substandard lots as acceptable is called consumer risk, commonly referred to as the β error or type II error. A type II error is made when one fails to reject the null hypothesis when it is false. The magnitude of the classification error varies both with the sample size and the maximum number of units without the intervention permitted in the sample (d).

6. Mozambique MIS • Mozambique MIS was conducted by the NMCP in partnership with the Malaria Consortium and PMI. • Included a sample drawn from a subset of EAs from a population proportionate sample from the 1997 census. • A total of 346 EAs with 5990 households were selected from the 1510 EAs of the primary sample. • After data cleaning, a total of 5745 household records were available for analysis. • The survey was conducted June - July 2007.

7. LQAS Parameters • We based our decision rule on a coverage target (pc) equal to 70% and defined our α error to be 10%. • Year 1 Mozambique PMI target for bed net possession called for 70% of households to own at least one ITN. • In urban clusters, the typical sample size was 20 households. In rural clusters, the typical sample size was 15 households. • Based on these parameters, we calculated the decision rule for both urban and rural enumeration areas. • The decision rule for urban clusters was 12 households; for rural clusters it was 8 households.

8. LQAS Results: Any Bed-net (Urban EAs)

9. LQAS Results: Any Bed-net

10. LQAS Results: Any ITN

11. Mozambique Bed-net Coverage

12. LC-LQAS Background • May not be feasible to conduct LQAS in the traditional fashion; instead only a subset of the areas in the catchment region represented would be selected. • Pooled data would no longer be a stratified random sample, but instead a cluster sample, if the areas had been chosen at random. • The recently developed large-country lot quality assurance sampling (LC-LQAS) method is used to integrate LQAS with cluster sampling. • We next investigated the possibility of taking only a sub-sample of the EAs included in each province to calculate the provincial and national coverage estimates. • This second aggregation was done in order to compare the estimates obtained using a smaller sample (220 EAs) to those estimates resulting from the full sample (346 EAs).

13. LC-LQAS Results: Any Bed-net

14. LC-LQAS Results: Any ITN

15. Comparison of Coverage Estimates

16. Strengths of LQAS • Allows for pass/fail classification based on various decision rules; • Dichotomous nature of the results can provide local supervisors with a decisive judgment about action to be taken; • Sample sizes are typically smaller than those required to perform other estimation analyses; • Inexpensive relative to traditional surveys; • Surveys can capture variability across local areas, which encourages local level monitoring and accountability; • Surveys can be conducted more often, providing information more frequently than HH surveys; • Sample can be treated as a stratified sample for further analysis; • With adequate sampling and appropriate weighting, can provide estimates at aggregate levels (provincial, national); • To accommodate large countries, sampling areas may be chosen randomly based on cluster sampling methodology (LC-LQAS).

17. Limitations of LQAS • A high level of technical competence is required to achieve more than the basic pass/fail classification; • Understanding of sampling frame requires knowledge of household selection, weighting and binomial probability; • Perception that methodology is ‘simple’ belies technical complexity; • Decision rules may seem arbitrary if they are not based on explicit program targets or certain thresholds related to transmission; • Sample sizes tend to be small, so estimates may have large confidence intervals when sample combined; • Pass/fail outcomes do not give information on scale of change at the local level, and it is not clear how dichotomous outcomes will be used to track progress in programs; • Survey EAs must be selected in accordance with health districts or administrative units in order for results to have programmatic relevance.

18. Potential Applications in the Field • The demonstrated methods could play a tangible role in the field where there is a growing interest in obtaining coverage estimates for small geographic units. • LQAS could be used to assess EA data collected by standard MIS surveys to provide more frequent tracking of coverage indicators for districts country-wide or within key target areas. • The standard MIS methodology requires large sample sizes to obtain reliable district level point estimates, making the survey both cumbersome and costly to conduct. • LC-LQAS could provide local and national level estimates for coverage when it replaces the standard MIS sampling. • LC-LQAS methodology would be applicable in those countries in which survey sampling units are selected in accordance with health districts or administrative units.

19. Potential Applications in the Field • LQAS offers an alternative to malaria control program managers who are interested in tracking coverage at a local level to improve their service delivery strategies and to adjust priorities. • As more countries aim to control malaria and scale-up coverage to do so, it may become necessary to offer alternatives to the national level surveys that provide a single set of indicators for a country and do not present differences across local levels. • National malaria programs are increasingly interested in getting data from smaller geographic areas; LQAS may be a suitable method for such programs to use.

20. Acknowledgements Amy Ratcliffe, CDC Malaria Branch Albert Kilian, Malaria Consortium Marcello Pagano, Harvard School of Public Health Marcia Castro, Harvard School of Public Health Bethany Hedt, Harvard School of Public Health Joe Valadez, Liverpool School of Tropical Medicine Juliette Morgan, CDC-Mozambique