1 / 42

Layers: An LQAS Application for the Collection, Entry and Analysis of Monitoring Data

Background: Why Layers?. Originally developed to help USAID Missions fulfill their oversight responsibilities of PL-480, Title II ProgramsInitial impetus given by following regulations:Cooperating Sponsors have primary responsibility for implementing, monitoring, reporting on and auditing their

uma
Download Presentation

Layers: An LQAS Application for the Collection, Entry and Analysis of Monitoring Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Layers: An LQAS Application for the Collection, Entry and Analysis of Monitoring Data Gilles Bergeron Food Aid Managers’ Course January 8th, 2004

    2. Background: Why Layers? Originally developed to help USAID Missions fulfill their oversight responsibilities of PL-480, Title II Programs Initial impetus given by following regulations: “Cooperating Sponsors have primary responsibility for implementing, monitoring, reporting on and auditing their Title II program activities, commodities and funds. However, USAID Missions are expected to monitor CS's management of the commodities and use of sales proceeds and grant funds” “Oversight and monitoring should include (…) regularly scheduled visits to distribution centers and warehouses, and established time tables for submission of required reports and program documentation.” Chapter 9, Section B of DCHA/FFP enabling regulation

    3. System typically used by Missions to monitor T-II programs: Commodities arrive in country and USAID signs bill of lading CSs receive their food commodities and ship to inland distribution points Daily management of commodities (storage, distribution) is done by local CS representative Periodically, the site is visited by USAID Food Monitors to verify for compliance with directives on proper storage and disposal of food. The intent is for Food Monitors to progressively visit all sites where food is handled. Food monitors report on their visit to USAID upon return. If discrepancies with contract or Reg. 11 are noted, the CS is notified. There are no other specific methodological guidance from USAID to direct the work of food monitors in terms of sampling, indicators, etc.

    4. Issues & options with current approach ISSUES Only commodities targeted. Program activities not considered Selection of sites left to convenience. No systematic sampling of sites Only site visited assessed. No program wide monitoring. Designed for USAID’s use only. No transparent feedback to partners OPTIONS Take advantage of field visits to assess program activities Select sites randomly using systematic sampling Move from site-specific assessment to program wide monitoring Share feedback with partners to improve program performance.

    5. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    6. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    7. Haiti T-II example

    8. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    9. LQAS: A simple, low cost random sampling methodology Originally developed in the 1920s to control the quality of output in industrial production processes Involves taking a small sample of a manufactured batch (lot) and test sampled items for quality. If the number of defective items in the sample exceeds a predetermined criteria (decision rule), then the lot is rejected. The decision rule is based on desired production standards, and a statistically determined sample size n is chosen so the manager has a high probability of accepting lots that meet the quality standards, and a high probability of rejecting lots that fail to meet those standards

    10. Useful LQAS definitions Standard LQAS theory Production standard: % of items that must “pass” before the lot is accepted Production unit: The machine or team that produced or assembled the lot Lot: Total number of items produced in given time by the production unit T-II programs Coverage % of persons to be covered by the service (i.e. received food, or were vaccinated) Supervision unit The CS, or program implementer that delivers the service Supervision area: Total number of sites in a given zone delivering service (food, vaccines, etc)

    11. Why 19?

    12. Optimal LQAS Decision Rules for Sample Sizes of 12-24 and Coverage Benchmarks of 20%-95%

    13. Conclusion on LQAS as a method Low sample size needs (n=19 in most cases) Simple to apply yet very specific conclusions Result=High quality information at low costs BUT Only dichotomous outcomes allowed (pass/fail, complies/not complies, yes/no) Subsets are problematic

    14. Application of LQAS to monitor T-II programs Each CS program is defined as a supervision area. Sample units are the facilities where food is stored for later distribution (warehouses, schools, health centers, orphanages) What is evaluated is the overall success of each CS in delivering specific services Services observed include food distribution itself, plus the activities paired with the delivery of food (e.g. vaccination with MCHN, school attendance with FFE, quality of roads with FFW, etc).

    15. Food distribution points by CS and by program area (hypothetical example)

    16. LQAS sampling scheme for all CSs by type of food storage facility (N : n)

    17. LQAS sample size and decision rules for a performance standard of 90%

    18. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    19. Management of food commodities: Commodity movements Ledger and physical inventory agree? Tracking system in place? Storage Storage site appropriate, clean, ventilated? Traces of losses, due to water, rodents, broken containers? MCHN: Immunization program Cold chain OK? Drug and vaccine stock outs? Drugs and vaccine obsolete (pass due)? FFE: Educational performance Is attendance up to expectations? Teaching quality Are teachers present and adequately equipped? Does the school infrastructure meet minimal standards? Support systems Is water and sanitation adequate? Is PTA present and active? FFW Roads Does road construction follow EIA standards? etc What indicators? Aspects of performance to measure

    20. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    21. Simple data entry Large amounts of information generated: each round = 180 cases, each case +/- 100 questions, so 18,000 data points in total per round For analysis purposes, data must be digitized and cleaned before processing - this takes time Currently, questionnaire forms are filled in paper, not computerized. Layers automates all this in one single step using PDA devices and a specially designed data entry and retrieval application

    30. Recap: Data Entry Process 1) Go to site, fill in the questionnaire. Repeat in each site 2) Return to headquarters and connect PDA to server 3) Data is automatically downloaded to server and made ready for analysis

    31. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    32. Data analysis routine Analysis with LQAS is very simple: Count number of “pass/fail” for each indicator Apply decision rule in each case Make judgment on overall quality of services

    33. Making LQAS judgments on the overall quality of services offered

    34. Example of report See handout

    35. Methodological Requirements Relatively homogenous CS programs Rapid, low cost sampling method Well defined indicators Simple data entry method Systematic data analysis routines Clear definition of all audiences

    36. Reporting to target audiences Primary audiences are: USAID, and the CSs Audiences include also all entities that share responsibilities for the delivery of the services MoH: Provision of vaccines, cold chain equipment, etc MoE: Ensure teacher competence, pedagogical supplies Other partners (e.g. CARITAS, etc) Identifying who is responsible for what, and making them accountable will help the CSs achieve expected results Frequency of reporting: every time a round is completed (Two rounds per year possible if n = 180 and 3 full time FM employed)

    37. Extensions of Layers Currently, Layers covers only sites where food is distributed (Health Centers, schools, FFW storage points.) This limits its usefulness Requests to expand its application to all operational levels: Community health centers/rally posts (used in MCHN programs) Farmers’ fields (AG programs) Road sections (FFW programs) CSs and Missions can use the principles developed here, changing only: The definition of supervision area The definition of supervisor The choice of indicators

    38. USAID Supervision area

    39. PVO 2: District 3 as Supervision Area

    40. By USAID Missions: To evaluate condom availability in HCs To evaluate the success of immunization campaigns Customer satisfaction surveys By PVOs “Disassemble” Layers: Use only the PDA app on 100% of food distribution sites (PVO CMUs) Use only the LQAS part to do special studies (eg. customer satisfaction surveys, sampling for specific accounting needs) Test the quality of field staff (capacity to deliver services) Further extensions of Layers: Other uses

    41. For more information… Gilles Bergeron gbergero@aed.org Megan Deitchler mdeitchl@aed.org WWW.FANTAPROJECT.ORG

    42. Acknowledgements The following persons have contributed to the development of Layers: Carell Laurent, Florence Cadet, USAID/FFP: original idea Joe Valadez, The World Bank: LQAS methodology consultant David Cantor and John Thies, ORC/MACRO: software development Layers is written and managed by Gilles Bergeron and Megan Deitchler, FANTA Project

    43. This presentation was made possible through the support provided to the Food and Nutrition Technical Assistance (FANTA) Project by the Office of Food for Peace of the Bureau for Democracy, Conflict and Humanitarian Assistance and the Office of Health, Infectious Disease and Nutrition of the Bureau for Global Health at the U.S. Agency for International Development, under terms of Cooperative Agreement No. HRN-A-00-98-00046-00 awarded to the Academy for Educational Development (AED). The opinions expressed herein are those of the author(s) and do not necessarily reflect the views of the U.S. Agency for International Development.

More Related