Sampling issues in PIES and PETS

1. Sampling issues inPIES and PETS

2. But if you care about impact evaluation, register for Module 6right now ! Impact Evaluation Studies • Need to compare two populations, differentiated by • Time before and afterthe crisis,the boom,the earthquake,the program, … • Space North/South,rural/urban,mountain/forest, … • Policy program vs. non-programtreatment vs. control

3. The basic question • Let y1 and y2 be the values of an indicator in populations 1 and 2 • Prevalence of malnutrition • Average income • Literacy • etc… • We are interested in the difference y2- y1 • y1 and y2 are both unknownWe estimate them from samples • The estimations ŷ1 and ŷ2are affected by sampling error • What is the error of the difference?

4. The basic answer • Let e1 and e2 be the standard errors of ŷ1 and ŷ2 • If the samples are independent*, the standard error of ŷ2 - ŷ1 is * this is the case when the populations are different

5. Panel Surveys can measure change better It seems that y2 < y1 but… …both measures are affected by sampling errors (e1 et e2) The error of the difference ŷ2 - ŷ1 is… …√ (e²1 + e²2) if the two samples are independent …only √(e²1 + e²2 – 2ρ[y1,y2]) if the sample is the same

6. Analyticaladvantages Can measure changes better Permit understanding better why things changed Permits correlating past and present behavior Analyticaldisadvantages Become progressively less representative of the population Advantages and disadvantagesof panels • Practicaladvantages • No sampling design needed for the second and subsequent surveys • Practicaldisadvantages • Sample attrition • Much harder to manage • Vulnerable to manipulation • Design them prospectively rather than in afterthought

7. Power of the test Significance level of the test One sided: Two sided: For independent samples State of the world Decision rule Null hypothesis H0 trueH0: y2 – y1 = 0 Alternative hypothesis HA trueH0: y2 – y1 = Δ H0 accepted ifŷ2 – ŷ1≤ D OK P (correctly accepting H0) = 1 - α P (Type II error) =β Type II error Researchfindingsŷ1ŷ2 H0 rejected ifŷ2 – ŷ1> D Type I error P (Type I error) =α OK P (correctly rejecting H0) =1 – β With Simple Random Samples,e²1 = σ²1/n1 and e²2 = σ²2/n2 But real-life samples are almost never SRS,design effects cannot be ignored ! The decision rule based on ŷ2 – ŷ1 intends to detect changes in one specific direction (a 1-sided test) To detect changes in any of the two directions, it should be based on | ŷ2 – ŷ1 | instead (a 2-sided test) But impact evaluation is almost always one-sided !

8. Public Expenditure Tracking Studies • PETS need to observe different kinds of units • Large administrative areas (e.g., regions) • Medium administrative areas (e.g. districts) • Service delivery facilities (schools, hospital, clinics…) • Service providers (teachers, doctors, …) • Clients (students, patients, …) • Households • Analyses need to account for their hierarchical relationships • How should the samples be selected?

9. ?

10. Top-Down(A multi-stage sample) Select regions first Then districts in the selected regions Then Hospitals in the selected districts Bottom-Up Take a sample of facilities first, This implicitly defines the sample of districts Which in turn defines the sample of regions Two approaches • Disadvantages • High design effects • Poor understanding of the upper levels • Vulnerable to unscientific choices • Disadvantage • Higher (slightly higher) costs

11. In each facility • To select providers (teachers, doctors, nurses, etc.) • The samples need to be random (perhaps stratified) • The selection has to be entrusted to fieldworkers… • …but it should be repeatable (for supervision) • Can be done with Kish tables, or with random stickers • To select clients in health facilities • The selection also needs to be random and has to be entrusted to fieldworkers. • It cannot be repeatable, but the procedures should try to avoid biases due to the • Day of the week • Time of the day