Social Assistance Pilots Program SA P ilot s Seminar Hybrid Means Testing (HMT) Model Development

1. Social Assistance Pilots Program SA Pilots Seminar Hybrid Means Testing (HMT) Model Development Roman Semko CASE Ukraine March, 2010

2. Content • Introduction to modeling • Data analysis • Methods for estimation • Simulations • Income from assets (agriculture) • Double-blind experiment results • Model comparisons and conclusions 2

3. Concept • The World Bank has developed a methodology for income estimation which is based on regression analysis – HYBRID MEANS TESTING (HMT) • Under HTM method, eligibility to the SA program is assessed based on the households income modeling • Total income is divided into two parts: easy to verify (e.g., pension, stipend) and hardtoverify (e.g., dividends, shadow wage) • The final goal is to estimate hard to verify share of the income based on a set of variables, which can be accurately measured and reflect the hard to verify income • Hard to verify income is divided into income which is not generated by long-term assets (estimated by regression model) and income from assets (estimated by formulas) 3

4. The main goal of the model is to predict most precisely total family income Model Criteria Equation which estimatesapplicant’s income based on the available information: Y = β1*X1 + β2*X2 + β3*X3 + … Theoretical validity Simplicity Goodness of fit Significance of explanatory variables Methods Y X1, X2, X3, … Data and Knowledge • total income • hard to verify • family structure • type and sector of employment • education • region • other Source: Finance Ministry of Ukraine 4 Application and Simulation

5. Good model should use all available relevant information for income prediction HBS 2008 Pilots dataset 10,622observations of households with total income • > 3,000 observations of families with declared income • Cannot be used separately for model estimation since total income is not available A lot of information could/should be used to guarantee acceptable level of precision Declared income (DI) is animportant indicator in total income (TI) assessment MATCHING TI Characteristics TI Characteristics Characteristics DI DI 5

6. Observations are matched in a way to guarantee the highest similarity between them Pilots dataset HBS 2008 Procedure Form groups based on the follow-ing variables: type of settlement, type of assistance, household’s size, # of children, working persons, pensioners, sex of the single-heads household Match each observations from HBS tothe observation from pilots dataset from the same groups based on the similar characteristics: age of the head, education of the head, etc. using Euclidean distance function Each observation from pilots dataset is used for matching no more than 2 times Aggregate the groups if there are no good candidate for HBS observation from corresponding group from pilots datasetand match again Observation 1 Observation 1 Observation 1 Observation 1 Observation 1 Observation 1 Group 1 Group 1 Observation … Observation … Observation … Observation … Observation … Observation … Observation L… Observation K1 Observation KN Observation LN Observation K… Observation L1 Group … Group … Group N Group N 6

7. Data comparison: a main difference between HBS and pilots applicants occurs in their incomes, while most of other characteristics are similar Total vs. declared income comparison (without SA) 7

8. For some regions average income in HBS significantly differs from the Personal Disposable Income (PDI) Statistics Statistics,PDI HBS Chernigiv Chernigiv Lutsk Lutsk Rivne Rivne Sumy Sumy Zhytomyr Zhytomyr Kyiv Kyiv Lviv Lviv Ternopil Ternopil Poltava Poltava Kharkiv Kharkiv Khmelnytsky Khmelnytsky Cherkasy Cherkasy Lugansk Lugansk Ivano- Frankivsk Ivano- Frankivsk Vinnytsa Vinnytsa Kirovograd Dnipropetrovsk Kirovograd Dnipropetrovsk Uzhgorod Donetsk Uzhgorod Donetsk Chernivtsi Chernivtsi Zaporizhzhya Zaporizhzhya Mykolayiv Mykolayiv Kherson Kherson Odesa Odesa Simferopol Simferopol Differences in income without SA per capita compared to Chernivtsi region, UAH – >200 – 100-200 – <100 8

9. Bayesian econometrics allows combining data with aggregated publications of regional PDI Calibration Bayesian estimation Standard estimation Researcher artificially determines the model coefficient(s), e.g., if regional macrodata say that income in Kyiv city is 1108 UAH higher than in AR of Crimea, than it is assumed that for Kyiv city applicants income is 1108 UAH higher than for AR of Crimea applicants, other things equal Combines both approaches. Estimated coefficient lies between calibrated and estimated in a standard way Coefficients are determined based on the collected observations using standard regression tools (classical econometrics) Does not lead to significant changes within regions but for regions across Ukraine changes are significant: average predicted income for regions has changes 9

10. Linear model is the most simple Linear • Linear relation between income and family characteristics • Dependent variable is under the logarithm (log-linear) • Independent variables (IVs) include easy to verify income • Other IVs are: number of children, of working persons, of the elderly, type and sector of employments of household heads, education level Description R2 58 % (large cities – 65%, small cities – 63%, villages – 48%) Concept: the more income the applicant declares, the lower the additional predicted income is – a sort of a “zero sum game” Predictions – declared income – predicted income 10

11. Nonlinear model is performing well when income differences are high – for the whole HBS sample NonLinear • Nonlinear relation between income and family characteristics. The form of relation: cubic or quadratic – since total income sorted in ascending order increases as a polynomial of 2nd or 3rd order • Dependent variable is under the logarithm (log-linear) • Independent variables are as in the linear model Description R2 R2-square is not bounded in [0%,100%] region Concept: as for the linear model Predictions – declared income – predicted income 11

12. Two-step model is effective when there is a large number of families with zero and nonzero hard to verify incomes Two-stage • At first stage probability that family has shadow income is estimated and then linear relations between income and family characteristics with a hazard of having shadow income is used for estimation • Dependent variable is under the logarithm (log-linear) and does not include salary Description R2 47 % (no division by cities) Concept: Stable additional income is added to the declared – “the game with constant markup”. Predictions – declared income – predicted income 12

13. Each model needs a set of adjustments in order to become fully useful Adjustments Description Informal (shadow) salary was incorporated into the dependent variable (hard to verify income) since it is not easy to verify income 1. DEPENDENT VARIABLE Some EVs which can be used for predictions are hard to verify, e.g., number of mobile phones cannot be accurately measured 2. EXPLANATORY VARIABLES (EVs) In order to compare incomes across different time period, average growth rates of PDI and its elements were used for time adjustment 3. TIME INCONSISTENCIES The definitions of family heads are standar-dized: male co-head and female co-head are used instead of voluntary definitions 4. FAMILY HEADS Prediction does not change significantly unless dependent variable is redefined. If the dependent variable is redefined, additional predicted income becomes more stable and decreases with the increase of declared income at a lower rate 13

14. Average predicted income exceeds declared by 26% Declared vs. Predicted income (by models) 14

15. 27% families will be excluded from the SA programs Number of beneficiaries (hypothetical scenario) 15

16. Average assistance will drop significantly, except for low income and fuel subsidies Average assistance (hypothetical scenario) 16

17. Total budget for SA expenditures will decrease by 27% Total expenditures on SA (hypothetical scenario) 17

18. Income from agriculture assets is calculated based on the developed normatives Current situation New approach • Agriculture income is calculated as income per hectar • Normatives are not unified across regions • Income calculation per hectar and per each animal • Differentiation between cities and villages • Normatives are unified since they are based on the same methodology and data Calculation procedure Information, certified by the village/city council Is not applied to families with disable persons or elderly (>70) If applicant lives closer than 10 km to the city – apply city normatives Income from land is a product of land area and normatives Income from payi is calculated sepatately Income from lifestock is the product of number of livestock heads times the normative Average predicted income exceed declared by 28% 18

19. Example of income calculation from agriculture CROPS ANIMALS LAND AREA NORMATIVE ANIMALS NORMATIVE AGROINCOME Only farm-stead area of 0.56 hectars, located in village (Donetsk region) 127.62 per hectar per month Possess one cow and 10 chickens Only related through the hayfields and pasturage Current situation + 63.81 UAH per month Only farm-stead area of 0.56 hectars, located in village (Donetsk region) 412.44 per hectar per month Possess one cow and 10 chickens (the same) 270.83 for cow, and 4.48 for one chicken New approach + 521.85 UAH per month 19

20. Double-blind experiment: case study Family description Model result Declared Income = 211 UAH LESS than Eligibility threshold = 255 UAH Father: unemployed and not registered in employment center Mother: housewife GRANT? WHILE Model prediction = 308 UAH immediate decision – risky family, need home visit Age: <18 Age: <3 Age: <3 Age: <3 MORE than Commission case and home visit Eligibility threshold = 255 UAH 20 DENIAL DENIAL

21. Cases of SA denials through commission, based on home inspections 21

22. Predicted income helps to select families for home inspection Families selected for inspection Comments • Each family has a chance to be selected for home inspection • The probability of selections increases as the predicted income is significantly different fromdeclared in absolute and relative terms 22

23. Model comparison 23

24. Conclusions Income estimates generated by the models significantly differ from the incomes declared by the SA applicants 1 2 Further empirical tests with the models are needed Initially model results should be used only as an advice rather than a criterion for granting SA benefits 3 The models may be used as an instrument for selecting families for home inspections 4 24