Measuring the impact of Microfinance

1. Measuring the impact of Microfinance P.V. Viswanath

2. Learning Goals • Does microfinance make a difference? • What are we measuring? • Why is causality so difficult to establish? Causality and Counterfactuals • How do Randomized Control Trials resolve the issue of counterfactuals? • Are Quasi-experiments the next best thing to RCTs? • How can we use regression models to investigate causality?

3. Does Microfinance make a difference? • Suppose an NGO provides microcredit to a certain population. Since the intervention required the use of (scarce) resources, it is necessary to ask if the intervention was worth it. • Let us consider the intervention of Grameen USA in Queens. Was this intervention successful? • Some of the metrics that Grameen USA uses to evaluate their intervention are: • Change in Experian credit scores – 705 increase in credit score • Income Boost – 60% increase in family incomes from $15K to $24K • Savings Increase – initial savings: zero; current saving $536 • Do these numbers prove that the Grameen intervention was successful? • Also, was this the best use for Grameen’s money? What is Grameen’s goal? • Would our answers be different if the goal were poverty alleviation? Or women’s empowerment? Or job creation?

4. Does Microfinance make a difference? • This question can be reframed as: “How have outcomes changed with microfinance intervention relative to what would have happened without the intervention?” • Coleman (2006) found that clients that would later go on to become microfinance borrowers tend to already be significantly wealthier than their nonparticipating neighbors. • If microfinance clients tend to be wealthier, then the results of the microfinance intervention for these clients relative to the poorer non-clients might simply reflect the better access of richer clients to other services that are complementary with access to credit, as well as better access to education and business training. In fact, these richer clients may well have better access to other credit, as well. • The observed differences between clients and non-clients in studies may be accounted for by the greater initial wealth of microfinance clients.

5. Does Microfinance make a difference? • Hashemi’s Bangladesh study (1997) looked at reasons why some people do not take loans. He found that they felt they could not generate adequate profits to reliably repay loans. Others opted out due to religious and social sanctions that restricted the ability to participate in meetings outside of the home with non-family males. • Thus, those who do not choose to take loans might very well do so because they do not have access to profitable investment opportunities. Hence looking at the impact of borrowers versus non-borrowers might simply reflect this access to profitable investment opportunities. • Similarly, non-participants who feel religiously more restricted might be restricted in other ways, such as the ability to participate in business transactions that are prohibited by Islamic law. • Can we start out with a sample of potential clients, all of whom wish to borrow and are credit-worthy, and then randomly lend to some and not to others? If so, then we can see if the provision of credit makes a difference. • Let’s start with some basics.

6. What are we measuring? • Children’s health • Incomes • Leisure • Wealth • Employment • Business investment • Women’s empowerment • Reduced friction in the home

7. Substitution effects and income effects • Income effects might increase consumption. If microfinance makes its clients wealthier, consumption on all sorts of goods would increase. • On the other hand, time spent on business might take away from time spent on other activities and expenditures on other activities, such as children’s health. • Hence even if microfinance works, there might not be a single unambiguous metric to measure its effectiveness.

8. What causes the effectiveness? • Is it awareness of opportunities that might be triggered by the availability of microloans in a certain village or presence of an NGO or MFI? • Is it the credit that is made available by the MFI? • Is the training that often accompanies the microloan? • Is it the communal interaction and learning from each other that might occur in programs like Grameen USA?

9. Mean effect or more? • A study might establish that there is a mean effect. • However, it might be that the effect is very skewed in the sample – what if most of the participants suffered negative effects while a few had very positive effects? • What is the channel by which the effect occurred? Knowing the channel might be as important as knowing that there was a positive effect, especially in terms of learning how to apply the treatment in other areas or situations.

10. Causality and Counterfactuals • Even after we have resolved all of these issues, there is a basic problem due to the fact that we are trying to establish causality. We want to be able to say – microfinance made things better; or microfinance made no difference; or microfinance made things worse. • We need to look at what might have happened to each program participant if s/he had not participated in the MF program. In other words, we are talking about counterfactuals – there is no way to observe this directly. • Hence we need to approximate the counterfactual. • There may be other factors that affect the observed outcome. Hence we need to adjust for these effects in order to isolate the effect of the MF treatment. • For now, let us assume that there are no other observable and measurable factors that affect the outcome of the MF treatment.

11. Counterfactuals • One way to partially address the issue of the counterfactual is to use a control group. • For example, if we wish to know the effect of a drug on obesity. • Suppose we have 30 pairs of identical twins. If we administer the drug to one twin in each pair, we can see how the treated twin’s weight changes over the treatment period and compare it to the change in the weight of the untreated or control twin. • In general, we will not have such control subjects that can be assumed to be very similar in all characteristics to the treated subject. We may also not be able to, or need to, pair each treated subject with a control subject. • Thus, in a typical microfinance setup, we may have one village where microloans have been given and another, where they have not. • Let us use T for treatment, C for control and Y for the outcome of interest, which we will assume to be income per capita. Let 0 indicate the value with the treatment and 1, the value without the treatment. Let us also assume that we are interested in the average income effect of treatment.

12. Randomized Controlled Trials • Yi1|T refers to the income of the ithtreated subject, conditional on being given the treatment. Yi0|T refers to the income of the ith treated subject, conditional on not being given the treatment. Yi1|T can be observed, but Yi0|T cannot, since a subject has to either get the treatment or not. If we’re talking about a treated subject, then we cannot observe what her income would have been without the treatment. This is the issue of the counterfactual. • Let us use E(.) to indicate averages; thus E(Yi0) is the average income of all participants at the pre-treatment instant. Then, what we are interested in, is E(Yi1-Yi0|T), i.e. for those who have been administered the treatment (i.e. given a microloan), what is the difference between their post-treatment average income and what their average income would have been had they not been given the treatment. • As already noted, this cannot be observed.

13. Randomized Controlled Trials • If we have a control group, then we can measure E(Yi0|C). Hence, we can measure E(Yi1|T) - E(Yi0|C). Algebraically, we can write:E(Yi1|T) - E(Yi0|C) = E(Yi1-Yi0|T) + E(Yi0|T) - E(Yi0|C). • The last term is what we call selection bias. • E(Yi0|T) - E(Yi0|C) is the difference between what the treatment subject’s income would be without treatment and what the control subject’s income would be without treatment. • What we need is to find some way to have this be equal to zero. Then, the observed average difference between the treatment group and the control group would be equal to the average treatment effect.

14. Randomized Controlled Trials • One way to do this is if individuals are assigned to get the treatment or not, in a random fashion. These are called Randomized Controlled Trials (RCTs). In this case, the selection bias would, on average, be zero. • For example, in the Karlan and Zinman (201) study in South Africa, the researchers identified a set of subjects whose credit scores were just below the lender’s cutoff. Of these, some were randomly given loans, while others were randomly denied loans. • In this case, looking at the difference between the incomes of those given loans and those denied loans would be equivalent to the treatment effect, since there would be no reason to believe that the no-loan income of the loan-recipients would have been different, on average, from those denied loans. • Such situations are often difficult to set up for various reasons; for one, some individuals would have to be denied treatment. If microcredit is considered good, then some people might consider it unethical to withhold microloans from individuals. What are the alternatives?

15. Quasi-experiments • Suppose instead of a control group, we have a village in which some women were given credit according to some kind of decision rule. • Can we look at the difference in income between those who received the loan and those who didn’t to estimate the treatment effect? • Only if there were no systematic differences between those who got credit and those who didn’t in a way that would be related to their income potential. • For example, if the lender chose women in large families to receive credit which is to be used for a business investment. • In that case, these women who received the loan might have more access to labor, given their large families, than those with small families who didn’t receive the loan. In this case, the measured effect would be greater than the true treatment effect because the selection bias would be positive. • Alternatively, women in large families might need to devote more time to their children and might not be able to pay as much attention to their new businesses. In this case, the selection bias would be negative.

16. Quasi-experiments • Now suppose it was not the lender that chose the loan recipients, but the loan recipients who chose to take the loans. • In this case, the more entrepreneurial villagers might have elected to take loans. Since income is probably positively related to entrepreneurial ability, the measured effect would be greater than the treatment effect; the selection bias would be positive. • Often, the micro-lender might use some kind of decision rule to pick potential recipients and then those potential recipients would decide whether or not to take a loan.

17. Unmeasured Entrepreneurial Ability Income Population Average Nonparticipants tend to come from this range Participants tend to come from this range Unmeasured Entrepreneurial Ability

18. ITT versus TOT • In this case, we’d have to decide what to measure. • We could measure income differences between those who were eligible and participated (TEP) and those who were eligible, but did not participate (TEX). We have already discussed the possible selection bias in this case. • However, we could also measure income differences between those who were eligible (TEP and TEX) versus those were not eligible in the same village (TNX). In this case, we are not measuring the treatment effect, because the TEX subset of villagers did not get the treatment. • This difference measures what is known as the Intent to Treat Effect (ITT), whereas the treatment effect that we have been discussing until now is called the Treatment on Treated (TOT) effect. • The ITT effect is also of interest and might be easier to measure because we don’t have to model the decision choice of those offered treatment; frequently, the factors affecting their decision are likely to affect the outcome measure and hence create a selection bias.

19. Example of impact evaluation strategies TNX: Not eligible CNX: Would not be eligible TEP: Participants CEX: Would be eligible TEX: Eligible, but did not participate Control Village – no microlender Treatment Village – microlender present

20. Spillover Effects • However, even if we only look at the ITT effect, there might be spillover effects. • That is, the provision of loans to one segment of the population might increase incomes throughout the village. For example, higher incomes for those given loans might lead them to buy more even from those people who were not given loans or even those not eligible for loans – either from their businesses or if these people don’t have businesses, perhaps by hiring them as workers. These people would then have higher incomes. • This would bias the measure of the ITT effect.

21. Quasi-experiments • But now suppose there is another nearby village. Could we simply look at the difference between incomes of loan-eligible in the treatment village and villagers in the control village (TEP+TEX versus CEX+CNX)? • Of course, this would not be correct because we would not be comparing apples to apples. Some of the people in the control village would not have been offered loans at all even if the microlender had operated in the village, and they might be systematically different from those who would have been offered loans. • In other words, the treatment/control variable is correlated with the outcome variable and not just through the actual treatment. • If we use the model Y = a + dT + , where Y is income, T=0 for the control group and T=1 for the treatment group, and  picks up the effect of all other factors, we can run a regression of Y on T, which is essentially the same as testing for the difference in the mean of Y between the treatment group and the control group.

22. Quasi-experiments • In the suggested test above, T=0 for CEX+CNX and T=1 for TEP+TEX. • So if the only relevant difference between T=0 observations and T=1 observations is the actual treatment, the estimated regression coefficient d correctly measures the treatment or MF effect. • However, in our case, what we are saying is that there are systematic differences between T=0 and T=1 observations that may be related to income. If we think of this variable as Z, what we are saying is that the true model is Y = a + dT + bZ + , where Z is correlated with the treatment dummy. Leaving out Z in the regression is going to cause d to pick up some of the effects of Z, thus biasing it. If we had a measure for Z, we could put it into the regression, but the problem usually is that we don’t have a measure of Z. • This was the situation in the study done by Pitt and Khandker (1998); they looked at several outcome variables, such as household expenditure, household net worth, household employment hours, children’s education etc. • Now suppose we have a variable that is correlated with Z, but is not correlated with Y. This is what we call an instrumental variable.

23. Quasi-experiments • Pitt and Khandker (1998) noted that the microlender required that borrowers have no more than 0.5 acres of land. They also postulated that having more or less land did not have any effect on the outcome variables, arguing that land ownership was simply a result of inheritance. • The solution to our problem, then, is to only pick villagers with no more than 0.5 acre of land in the control village. This allows us to separate out CNX and then we can simply compare TEP+TEX with CEX. • If we think of it in regression terms, what we are doing is using land ownership as an instrumental variable. Instead of using the actual values of T in the regression model Y = a + dT + , we regress T on Z and then use, T’, the estimated values of T from this auxiliary regression instead, so that we are using the model Y = a + dT’+  • This works out effectively, as we suggested above, to a restriction of control village observations to those with no more than half an acre of land. • In this simple case, we could have dispensed with the regression model and simply worked with the restricted sample of control village observations and tested to see if the mean value of Y differed between the TEP+TEX sample and the CEX sample.

24. Regression Models • However, we often have to have recourse to the broader regression model. For example, if we were using a control village, we’d have to make sure that the control village was similar in all essential respects to the treatment village. Usually, this is not the case. The micro-lender usually chooses treatment villages with specific criteria in mind – this is called non-random program placement. • Suppose, therefore, that there are differences between the villages, but that these differences are observable. Suppose, also, that there are differences between the villagers that might affect their income potential, but, again, that these variables are observable. • In this case, we could need to adjust for these differences by using a regression model. We’d run a regression of Y against the treatment variable and other village-specific variables and villager-specific variables. • Another way to adjust for village-specific differences is to first take the difference between TNX and TEX+TEP; and between CEX and CNX. We then take the difference between these differences.

25. Sources of Income for Treatment and Control Groups

26. Regression Models • Further, we might not be able to predict the eligible villagers in the control village perfectly. Now, we need to use the instrumental-variable regression (IVR) approach to correct for selection bias. • Note that in actual IVR, the estimates are computed in one-shot, rather than in two separate regressions. However, that is effectively what happens. • Another reason for using regression is that we might not want to simply measure treatment with a 0-1 variable. We might want to use a continuous variable, such as the amount of the loan to measure the treatment. This, again, would require the use of regression. • The complete regression model might look like this:Yij = Xija + Vjb + Tijd + ij, where i refers to loan-recipients and j to village. • The Y is the outcome measure, the X variables are individual specific, the V variables are village-specific, and T is the treatment variable, which might be the amount of the loan. Since there may be many X and V variables, they are vectors and are shown in bold.

27. Longitudinal studies • This approach can also use data collected over time (longitudinal data). • In this case, the regression model would become:DYij= DXija + DTijd+ Dijd, where the D sign indicates change over time. • The village-specific variables would drop out unless they also varied over time. • Of course, selection bias still has to be addressed using instrumental variables or by using an RCT approach.

28. Back to RCTs • RCTs are often referred to as the gold standard of research because they solve the problem of selection bias. Hence a positive estimated treatment effect can be considered evidence of causality, i.e. the treatment causes the superior outcome. • The main problem with RCTs, though, is that while they have high internal validity, they have low external validity. That is, the causality inference may indeed be appropriate for the sample; however, the purpose of research is usually to learn more about the results of applying the treatment elsewhere. • RCTs usually require special conditions with the MFI being willing to make adjustment to the researchers conditions. Also, since the cost can be high, the sample size is smaller. This is especially so if randomization is at the group or village level, as opposed to at the individual level. • Finally, with microfinance being so widespread, RCTs might be difficult to implement.