Various Approaches to Evaluations: Perspectives from Resource Organisations

1. Various Approaches to Evaluations: Perspectives from Resource Organisations S. Galab Centre for Economic and Social Studies Hyderabad, Andhra Pradesh National Workshop on Monitoring and Evaluation of Rural Livelihoods Programmes Hall-4; VigyanBhawan, New Delhi August 12-13, 2013

2. Concept and Content of IKP Social mobilisation and formation of Self Help Groups (SHGs) and their federations at the village, mandal and district levels to generate micro-processes for influencing the institutions and policies for improving the livelihoods of the poor is central to IKP. IKP programme assumes that empowerment of poor women should come prior to the access to financial capital for the better utilization of existing resources and expanding the resource base.   This programme aims at poverty reduction in Rural Andhra Pradesh through Social Mobilization, Institution Building, Capacity Building, enhancing assets, incomes, capabilities and the ability of the poor to deal with shocks and risks. This programme has acquired the character of the model of sustainable livelihoods overtime to address the concerns of poverty alleviation in its totality.

3. Contd.. The sequential and/or simultaneous addition of different components to the basic parsimony model overtime assumes paramount importance in this process. The bottom up approach, the demand-driven approach, followed in this process has been the hallmark of this model. The continuous and constant attempts to reexamine and refine each of the components to suit to the contexts that have arisen because of the dynamics external to IKP reflect the dynamic perspective to internalize the external factors as much as possible so that the model becomes sustainable in reality. Significant changes have taken place within Velugu – now coined as IKP since its grounding through APDPIP and APRPRP. New components are added and the components that were in nascent stage in terms of their coverage have been extended to wide areas of these programmes over a period of time. Agrarian distress has been addressed through CMSA Thus, any methodology for impact assessment should take into cognizance of the changes that have taken place in the environment external to the programmes and the changes that have taken place in the components of the programmes.

4. Methodologies Sustainability Livelihoods Model: The Conceptual Framework for Impact Assessment The basic premise of Sustainable Livelihood Model (SLM) is to assess the livelihoods of poor and identify the determinants of livelihoods of poor that are perpetuating poverty and contributing to non-sustainability of livelihoods and also identify potential entry points to formulate intervention strategy for improving livelihoods of poor. As per this model, livelihood comprises the capabilities, assets (including both material and social resources) and activities required for a means of living. Assets are classified into five types-Human, Natural, Physical, Financial and Social Capital. The factors conditioning the abilities of poor for converting bundle of assets into a living include policies of State - projects, programmes and institutions (formal and informal, and market). This helps to examine the linkages between macro and micro initiatives. Further, livelihoods of poor should be sustainable.

5. Contd.. A livelihood is sustainable when it can cope with and recover from stress and shocks and maintain or enhance its capabilities and assets both now and in the future, while not undermining the natural resource base. Thus, sustainable livelihood approach shows how, in different contexts, sustainable livelihoods are achieved through access to a range of capitals, which are combined in the pursuit of different livelihood strategies. Thus, the functional model of sustainable livelihoods consist five key components - Livelihood assets; Vulnerability of livelihoods to external factors; Conditioning variables that influence the ability to convert livelihood assets into a living; Livelihood strategies; and Livelihood outcomes. The improvement in the quality of life due to the programmes should be sustained. Indications of well-being are located in domain of credit, assets and livelihoods and organizations of poor women. Indicators in different domains considered for assessing the sustainability.

6. Sample Design Multi stage stratified Sample Design- Districts, Mandals, Villages and Households Representation from the Three Regions (APDPIP) Telangana, Rayalaseema and Costal Andhra Representation from Five agro-climatic zones (APRPRP) Instruments of Data Collection Household- Male and Female Questionnaires Separately Village Questionnaires SHGs, VOs, MSs, DSs Livelihoods Questionnaire separately

7. Different Methods Adopted to Assess the Impact Double Difference Method The traditional DOUBLE DIFFERENCE METHOD is useful to assess the impact of the programme if there are strictly comparable control areas. Comparison of programme areas with that of control areas will facilitate to assess the impact of the programme. However, when programme became universal; participant and non-participant approach was adopted in assessing the impact of APDPIP.

8. Methodology Adopted for the Impact Assessment of APDPIP is as follows Impact evaluation based on three rounds of surveys- Baseline, FUS-I and FUS-II FUS-I developed many parameters of impact evaluation independent of the BLS FUS-I also used recall method to construct BL data in some of the impact areas Cross section and panel survey methods used in FUS-I All the FUS I sample HHs revisited in FUS II Comparison of FUS-II with FUS-I made to assess the contribution of the programme Phase I and II mandals considered for the impact assessment Programme became universal, Program and Control samples strictly not comparable Followed participants and non-participants of the programme Used different methods in the Mid-term Appraisal to assess the contribution

9. Important Findings of APDPIP Final Evaluation Empowerment gone up significantly at the household and community level. Quantity and quality of credit has gone up. Women’s share in savings and borrowing capacity gone up Formal credit institution have become inclusive of poor Risks managed well by resorting to savings and borrowings Well being improved

10. Contd.. However, the traditional DOUBLE DIFFERENCE METHOD suffers from its inadequacy in accounting for the spillover benefits, generated through externalities of the programme, accrued to the poor households with lowest duration of membership in the programme as well as to the other households with different duration of membership. In the traditional DOUBLE DIFFERENCE METHOD, the “untreated” are assumed to be completely unaffected by who else gets treatment. In any social setting, this assumption is very strong, and many economists have built models to account for various versions of social interactions and their consequences for policy evaluation. In the IKP programme, the participants negotiate with the state, market and civil society. These had resulted in better functioning of formal institutions that provide public goods and services, elimination of age-old undesirable informal social practices like child marriages, child labour etc., and making the markets, (like credit, labour, land and product) more competitive and also democratising the intra-household relations. Thus these changes brought in public and private spaces benefits the untreated as well as treated households.

11. Modified Double Difference Method to Capture Spillover EffectsThe Modified Double Difference (DD*) = The Traditional Double Difference (DD) +Spillover Effects. The spillover effects are caused by the externalities generated by the programme. For example, the program may have induced a reduction of interest rates or increase in wages in the community which in turn would benefit all the households including non-participants. The externalities generated in the credit market are considered for estimating the extent of externalities generated in the project area. We assume the same degree of externalities to prevail in the case of all the institutions (public institutions) with which all the economic and social groups interact. The basic premise of the programme is to provide quality credit - credit with low transaction cost and reasonable interest rates. For example, the program may have induced a reduction of interest rates in the community which in turn would benefit all the households including non-participants. This is possible if the flow of credit from the formal institution should get enhanced due to IKP.

12. Contd.. Thus the share of increased credit from formal institutions in the increased total credit between Baseline survey (BLS) and Follow up Surveys (FUS) should go up. The analysis of APRPRP is evidence to this and the gist of methodology used is as follows: This is denoted by α, which is derived as a ratio of change in the credit amount from formal institutions between BLS and Follow-up Survey to change in total credit from all sources between BLS and FUS. It takes values between zero to one. The α takes value one if the increase in the total credit comes from only the formal institution between BLS and FUS. This indicates that the programme has generated full externalities. The α takes value zero if the increase in the total credit has not come from the formal institutions at all. This indicates that the programme has not generated externalities at all. The α takes values between zero and one, if part of the increase in total credit has come from the formal institutions. The mathematical representation of these formulations is in order. Let ∆1 = gains of Participant household ∆2 = gains of Non- Participant household N1 = number of Participants households N2 = number of Non-Participants households α = fraction of gains of the non-participant households that can be attributed to the program due to externalities. Gain to participant household due to program = ∆1 – (1-α)* ∆2 Gain to non-participant household due to program = α*∆2

13. Contd.. Variant-1 = Program Does Not Generate Any Externalities i.e. α =0 Gain to participant household due to program = ∆1 –∆2 Gain to non-participant household due to program = 0 Per household gain to all those in the project area due to program = N1*(∆1 –∆2)/ (N1+N2) This is the upper limit of the impact of the programme. Variant-2 = Program Generates Full Externalities i.e. α =1 Gain to participant household due to program = ∆1 Gain to non-participant household due to program = ∆2 Per household gain to all due to program = [N1 *∆1 + N2*∆2]/ (N1+N2)   Variant-3 = Program Generates Partial Externalities i.e. 0<α<1 In this case, a fraction of the gains of non-participant can be attributed to the program due to program externalities. Gain to participant household due to program = ∆1 – (1- α)*∆2 Gain to non-participant household due to program = α*∆2 Per household gain to all those in the project area due to program = [N1 (∆1 – (1- α)*∆2) + N2 α*∆2]/ (N1+N2) In the impact analysis of APRPRP under IKP, we confined to variant 3 only, because the reality lies between the variant 1 and variant 2.

14. Important findingsof APRPRP Final Impact Evaluation The programme has contributed to the increase in the incomes of the households through enhancing the access to formal as well as informal credit, enhancing asset base especially the land and enhancing women’s empowerment. Participation in the programme itself has contributed significantly to the incremental income. However, BC and OC households derived more benefits. The households with initial higher incomes have derived less incremental income. This indicates that this programme has contributed to reduction in income inequalities. The households with higher increase in the change of worker-dependency ratio have derived less incremental income from the programme. The increase in the change of worker- dependency ratio reduces the incremental income. An increase in the change of score of women empowerment by one unit enhances the incremental income by Rs.1046 for the participant households as a whole. It benefited more for the poor from SCs followed by poor from STs. An increase of one acre of land in the change of land had resulted in an increase of Rs.1159 in the incremental income for the all participant households as whole. This has more pronounced for SCs and BCs.

15. Constructing of Matching Control Group with Treated Groups One can also use the following two methods to construct Control Groups The Propensity Score Matching The households who have entered late into the programme may be the poorest of poor. A comparison between households of longer duration in the programme who may be relatively better with that of late entry households may provide over estimate of the impact of the programme. The technique of Propensity Score Matching method will facilitate to identify the appropriate control group. This also helps us to overcome the problem of selection bias.  Constructing of Matching Control Group with villages in the neighboring state Another method we proposed to IKP was to indentify a mandal / block in each of the neighboring state which will be more or less similar in respect of important parameters with that of nearby sample mandal within the state of Andhra Pradesh. The proposed neighboring mandals/blocks include – one each in Orissa (adjacent to Srikakulam), Karnataka (bordering Anantapur), and in Maharashtra (bordering Adilabad) for the 1st phase of IKP (Veluguprogramme) i.e. APDPIP. In each of the new mandals/blocks proposed, villages and households will be selected following the methodology adopted for the APRPRP.

16. Mini-Mid-Term Appraisal of APDPIP Mini Mid-Term Appraisal (MMTA) during 2002-2003 as a prelude to Mid-Term Appraisal of APDPIP was conducted. The objective of the MMTA was to assess the effectiveness of the seeds of change (i.e. APDPIP interventions). The basic approach of the study was to assess the changes that have taken place at the household, community and village / habitation level. It has combined Participatory Rural Appraisal (PRA), Focused Group Discussions (FGDs) and Case study methods to assess the impact of APDPIP. Thus, the study relied on survey and non-survey methods. The parameters to assess the changes in the livelihoods of the poor and the PoP have been derived against the backdrop of the intervention logic of the DPIP. They are configurated in the sustainable livelihood framework, an analytical framework, to understand the changes in poverty in all its forms among the poor and the PoP covered due to the seeds of change. A semi-structured questionnaire was administered to the sample households for obtaining information on livelihood capitals.

17. Contd.. Framework interview has been organized to collect information on vulnerability of the households. A ten-seed scoring exercise has been conducted to collect information on the livelihood strategies from the select households. Similarly, a trend line exercise has been organized to collect information on livelihood outcomes from the households. Focus group discussions (FGDs) were conducted with the select SHGs, VOs, and MSs to collect information on the initiatives undertaken by these groups and their impact on the formal and informal institutions that have a bearing on the livelihoods of the poor and the poorest. Similarly, FGDs were also conducted to collect information on covariant risks encountered by the communities and on empowerment of women. The livelihood impacts of district specific CIF- SPs were also covered in the non-BLS districts i.e., Vizianagaram, Chittoor and Mahaboobnagar. Non-poor households were interviewed to assess their perceptions on the livelihood impacts of DPIP.

18. Thanks