A Critical Analysis of Current M&E Frameworks

1. A Critical Analysis of Current M&E Frameworks Insights from AWID’s research on M&E frameworks By Srilatha Batliwala, Scholar Associate, AWID SBatliwala, AWID

2. First, let’s remember that: “When you work for women’s interests, it’s two steps forward - and at least one step back. And those steps back are… often evidence of your effectiveness; they represent the threat you have posed to the power structure, and its attempt to push you back. Sadly, even our ‘success stories’ are sometimes nothing more than ways the power structure is trying to accommodate and contain the threat of more fundamental change by making small concessions to us.” Sheela Patel SBatliwala, AWID

3. Assumptions in M&E Everything should be measured Everything can be measured – even complex processes of social change Measurement will enhance our ability to accelerate, deepen, replicate positive change, or improve achievement of desired goals Change is predictable – we know what it will look like, where it will occur, and how to assess it SBatliwala, AWID

4. Assumptions in M&E The macro-political environment is stable, democratic, upholds basic rights, and protects change agents and change processes – i.e., law and order, an impartial judiciary and police, due process, rights of association, civil liberties, an independent media, etc. are inevitably present in every context The organizational environment is stable and unchanging – staff don’t leave, core operational costs are secure, and you haven’t been shut down or harassed SBatliwala, AWID

5. Why M&E? • In theory, to: • Learn how change happens, make it happen faster, better, etc. • Analyze and sharpen our role in the change process • Empower our constituencies • Practice accountability – to donors, constituencies, peers, public, etc. • To advance our advocacy agenda SBatliwala, AWID

6. Why M&E? • In practice, to: • Satisfy donor demands; • Prove we’re doing what they funded us to do • Leverage more funding • Protect ourselves - from backlash, attacks, slander campaigns, etc. SBatliwala, AWID

7. What isn’t working • Very few M&E frameworks actually enable us to understand whether change has happened, or gender power altered – they measure performance(program inputs and outputs), not change • What they measure may not be indicative of the change they’re measuring at all – (e.g., GDP!) SBatliwala, AWID

8. What isn’t working • Many current frameworks are very linear, and look for simplistic cause-effect relationships (x intervention = y effect = z change) • One such, the logical framework, is often focused on performance - “[log frames] focus on the expected achievements laid out in the matrix – rather than the work itself.” , SBatliwala, AWID

9. What isn’t working • Most frameworks do not provide for tracking negative change, reversals, backlash, unexpected events, etc., that push back or shift the direction of the change process. In women’s rights work, this is vital, because the most effective work seriously challenges patriarchal / other social power structures, creating negative reactions. • Reactions / backlash / negative change is often evidence of positive impact! SBatliwala, AWID

10. What isn’t working • Most current frameworks are completely inappropriate for certain kinds of social change organizations and strategies: E.g., advocacy, training / capacity building, knowledge production, challenging discourse, etc. • Consequently, these organizations are forced to measure their processes, outreach and outputs (number programs held, number of participants, publications, attendance at rallies, etc.), rather than their impact. SBatliwala, AWID

11. What isn’t working • Several false binaries and dichotomies are embedded within or underlie many M&E approaches – e.g., “quantitative-qualitative”, “subjective-objective”, “macro-micro”, “success-failure”, and so forth. • These create problematic hierarchies rather than approaches that can integrate and transcend such dualities. SBatliwala, AWID

12. What isn’t working • Disjuncture between change measures and our time frames. The changes we are trying to track may not be visible within the time frame in which we are seeking it • Most current M&E frameworks are neither gendered nor feminist in their principles or methodology. “It takes ten years to build an organization, twenty years to build a movement, and thirty years before you see lasting impact.” Ela Bhat SBatliwala, AWID

13. Measuring Social Abstractions – the key questions What is it? Definition Where is it? Location (geographic, social, political, institutional, sectoral) What are its boundaries? Spatial, demographic, and conceptual What does it look like? Characteristics & situational analysis What can be measured? Measurable dimensions, sub-units, indicators When do we measure? Baseline + frequency of assessment SBatliwala, AWID

14. Elements of Feminist M&E • Feminist answers to the social abstraction questions • Informed by / embedded in feminist values and principles • Based on complexity • Use the best available tools (both quantitative and qualitative) • Appropriate and differentiated time frames • Designed for the level, nature of work and strategies of each organization SBatliwala, AWID

15. Principles of Feminist M&E The right of our constituency to inform, co-design and participate in the monitoring and evaluation of change processes Respect for the voice and perspective of all key stakeholders Prioritizing learning in our M&E goals Positioning M&E as a political activity Integrating political and social forces into our analysis & frameworks SBatliwala, AWID

16. Principles of Feminist M&E Avoiding attribution, assessing contribution Eschewing false binaries / dichotomies Not using M&E for punitive purposes Capturing, analyzing and addressing negative changes, reversals Willingness to abandon, revise, recast our frameworks SBatliwala, AWID

17. First steps in doing it differently: • Problem definition & situational analysis • Enunciation of our M&E principles • Development of customized M&E framework & indicators (see forthcoming AWID guideline) • Establish a baseline • Track and map • Our processes • Our performance and • External change SBatliwala, AWID

18. Social Watch Gender Equity Index • Education Gap • literacy rate • primary school enrolment • secondary school enrolment • tertiary education enrolment Empowerment Gap % of women in technical positions % of women in management and government positions % of women in parliaments % of women in ministerial posts • Economic Gap • economic activity rate • estimated earned income SBatliwala, AWID

19. SBatliwala, AWID