Lessons learned 1: Importance of having clear conceptual framework to guide evaluation

1. Lessons learned 1: Importance of having clear conceptual framework to guide evaluation

2. SASA! Intervention Uganda • Clear conceptual framework for intervention about process of change trying to achieve in community • Stages of intervention focus and associated activities clearly articulated and translated into programming • Used to inform M&E design SASA! Phases / Process of Change

3. SASA! Logic Model Levels of SASA! Activities reaching each Circle of Influence Longer term outcomes Impact Context Initial Outcomes Intermediate Outcomes Socio-demographic Factors Sex Age Income Education Employment Religion Place of residence Socio-cultural Factors Family Characteristics Social Support Alcohol Use • Knowledge • Recognizing VAW a problem • Types of VAW • Consequences of VAW • VAW/HIV linkage • Skills • Response to women experiencing violence • Hold men accountable • Promote balanced power • Support to activists • Individual & Collective Capacity • Supportive environment • Enhanced ability to prevent and respond to VAW Reduced social acceptance of VAW Decrease in experience / perpetration of VAW Improved response to women experiencing violence Decrease in HIV/RH risk behaviors Societal Nat’l policy makers, media Community Police, LCs, HCPs, NGOs, Ssengas, RLs, • Action • Intention to act • Personal change: balancing power • Public change: sanctions ag VAW • Acceptability of expanded gender roles • Awareness • Root cause as imbalance of power between women and men • Change can happen • Behaviors • Balancing power • Communication with partner • Decreased risk behaviors • Community activism Relationship Relatives, elders, neighbors • Critical Thinking and Dialogue • Public debate and discussion • Personal reflection • Acceptance and Influence • Attitudes toward power, gender, human rights • Silence broken • Sustained Action • Changed policies • Organized groups • Changed practice in relationships, community, institutions Individual Women, men, youth, CAs • Participation • Activists at grassroots, in leadership, in institutions • Connection • Activists/leaders/professionals increasingly connected and active SASA! Phases / Process of Change

4. Key elements in conceptual framework • What are the range of potential exposures people may have to intervention? • What is expected process of change? • What are likely pathways of change? • Among individuals receiving the intervention • In the broader community • What is a realistic timeframe for this change to occur? • How do we measure this?

5. Lessons learned 2: Power of multiple forms of evidEnce (IMAGE STUDY)

6. Did IMAGE affect MFIs financial performance? • 50% reduction in centre “vulnerability” = attendance, arrears, savings • Impact persisted 2 years after intervention completed • Similar impact on drop out rates.

7. Past year experience of intimate partner violence reduced by 55% (aRR 0.45 95% CI 0.23-0.91) Adjusted risk ratio 0.1 1 10 Past year experience of physical/sexual partner violence - pushed you - hit you - forced sex - scared to say no to sex 0.45 (0.23-0.91) Pronyk et al. The Lancet Dec. 2006

8. How was violence reduced? Changes in women’s relationships “Now that we have money we are able to say how we feel without fearing that your husband will stop supporting you.” “You can buy him cigarettes from your profit. Because of SEF’s money we are experiencing fewer problems in our households” Women supporting women “We are able to overcome abuse when we are in SEF because we get support from the women in the groups. When you engage yourself with other women and listen to their problems that will help you to cope”

9. LESsON learned 3: COSTING important part of INTERVENTION evaluation

10. What does costing do? • Document all inputs into intervention • Capital & recurrent expenditure • Ingredients approach to costing puts a financial value to all inputs: • Monetary • Non-monetary • Can be used to: • Look at break down of costs by activity • Estimate unit costs • Estimate cost-effectiveness

11. Scaling up IMAGE in South Africa Scale-up Additional cost = US $13/client Pilot Study Additional cost = US $43/client 2001-2004 2005-2007 2008-2010 430 households 4500 households (30,000) 15 000 households (80,000)

12. CHALLENGES 1. Reporting bias

13. Physical and/or sexual partner violence against women at baseline – SASA! study in past 12 months

14. The potential for reporting bias about gender norms & violence • Women tend to under-report experiencing violence • May be more willing to disclose following exposure to intervention • Could find higher disclosure in intervention arms • Men may not disclose perpetrating less socially condoned acts of violence • May be even less willing to report disclosure following contact with intervention • Could find higher disclosure in intervention arms

15. Importance of choosing quantitative outcome measures carefully Rely on women’s reports on experiencing of violence, rather than men’s reports of perpetration If find reduction, effect less likely to be due to reporting bias Use qualitative research to triangulate Implications for evaluation

16. CHALLENGE 2: Capturing Social mobilisation difficult and unpredictable

17. IMAGE: community mobilisation led to a diverse range of activities • Many focused on violence and HIV • Also focused on other issues of concern to women • Difficult to fully document all of activities Community mobilization: • 40 village workshops • 16 meetings with local leaders • 5 public marches • 2 partnerships with local institutions • 2 new village committees target Crime and Rape

18. Timelines as part of M&E activities in Uganda • Periodically using timeline with key informants • Discuss key actions and key events in the community • Aim to map out scale and geographical diversity of activities over time

19. CHALLENGE 3: EVALUATION DIFFICULT

20. Getting reliable evidence on impact is not easy • Getting strong quantitative data on impact difficult: • RCTs gold standard study design, but expensive and technically difficult • Require partnership between intervention agencies & research organisations • May be rare that can show intervention impact on HIV or VAW directly • Seeking to evaluate complex interventions on complex issue but: • Trial design generally focuses on limited number of impacts – need to identify primary outcomes • Forms of social change may have multiple benefits • Limited evidence about what may be reasonable timeframes for different forms of change • May be lack of recognition of importance of change in pathway variables

21. The challenge of building a strong evidence base • Data from control communities important to help attribute impact • Challenge of getting sufficient number of study and control communities • Difficult to identify and maintain control populations • May not be feasible if intervention national level • Difficult to identify and maintain control populations • Difficult to fund evaluation research: • IMAGE had 9 + donors • SASA! funding from 3 donors for baseline alone • GBV not primary focus of most research donors • Leads to under-powered studies • Less focus on statistical significance

22. conclusions

23. The value of good M&E evidence • Growing body of experience, but extremely limited evidence about intervention impact on VAW & IPV • Importance of M&E to provide insights about: • Process & scale of project implementation • Impact on recipients and broader community • Resources / costs required • Multiple forms of evidence important: • Quantitative - scale of activity, magnitude of impacts • Qualitative – acceptability, what happened, what means, quality, role of key players, unexpected events • Economic - forms of input, economic value, potential costs if replicate • Lessons from M&E important to inform • ongoing programming • replication locally, nationally & internationally

24. The challenge of building a strong evidence base • Fundamental challenge of how to move from documenting levels of activity to level of change • Getting strong data on impact difficult: • RCTs expensive and technically difficult • Require partnership between intervention agencies & research organisations • May be rare that can show intervention impact on HIV or VAW directly • Seeking to evaluate complex interventions on complex issue: • Timeframe for change my be longer than evaluation funding • Importance of capturing broad benefits of social empowerment interventions • Mixed methods likely to provide most reliable evidence • Draws on strengths of each approach • Provides opportunities for triangulation

25. The way forward? • Document experience and lessons learned • From broader field of evaluation research – community interventions, service delivery, media interventions • From current M&E of different models of GBV intervention • Experience combining quantitative and qualitative methods • About realistic timeframes and process of change • Lobby for strategic investment in multi-disciplinary evaluation research: • Women’s empowerment • School based programmes • Community mobilisation • Masculinities • Alcohol • Service provision • Role of legal and policy reform • Recognise need for methodological work • Use evaluation research to learn about pathways of change, and broader lessons for M&E • How to conceptualise, define and measure key variables • Integrating participatory evaluation methods with quant evaluation frameworks