Monitoring and evaluation for social and behavioural change communication

1. Monitoring and evaluation for social and behavioural change communication Dr Renay Weiner Soul City: Institute for Health and Development Communication

2. Outline Soul City: background What do we measure? Selecting indicators Approach to evaluation Issues and challenges

3. Soul City: mission Mission is to make an improvement in people's health and quality of life by: • Harnessing the power of the mass media • Developing high quality education material through thorough research and evaluation involving the communities we serve • While remaining an organisation not for gain, ensure the sustainability of our  effort through sound business principles and practices • Being strong advocates for health and development • Measuring impact on health and development

4. SC model of SBCC • Effect change at individual, community and societal levels • Methods: MEDIA; SOCIAL MOBILISATION and ADVOCACY • Learning and Reflection; self-efficacy • Draw on different theories of SBCC

5. 5 Pillars of Health Promotion(Ottawa Charter): • Create enabling environments • Develop personal skills • Advocate for healthy public policy • Re-orientate services • Stimulate community action

6. Social Cognitive Theory • A person’s behaviour is influenced by : • Personal factors (cognitive, affective, biological events) • Environmental factors Social Learning Theory Modelling Self efficacy People learn by observing behaviour of others and the outcomes thereof People adopt behaviours if they believe they are capable. Important theory specifically for edutainment because it includes concept of traveling on a journey with characters and seeing the results of risky / negative behaviour and can role model positive behaviours and outcomes) In many instances, without social change, individual behaviour change, is almost impossible Soul City aims to impact on individual, community and societal levels to enhance those things that help behaviour change, and decrease those things that hinder.

7. Changing behaviour is a complex endeavour In many instances, without social change, individual behaviour change, is almost impossible Soul City aims to impact on individual, community and societal levels to enhance those things that help behaviour change, and decrease those things that hinder.

9. Soul City Institute for Health and Development Communication 2008-2012, Soul City is tackling two important social and public health issues in South Africa: HIV/AIDS and violence. Campaign approach • Onelove campaign: SA and regional • PhuzaWize: SA

10. example of regional level OL

11. Soul City: interventions and activities MEDIA • Soul City TV drama: 13 episodes • Soul Buddyz TV drama • Soul City radio drama: translated into all languages • Other: Kwanda; Love Stories, BOM • Print booklets SOCIAL MOBILISATION • Soul Buddyz programme • Adult training and community dialogues ADVOCACY

13. What do we monitor and evaluate? Need to consider objectives of programme/project REACH OF Onelove REACH numbers and % by age and sex geographical location province and geotype

14. Selecting indicators:deciding what to measure MEDIA SOCIAL MOBILISATION ADVOCACY Social Knowledge Community---Intervention --- Attitudes ---Behaviours---- Prevalence Individual Self-efficacy Incidence INPUT OUTCOME IMPACT PROCESS OUTCOME IMPACT OUTPUT

15. Selecting indicators Example: HIV prevention

16. Quantitative questions Relate to `quantities’ Count or measure phenomena How many? What proportion? How often? eg % of youth that abstained in past year Qualitative questions Relate to `quality’ of variations in experience or the meaning of experience for different people Classify events or phenomena How? Why? Reasons for abstaining---lack of opportunities versus choice to self-protect Approach to evaluation: quantitative and qualitative approaches

17. What do we monitor and evaluate? PROCESS EVALUATION • How was the intervention implemented? • Was the intervention implemented according to plan? • Under what conditions was implementation successful?

18. What do we monitor and evaluate? IMPACT/OUTCOMES Individual level knowledge attitudes/beliefs self-efficacy behaviours Community level -Social norms -Community cohesion/social capital -Civic action -Sustainability

19. Quantitative Aims to quantify research findings using numbers, %, ratios etc Quantify frequency of disease/health events, risks, measures associations Standardised procedures decided in advance Samples used to approximate status in the population Measures are estimates, with defined levels of certainty eg HIV prevalence: 10% (95% CI:8-12%) Random sampling allows generalisability to population Sample sizes large enough to ensure precision of estimates Evaluation: quantitative and qualitative approaches

20. Impact evaluations of SBCC require household surveys

21. Qualitative Concepts presented in written words and images Good for explaining how and why; good for exploring beliefs and behaviours Methods and Qs can change as data collected; can measure impacts that have not been predicted and generate new ideas Sampling is purposive---data not statistically generalisable Sample sizes small Analysis: extract themes, organise data into coherent findings “Yes, Desert Soul has made a difference to my life because I used to have many boyfriends. When I read this book I have changed. That means that I have dumped all my boyfriends because I read that it is dangerous to have many partners that increases the risk of being infected by HIV”. (Rural female, 26-35; FG 8) Evaluation: quantitative and qualitative approaches

22. Results: Impact - Behaviour

23. Results: Impact - Behaviour • “I have realised the importance of protected sex and have since refrained from sex without a condom. The booklet also influenced me to go for voluntary counselling and find out my HIV status” – (Lesotho) • “I never used to like condoms but now with my girlfriend we buy and use” (Zambia)

24. Quantitative approaches:RCT and evaluating mass media intervention • Cannot allocate intervention- implemented via national public media • Cannot restrict intervention-aimed for maximum reach of largest possible audience • Intervention tested in development phase through target audience research • Recent systematic review effect of communication programs on HIV/AIDS—no randomised studies for full coverage programmes (Bertrand et al)

25. Evaluation methodologies: quantitative approaches What study design to use to evaluate impact of health communication interventions? Observational Cross-sectional, case-control, cohort Intervention Quasi-experimental (pre-post), RCT (individual/cluster) `GOLD STANDARD’ randomisation

26. Evaluation methodologies: quantitative approaches What study design to use to evaluate impact of health communication interventions? Consider: level of implementation of the intervention

27. Quantitative approaches:alternative study designs • Cross-sectional analytic exposure to SC measured impact compared in exposed and non exposed; dose response relationships measured reverse causality, need to control confounders • Cohort follow-up same group over time impact compared in exposed and non exposed 3. Pre-post/quasi-experimental baseline for comparison

28. Evaluation: qualitative methods examples • Focus Group Discussions: facilitator guides group; participants have similar characteristics • Interviews in-depth: guided by the respondent semi-structured: loose set of questions • Ethnographic observation: immersion in another’s culture • Most Significant Change methodology

29. Participatory M and E AIMS • To generate data on activities being carried out at the local community level • To increase beneficiaries commitment and understanding in planning and implementing community based development projects Examples: Community Dialogues

30. Evaluation: triangulation Refers to the use of two or more different research methods in combination, principally as a check on validity SC series 7 and 8 • Quantitative data HIV/AIDS is the result of sinning Multimedia exposure Agree/strongly agree None 47.7% Low 28.2% High 19.1% • Qualitative data `To me, to watch SC changed my whole life because I was affected with HIV and AIDS .I was able to teach others to disclose to their partners….’ (female KZN, rural 16-25)

31. Evaluation: triangulation Routine data: • National media monitoring Television Audience Measurement Surveys 2. Helpline data AIDS helpline

33. Community responsesLetters and newspaper articles

34. Issues and challenges Multilevel intervention—multiple sources of data New activities---new indicators: dynamic M and E plan Reach • Media monitoring: data limited especially radio; cannot tell if same or new viewers • Survey data better but infrequent

35. Monitoring media: special issues • Need ongoing monitoring data to estimate reach • Two sources of mass media monitoring data: SAARF media monitoring Response to advice lines • Print media monitoring: distribution data

36. Media surveys: SAARF South African Advertising Research Foundation 1.AMPS: All Media Products Survey Annual national household survey; F2F interviews sample of 25 000 adults TV viewing information: past 4 weeks; past 7 days, yesterday 2. TAMS: TV Audience measurement survey `Peoplemeters’ attached to TV equipment of sampled households to monitor what is being watched Every household member must log onto the system when viewing Measures viewership patterns at home only Daily reports provided

37. Issues and challenges REACH • Print distribution as a proxy for booklet reach—survey data better • Training data: from partners —m and e support/quality audits; feedback IMPACT • For mass media cannot do RCT • Costs of surveys

38. Conclusion • Monitoring and evaluating social change communication complex • Quantitative and qualitative approaches complementary • Alternative study designs to RCT needed for evaluating media interventions