Monitoring & Evaluation of BCC Activities to Ensure LLIN Access and Use

1. Monitoring & Evaluation of BCC Activities to Ensure LLIN Access and Use Hannah Koenker (JHUCCP), Beatie Divine (CDC/PMI), Elena Olivi (PSI) The Alliance for Malaria Prevention Behavior Change Communication Workshop for Long-Lasting Insecticide-Treated Net (LLIN) Scale-Up to Universal Coverage and Use Bamako, Mali 21 – 24 September, 2010

2. Introduction • Today we’re talking about Monitoring and Evaluating BCC Activities to Increase LLIN Access andUse

3. Before delving into M&E we’d like to address determinants of LLIN use

4. Determinants/Barriers to LLIN use • What do we know • Owning more nets is associated with increased use of nets • Once knowledge is high, other factors need to be considered • Harmonized messages, disseminated through a variety of channels, is more effective than using just one channel • Nets are used more in rainy season than dry • Household visits can greatly increase use of nets, IF well implemented

5. Ways to collect information on determinants/barriers to net use • Review existing literature/studies • Qualitative study • Comparing users to non-users (segmentation) • Secondary analysis of DHS etc • Ways to NOT collect information on determinants/barriers to net use • Ask around the office and your grandma • Roll a die • First identify determinants, then use determinants to inform BCC messaging, then measure whether BCC messaging impacted those determinants • Determinants can change over time, so need to keep measuring whether they are linked to LLIN use

6. Determinants to consider • Ownership & treatment of nets • Age of person using the net, and age of caregiver where relevant • Socioeconomic status • Residence urban/rural • Risk area: epidemic/endemic; altitude • Administrative unit: district/province/region • Ethnicity/tribe • Education level (e.g. caregiver) • Distance from health facility • Intra-household access: ratio of ITNs to sleeping spaces or HH members

7. Qualitative Research • Semi-structured interviews, group discussions, drawings, etc • Identification of key themes that arise • Useful when information on determinants is lacking • Richness of data because you have flexibility; vivid picture of target audience • Useful for pregnant women • Relatively quick and low-cost • Limited ability to generalize to entire target

8. Quantitative Research --Segmentation • Divide LLIN users from LLIN users and understand what kinds of knowledge/attitudes/practices the LLIN users share • Must control for socio-economic status and other factors that influence net use • Require very big sample size in order to be statistically sound • Useful when LLIN ownership is high and many people have the possibility of using LLINs • May need to whittle down list of possible determinants with qualitative research before segmentation • Expensive and time consuming

9. Back to M&E- • Ideally, M&E of BCC activities for the net campaign should be part of the larger M&E plan for the net campaign AND • Ideally, BCC activities for the net campaign should fit into the NATIONAL BCC plans, strategies, and messages

10. Objectives of Monitoring and Evaluation of BCC Activities to Increase LLIN Use • Ensure that BCC activities are carried out as planned and reaching the target audience (Monitoring – during and post intervention)\ • Make adjustments as needed • Determine whether BCC activities had an impact on net use or other targeted behaviors (e.g. net care and repair) (Evaluation – post intervention) • Use gathered information to inform and improve future LLIN BCC messaging and activities (post intervention)

11. Tools for Monitoring and Evaluation • Monitoring: • Supervision checklist (to monitor volunteer activities) • Materials inventory, activity reports, radio logs, staff surveys, observations, supervisory reports • Rapid monitoring surveys • Evaluation: • Process evaluation (compiling reports; review meetings; interviews) • Post-campaign survey (various kinds of HH surveys – MIS, DHS, MICS, or smaller)

12. For example • Monitoring the BCC workshop

14. Forget about malaria and LLINs • There’s a new bacteria that has infested all vegetable products in Mali. If you are infected by this bacteria, you will lose all your hair. • The only protection is to ingest a very specific piece of candy 1 hour before eating vegetables. • There will be a mass distribution of candy • What is the communicator’s role and objectives in this intervention? What is our BCC intervention strategy? LUNCH IS IMMINENT!!!!!

17. Where can a communication intervention go wrong? • Radio – people not listening to radio, radio not airing spot as planned, people not receiving a high enough dose, people not remembering message, message not convincing people to change behavior • Town crier – not properly trained, inconsistent message, spreading message at the wrong time, message not convincing • Flyer – boring/forgettable image, not distributed, illiterate population, not hung in the right places, no training, ineffective message • Take away point – final evaluation at the end doesn’t tell you where the problem was. Need monitoring throughout.

18. Supervisor Checklist Registration • Is the household adequately informed of: • purpose • campaign dates, • who/what to bring • location of distribution sites • Are households informed about importance of using nets? • Key message 1, 2, 3 etc • Coupon or voucher is correctly filled out

19. Supervisor Checklist Distribution • Does the distribution site have a banner displayed? • Has the team received poster/materials and at least one is seen/posted in the area? • Distributerswelcome and set up the communitysotheycanhear the education session • Beneficiaries are informed about the importance of using the nets? • Nets prevent malaria • Air for 24 hoursbeforeusing • How to wash and dry nets • Use all year round • Mend nets whentorn • Beneficiaries receive flyers • Hangingdemonstrationisdone • Distributersaskhouseholdheads to check on net use regularly in theirhouseholds • Distributersremindmothers to seektreatmentquickly for fevers

20. BCC Monitoring Indicators – These depend on the specific BCC activities used in the net campaign.

21. Mass media – TV example

22. Sample rapid monitoring survey • Quick and dirty benchmarking • Very small sample size, cost-effective, quick results • Can be as short as a single question: Have you heard radio spot X or seen this poster? (measures reach and intensity of channels) • Can you describe the message that what was in radio spot x? • Yes/no questions (no open-ended questions, unless they can be turned into yes/no questions later)

23. Discussion about monitoring • When has your monitoring data been particularly useful? Were you able to adjust your materials/activities? • Frustrations/constraints? • Lessons learned – both good and bad

24. Post-campaign Process Assessment • No survey • Allows summary of ‘how we did’, based on compiling: • review of monitoring and supervision data; • Supervisor and observer reports; • Any rapid surveys • District review meetings • Process indicators (#s of nets distributed; # radio spots done; # of household visits) • Should include successes, weakness, lessons learned, recommendations for future

25. Post-campaign Evaluation;Country Examples • What is a post-campaign Evaluation? A survey that measures whether the objectives (including behavioral objectives) of the campaign were met • Post-campaign HH survey can be: • National level • Using existing MIS, DHS or MICS surveys to get coverage estimates (Zambia 2008) • Senegal post-campaign survey results • Sub-national level • Kano State post-campaign survey results • Other partner surveys (e.g. TRaC) • Omnibus (last resort) • Coordination of country surveys (Tanzania)

26. MIS/DHS/MICS • National HH surveys • Statistically sound • Internationally standardized • Gold standard • Usually every couple of years there is one • Advantage of MIS is that it is specific to malaria, and thus conducted in the rainy season • MIS and DHS have net rosters; MICS does not • Zambia added questions to their 2008 MIS, but that’s not common/easy • Important to consider what international HH studies are being planned before creating new one

27. Senegal • Senegal had an MIS in 2008 • In June 2009 they distributed 2.3m nets to <5. • December 2009- January 2010 conducted a post-campaign survey (dry season) • Net ownership rose from 60% to 82% • Use was 34% (general population)

28. Kano State • Post-campaign survey, 60 clusters or ~1000 households. 2 nets per household given. • 26.8% of households were not registered • 46% of hh got information about the campaign; radio messages the most heard. • “Sleep under the net every night” was the message most heard, 40% of those who remembered hearing messages cited this one. • 74% used at least one of their nets; 57% used all their nets the night before the survey.

29. Madagascar TRaC

30. Association between behavior and exposure to PSI program activities for Malaria Prevention in Madagascar Nationally (2008)Risk Group: 15 to 49 Year Old Pregnant Women and Mothers/Caregivers of Children under 5 Years of Age in Madagascar NationallyBehavior: Slept under a Treated Net Last Night

32. Draft BCC Indicators for Household Surveys • BCC indicators presented for feedback, discussed. • PMI will be testing a few of these in Uganda in the fall • MERG, the RBM Monitoring & Evaluation Reference Group, has requested JHU & PSI to work on broader set of BCC indicators regarding net use for February MERG meeting.

33. Outcome Indicators

34. Exposure Indicators

35. Knowledge & Attitude Indicators

36. Net Care & Repair Indicators

37. Dose Response Correlation of information sources and messages remembered in three post-campaign surveys in Nigeria

38. Discussion on post-campaign HH evaluation surveys • What evaluation results have you seen in your countries? • Frustrations/constraints? • Lessons learned – both good and bad

39. Campaign M&E Subcommittee • The campaign M&E subcommittee will develop the M&E plan • Your job as members of campaign BCC Subcommittee: Ensure M&E Subcommittee includes communication indicators and questions in the M&E plan, in the monitoring and supervision tools, and in any post-campaign assessment survey.

40. Other questions?

41. Groupwork: Hypothetical scenarios • Working your groups, read the hypothetical scenarios and develop a plan for monitoring and evaluating the country’s BCC activities for its campaign.

42. “Taifaja” • Taifajais located along the Sahel – one rainy season, with low net use (20%) in the dry season and 80% net use during the rainy season. • NMCP will implement a rolling campaign for universal coverage, beginning in the south where the rains come earlier, using a strategy of 2 nets per household, with a coupon given to each household 7-10 days prior to the campaign during household registration. • There are several partners who have been implementing a nationwide net use campaign since 2007, with the slogan that translates to “malaria for no one” in the local language. The slogan is well recognized. • During the campaign, flyers will be given out during household registration, radio spots will be aired locally in the regions that are carrying out distribution activities, but no TV will be used since it is nationwide, and the campaign will be rolling, covering the country’s 12 regions, two per month. Interpersonal communication will be done by the community volunteers during the registration. There will be a hang-up visit 2 days after distribution to assist with hanging and to give further messages on net use, as well as net care and repair. Due to Taifaja’slong dry season, nets get dirty quickly from the dust storms, and the average household washes their nets between 2-4 times per month.

43. Instructions for Group Work • Monitoring plan – Include indicators that will help you determine if the BCC activities are being carried out as planned and if the activities are serving their function (e.g., is the message being understood?). • It is helpful to think of how errors or activities will be corrected. • Evaluation plan – Include indicators that will help you to evaluate the reach, completeness and impact of the BCC activities, linking to the communication objectives given. • List out materials and tools needed to monitor the rollout of campaign BCC activities. • Note whether or not these elements can be included in other tools (the household registration tally sheet, for example).

44. Discussion of Group Work • Present decisions made, indicators chosen • Discuss pros and cons • Elicit questions/concerns about using the indicator list, for inclusion in next draft/toolkit

45. Thank you! AMP BCC Workshop Bamako September 2010