Training Performance & Outcome Evaluation

1. Training Performance & Outcome Evaluation Lao & Thailand MID-BCC Project January, 2012

2. Table of Contents

3. Background

4. Study ObjectivesBetween 2009 and 2011, in cooperation with local partner organizations, the MID-BCC project has undertaken capacity building for infectious diseases in Laos and Thailand. Over 600 health workers, community leaders and other stakeholders have been trained at the district level. An impact evaluation was needed to determine how effective the training has been and what communication activities have been undertaken as a result of the training. Specific objectives have been set out as follows: • Determine whether trained community or village leaders have conducted relevant communication activities at the commune level; • Identify activities trained community or village leaders are doing in relation to different infectious diseases; • Highlight the audiences that have been targeted through these activities; • Identify the extent to which messages communicated have been focused on planning, prevention and response to infectious diseases; • Identify issues or problems faced in conducting these activities and how they have overcome these problems; and, • Compare differences between specific target groups.

5. Two Way Evaluation Approach • The two way evaluation approach, where training performance is linked to outcomes, can be depicted as follows: Training Program Participants Outcome (Actions taken) Link Their perspective of the training (Execution)

6. Methodology

7. Survey Details • The survey is based on feedback from people who have undertaken training in the period 2009 and 2011. Respondents were selected from a participant list provided by FHI360. • The respondent list provided by FHI360 had a total of 630 records. Of these 454 were up to date with usable contact numbers. An attempt was made to contact each and every person on the list. • Whilst there were no refusals, many of the training participants were not available at the time they were contacted for an interview. Still, the completion rate is very good at 60 percent.

8. Target Group • The respondents were selected from participant lists provided by FHI360. Before conducting the interview it was verified that the person actually had attended the training. • Most of the participants were human health workers followed by village leaders and members of the LWU.

9. Overall Training Performance

10. Training Performance Index weak Strong Performance Training Performance Index (TPI) • In order to have a simple mean of measuring and monitoring performance, a Training Performance Index (TPI) has been developed. The TPI is measured independently and focuses on how participants perceive the delivery and usefulness of the training. The TPI is is multi-dimensional and consists of four components including: overall performance, recommend to others, likelihood to participate in the future, and comparative usefulness of the training. Overall performance Future participation Recommend to others Comparative usefulness

11. Training Performance Index (TPI) Overall • The overall TPI for the FHI360 in 2009 was 108. For the 2011 evaluation it increased further to 114. That is a very good score and means the focus of future training efforts should focus on maintaining performance rather than improvements. TPI Excellent Performance Good Performance Improvement Zone FUNDING CRISIS Base: All respondents, n=272

12. Training Performance Index (TPI) by Sex and Segment • There was good representation of both male and female participants but most have been males. Females rate the trainings a bit lower than males. • Participants were grouped into 5 segments. It is interesting to see that the TPI differs only marginally between the groups with provincial office an LWU participants scoring somewhat lower compared to the other segments. % TPI % TPI Base: All respondents, n=272

13. Training Performance Index (TPI) by Region • The most distinct differences can be seen across regions with Thailand scores a bit lower with a TPI at 96. Participants in Savannakhet are marginally more satisfied with the training compared to other regions in Laos but across the board the scores are high. • Looking at the regional level the trainings have been relatively more appreciated at the village level. % TPI % TPI Base: All respondents, n=272

14. Training Performance Index (TPI) by Disease Focus • The respondents were asked which disease the training mainly focused on and for the vast majority of cases the focus was on AI. The malaria training was done in Thailand (Phuket) and whilst the TPI is still high at 94 it scored a bit lower compared to the trainings focusing on Dengue and AI. Base: All respondents, n=272

15. Training Performance Index (TPI) by Time Period and Age • Most of the people interviewed were trained in 2011 (63%). There is a bit of a drop in the TPI score moving from the 2009/2010 period to 2011. Many of the trainings focused on AI and this decline may also be influenced by the decline in AI as a ‘hot topic’. • Across different age groups the difference is less distinct but there is a tendency for older participants to rate the trainings a bit more favorably. % TPI % TPI Base: All respondents, n=272

16. Training Outcomes

17. Source of Information about infectious diseases – Overall • Virtually all participants have received information about infectious diseases, mostly from mass media such as television and radio but posters and billboards as well as word of mouth are also important information sources. Community forums and events are the media sources that have less influence. Base: All respondents, n=272

18. Prevention, Planning or Response Action taken • Virtually all participants (97%) have taken some action related to prevention, planning or response following the training. However, the proportion of participants who have engaged in regular activities, once per month or more often, are relatively few (17%). Frequency of Activities Relatively few regular activities Base: Those taking action, n=265 Base: All respondents, n=272

19. Communication Target – Overall • The target groups for communication activities extend across three tiers. First comes immediate family and neighbors with just over 90 percent. Secondly, people from the same village but now the effort drops to under 50 percent. Finally, supply chain, district officials and agents with a reach below 20%. The three tiers are most likely linked to the amount of effort required to reach the different target groups. Easy Effort Level Difficult Base: Those taking action, n=265

20. Activities Undertaken – Overall • On average, participants have engaged in two to three activities. Group discussions and club meetings have been the most common activities undertaken followed by putting up posters and distribute leaflets or booklets. Around one third have undertaken some form of training and one in five have promoted the issue at a community gathering. Base: Those taking action, n=265

21. Frequency of Communication – By Segment • In reference to activities carried out respondents were asked how frequently they had carried out these activities. For the purpose of clarity, frequency has been grouped into occasional and regular activities, with regular meaning at least once per month or more often. Animal health workers and those at the village level have on average undertaken activities less frequently. Overall Human Health Workers Animal Health Workers Province Level Lao Women’s Union District Level Village Leaders Village Level Provincial Office Regularly, once per month or more often Occasionally, less than once per month Base: Those taking action, n=265

22. Communication Messages – Overall • With respect to communication messages, a distinction was made between messages that focus on response, planning or prevention. And in addition to that, these were connected to different infectious diseases, namely: AI, Dengue and Malaria. • Not surprisingly perhaps, most of the messages have been in relation to AI. Regardless of the type of disease, most messages have been in relation to prevention. Base: Those taking action, n=265

23. Cross Disease Message Applications – Overall • What is learnt in a training focused on communication can also be applied to other thematic areas, in case, diseases. When looking at how the training has translated into different disease messages, it is evident that participants have gone beyond the focus of the training and applied what they learned to other diseases as well. • Those who undertook training for Malaria or Dengue have applied their communication skills to other diseases to a greater extent compared to those who did training on AI. Base: Those taking action, n=265

24. Traffic Light Parameters • In order to make an evaluation across segments, a traffic light system is used based on overall compliance with different outcomes. • Intervals below are the same as set in 2009 0-40 % Low 40-60 % Medium 60 + % High

25. Summary Traffic Lights – Region • The training program received a somewhat lower score from participants at the province and district level and may indicate that a different approach is needed for these groups. They have relatively good access to information about infectious diseases, However, regular communication is lacking and overall reach is relatively lower. A possible explanation for this is that government health promotion staff and members of the IEC teams do not always have the ability to cascade similar training or conduct campaign because the provincial or district office does not have the funds for such activities. • At the village level, access to information, target reach and frequency of communication declines significantly. To some extent this may be expected but there should be room for improvement. Base: All respondents, n=272

26. Summary Traffic Lights – Target Segment • Target reach, number of activities and regular communication are the main challenges. Reach may be difficult for some groups, especially at the village level, and again could be linked to funding issues. • It is interesting that whilst reach and regular communication is low, most groups have got their full range of messages right. For future trainings it may be worth while to place more emphasize on message dissemination, looking at target audience selection and effective media strategies, especially in situations where there is limited funding. Base: All respondents, n=272

27. Training Performance Drivers

28. For this study, all key capacity building areas have been included. The six capacity building areas were originally identified and agreed after discussions with FHI360 local office in Laos and regional office in Bangkok in 2009. The capacity building areas have been defined around different aspects of the training with focus on different activities. With respect to some of the individual training activities, however, some adjustments were necessary. Key Capacity Building Areas Evaluated

29. Training Attributes Measured • Formulated around the 6 major capacity development areas, specific activity based attributes were developed in cooperation with FHI360 to enable good insight into perceived training performance. By understanding which training aspects help to drive performance, FHI360 is able to formulate effective capacity building strategies. PREPARATION Information about the training beforehand Training location The training schedule The mix of people attending the training Organization of the training Organization conducting the training Adequate training duration TRAINING MATERIALS Broad range of different training materials (posters, flip charts, CDs etc.) High quality materials Effective use of materials Sufficient quantity of materials for distribution TRAINING CONTENT ON COMMUNICATION Interpersonal communication Effective use of communication materials How to work with journalists TRAINING CONTENT General understanding about diseases How to recognize symptoms in animals How to recognize symptoms in humans Impact of diseases on animals and humans Methods of preventing disease among animals Methods of preventing disease among humans What to do if there is an outbreak ABILITY TO IMPLEMENT Give me confidence to talk about the subject Easy to implement Easy to communicate concepts to others Easy to use support materials Prepare me to answer difficult questions Easy to follow training outline / training guide TRAINERS Knowledgeable on the subject Well prepared Make me think about the subject Engaging Good ability to communicate Listens carefully to questions

30. Decision Matrix • Based on the two dimensions, each activity (as well as Intervention area) can be examined to determine what the best recommendation for future action should be. • The decision matrix feeds straight into the programme strategy and it is particularly important to look at the first row to determine key improvement areas as well as positive strengths for which further investment may be warranted.

31. Overall Training Impact • Whilst the trainers and training materials used were given high ratings on average, several aspects on content were lower in relative terms. What is interesting is that ‘ability to implement’ falls in the ‘low impact’ zone with a below average rating. It may be worth while considering to shifting the content to focus a bit more on implementation as this also has been seen as a relatively weaker area in terms of outcomes. Content Content Communication High Impact Zone Content Diseases Preparation Training Materials Trainers Ability to Implement Low Impact Zone Execution Base: All respondents, n=272

32. Recommendations Excellent Trainers Training materials - Keep the trainers and continue with the same implementation partners. - Maintain the training materials both in terms of quality and quantity handed out to participants Very Good Preparation - Preparations are very good and should be maintained. It is important to advice participants as soon as possible should there be changes to the training schedules. Improve Content communication Content disease Ability to implement • Content related to communication skills rated high, but there is scope to improve how to effectively use the communication materials and how to work with journalists. • The focus of the training is on communication and there is potentially scope to build in more disease focus. • Most trainings focused on AI but there has not been any recent outbreaks. Hence, there is no urgency to take immediate action.

33. For more information please contact: Cecile M. Lantican, PhD Country Coordinator, Lao PDR FHI360 Tel/Fax: (+856) 21 240 315 Email:cmlantican@fhi360.org Conducted by Rapid Asia for FHI 360