Monitoring Quality of Social & Behavioral Change Activities Tom Davis

1. Monitoring Quality of Social & Behavioral Change Activities Tom Davis Senior Specialist for SBC (TOPS) Senior Director for Program Quality Improvement (FH)

2. What methods do you use to monitor the quality of your SBC activities? Spot checks Pre/Posttests Health Facility Assessment (especially Exit Interviews) Mini-KPCs Quality Improvement and Verification Checklists

3. What is performance? Performance is how well someone does a task or job.

4. Whataretheprocessesthatyoudooverandoveragaininyourprojects? • Demonstrating proper seedling spacing Monitoring a child’s growth • Promoting immediate breastfeeding.

5. A20%improvementofpromotionofITNsandprenatalcareseeking From a 20% improvement… • WeknowthatITNusecancutchilddeathsbyabout20% • 20%increaseinthenumberofhouseholdsusinganinsecticide-treatednet • SinceITNusecoulddroptheU5MRinHaitifrom87to70(20%lower)… • …if you are working with 100,000 children, and increase coverage from 60% to 72% (20%↑), an additional 12,000 would be sleeping under a net. • Result:204children’slivesaresaved([12,000*87/1000]–[12,000*70/1000]=204 • … And many more if including other effects of antenatal care.

6. ...can cause large changes in impact. REMEMBER: Small changes in often repeated tasks...

7. Is what we do “dangerous”? There are already high “failure rates” associated with the way things are done in communities where we work with (e.g., poor crop yields, high maternal and child mortality rates). There’s a great opportunity to avoid these failures, and do things optimally, and checklists can help assure we do it right each time. FH project in Mozambique saved an estimated 6,848 children’s lives in five years at a cost of $441/life saved. A lot of things had to go right for that to happen, and checklists were used to help assure that the main things (key processes, espec. HP) were done as they should be done. • YES! How we do it can mean saved lives. If we do it wrong, people can die.

8. Why did Juan die? • Was there something that the CHW should have done differently to avoid the death? • YES: The CHW should have: • Asked the mother about the child’s illness and dietary history. • Gave the mother advice on ORS and feeding during illness. • Explained to the mother dangers signs during illness. • Asked the mother to repeat back the main messages. • A checklist could have saved Juan’s life!

9. A Quality Improvement and Verification Checklist (QIVC) is a tool that has three purposes: to encourage, • to monitor, and • to improve a development worker and the work that he or she does.

10. Reviewing the Checklist and Watching the Process • English CDW QIV Checklist (click to download):www.caregroupinfo.org/docs/CDW_Practice_Promotion_QIVC_(English).doc • French CDW QIV Checklist (click to download): www.caregroupinfo.org/docs/CDW_Practice_Promotion_QIVC_(French).doc • Watch Skit #1 (Click Here to watch home visit to Community Development Worker) [2 mins] • Watch Skit #2 (feedback session with Community Development Worker) [15 mins] • Click Here to proceed to next slide (after watching skits)

11. What should you say to the health or ag worker when you visit him or her and want to use a quality checklist? Don’t worry! This is not a test, but a tool to help you improve your work. Do your work as you would normally do it. Generally, you should not talk much with me during the educational session (or home visit, etc.) or ask me any questions. I can answer questions when you are through, but not during the educational session. I am going to observe only.

12. What comments did the Supervisor make during the visit? No comments were made. You should explain to the mothers or farmers that you are visiting the development worker to see the work being done, but you should not make any comments during the process. When you are through, you can compliment the development worker in public if she did the process more or less well. Do not give any criticism, though, in public. Your role is to observe the process.

13. Where did the Supervisor talk about each of the points in the checklist? In private with the development worker, not in front of other people

14. Why did the Supervisor explain the checklist to the worker if it is a monitoring and evaluation tool? Because it is a method for improving the worker’s performance, not just to monitor the worker’s performance. The thing that we consider to be perfect performance should not be a secret. This definition of perfect performance should be clear and consistent in the minds of all the personnel involved with the work. Because of that, you should work jointly to create and perfect a checklist.

15. Did the Supervisor speak to the person in a very threatening or reprimanding way? Why? No. The Supervisor needs to be very gentle so the development worker does not feel shame. Even if the development worker did very poorly on the checklist, you should not emphasize the score, but instead emphasize changes since the last performance evaluation – if he or she is improving or not. The purpose of the checklist is to help the worker see how s/he is improving his or her work, little by little, or quickly, as the case may be.

16. It is important to be very gentle because: • 2.We are often working with volunteers, people who are donating their time to the project and will not continue to work for us unless they are made to feel good about their work. 1. Instructing gently works. • 3.We are modeling how we want people to treat each other.

17. It is important to be very gentle because... 4.To use a new skill, a person has to have self-confidence, which is usually nurtured by other people. Self-confidence is key to behavior change in mothers, health and ag workers, and ourselves. If a health or ag worker feels shame after an evaluation, s/he may not have the confidence to change his or her behavior (i.e., performance). 5.The development workers in your charge will teach and instruct in the way that they are taught and instructed. If you are very gentle with them, they will learn to be more gentle -- and thus effective -- with others.

18. If you are using a checklist for a process that does not happen everyday (e.g., the checklist for case management of diarrhea), how could you use the checklist to monitor the person’s performance? You can use the checklist in the form of a simulation – you pretend to be a mother with a child with diarrhea or a farmer who needs to learn how to do something. With you are doing the evaluation this way, the Supervisor should fill out the checklist after the simulation. Afterwards, the Supervisor and development worker should discuss the results in the usual way.

19. Principles of Giving Feedback 2.Share the evaluation tool with the workers. Give them a copy before the evaluation. 1.The person needs to be reminded of the purpose of the evaluation. 3.Generally, give feedback in private. You can give positive feedback publicly.

20. Principles of Giving Feedback 4.Acknowledge the positive. In general, positive feedback needs to outweigh negative feedback by about three to one for the person to feel that the remarks were balanced. Look for good things to say about the person’s performance. Be encouraging.

21. Principles of Giving Feedback 5.For things done wrong or in a substandard way, ask the worker about the item. It is much easier emotionally for a person to identify his or her own mistakes than to have someone else point them out. This also gets the person in the habit of reflecting on their own performance, which they will need to do to have perfect performance.

22. Principles of Giving Feedback 6.Ask the worker to summarize what s/he is doing that is most encouraging (correct), and then what s/he can do to improve further; assure that the person heard everything. 7.Ask the worker to summarize what s/he needs to do better; assure that the person heard everything.

23. Principles of Giving Feedback 8. Ask the worker for commitment to improve those things. People are more likely to do what they commit to do verbally in front of others.

24. What sort of management information do you think this tool can provide? 1.Information onSystem-wide Problems: What problems do most workers have with the process? What parts of the process need to be restructured? On what parts of the process do workers need to be retrained? • 2.Information onPeople Problems: Which workers are having the most problems? Who needs additional supervision or support?

25. Another Example of a System Problem

26. Which of these things is more important to determine: System problems or people problems? Probably the system problems. If everyone or almost everyone is doing part of the process poorly, that could jeopardize everyone’s performance and the project’s success, especially if the part that they are doing poorly is essential to the process’ effectiveness.

27. In order to calculate a worker’s QIVC score: 1.Count how many questions have the answer, “YES.” This is the numerator. 2.Now count how many questions that you used during the evaluation.You don’t always use each line. This number is the denominator. 3.Divide the numerator by the denominator. This is the person’s quality score.

29. Click here to download all training files

30. How to use the data from the QIVC to Evaluate and Improve Your Program • Some options for improvement: • Suggest that the worker look over the training notes from a previous workshop. • Suggest the worker review the checklist. • Suggest the worker pair up with another more experiencedworker. You should use a quality checklist monthly for the workers that have an unacceptable score (e.g., less than 80%). You should continue to use the checklist each time that you visit those workers until their scores are acceptable.

31. How to use the data from the QIVC to Evaluate and Improve Your Program • You do not have to continue to use the checklist on a monthly basis with a worker once s/he achieves a high score (e.g., over 90%). You could then use it on a semi-annual or yearly basis, either with all of the workers, or a randomly chosen sample (e.g., 30 Ag. Extension Agents chosen at random) assuming that you are using it strictly for evaluation of quality and not improvement of quality.

32. How to use the data from the QIVC to Evaluate and Improve Your Program • You should use the list less frequently (e.g., every 2 or 3 months) for the workers who have acceptable scores.

33. How to Know if a Worker is Progressing, and Follow-up when Opportunities for Improvement (OFIs) are found • You should maintain a file on each worker. This file should have each workers scores on the quality checklist. The best way to keep the scores is with a graph (similar to the growth monitoring card: months on x axis, score on Y axis).

34. Click here to download QIVCL Worksheet Data

35. (Lemonjello) • Who needs the most help? (Julia) • Who improved the most over the three-month period? Looking at the November data only, who has the best performance? (Julia) • Who improved the least? • (Lemonjello, but he had a high score to begin with.)

36. If you were the Project Manager or a Supervisor, what would you do when you saw this November data? • Supervisors should ask themselves why these two are performing more poorly than the others in order to improve future training/support. Supervisors should devote more time to Julia and Oranjello. • Supervisors should figure out why Lemonjello learned the process so well so that the others can learn from him.

37. How to Identify System Problems, and Opportunities for Improving the Worker’s Skills and Follow-up when OFIs are found • Talk to workers about reasons for failure (e.g., training was unclear; not enough supervised practice; no job aide to remember steps/correct procedure). Identify parts of the process that the majority of workers are not doing properly. • Implement changes and remeasure.

38. In general, QIV Checklists are used to evaluate things that: a worker does • that have multiple steps; and • that can be observed (seen and/or heard).

39. Can this be monitored with a QIV Checklist? Yes No Management of acute respiratory infection Teaching a farmer how to use leguminous plants for fertilization and mulching Yes No Being honest in your work Yes No Prenatal education during a home visit Yes No

40. Can this be monitored with a QIV Checklist? A Ag workers ideas about problems in the community Yes No The number of farmers who have improved silos Yes No Dosing of children with vitamin A Yes No

41. Can this be monitored with a QIV Checklist? How many children have signs of anemia Yes No Teaching treatment of receptacles so insects or other pests won’t damage grain Yes No Teaching members of an association to write bylaws Yes No

42. Several Guidelines for Development of QIV Checklists Do not base quality checklists only on “problem areas” (i.e., something that is being done poorly), but on often-repeated processes. • Develop the evaluation tools jointlywith workers • Each line is a question. • Number each question.

43. Several Guidelines for Development of QIV Checklists • Order the questions in their order in the process. • If there are parts of the process that are not always done, separate them out with two lines. • Each questions should be able to be answered with a YES or NO where YES is the correct behavior (e.g., “did not make fun of the mother”).

44. Several Guidelines for Development of QIV Checklists • You can use questions that require the person filling in the form to grade performance using a scale. • Each question should be something that can be observed (seen or heard). • The questions should be detailed enough to identify specific problems and specific things being done properly. This is extremely important. • Generally, checklists should be no longer than two pages.

45. Several Guidelines for Development of QIV Checklists • You should include a space at the end for comments. • Periodically review and modify each quality checklist. Change the process based upon what seems to work and not work, and new technical knowledge. Experiment. • When asking people who are illiterate or have low literacy, try using pictures for each question.

46. Several Guidelines for Development of QIV Checklists • Work towards getting uniform performance on a particular process first, THEN make changes to it to improve it.

47. Creating draft QIVCs • Work on your own to create a QIVC for an often repeated task

48. Other Resources • List of Social & Behavioral Change Key Competencies (handout) • List of FSN Network SBC Task Force Approved Methods and Tools • Please SHARE your QIV Checklists with the FSN Network! Email me your copies and we will have the appropriate task force review it.

49. Review Questions on QIVCs 2.For what type of processes should QIV checklists be developed? 3.What are the three main purposes of QIV checklists? 1.What is performance? 4.With QIV checklists, where should the supervisor provide the majority of the feedback to the development worker?

50. Review Questions on QIVCs 6.In order for a person to feel that positive and negative feedback are properly balanced, what needs to happen? What should the balance between positive and negative feedback be like? 5.What is “positive monitoring”? 7.What is the best way to give feedback about something that a worker did wrong during a session? What should you do first?