Paul Bolton Applied Mental Health Research Group Johns Hopkins Bloomberg School of Public Health

1. Public health systems: holding governments accountable. Establishing standards, measuring implementation Paul Bolton Applied Mental Health Research Group Johns Hopkins Bloomberg School of Public Health Baltimore, USA.

3. Outcome Measures • Survivors participate in the rehabilitation process (GC3) • Programmes…take into account a victim’s culture, personality, history and background (GC3) • So how can we have standard instruments/measures when client needs and situations vary?

4. Some mental health problems are both predictable and similar: • Depression (Hopkins Symptom Checklist or HSCL) • Anxiety (HSCL) • Trauma (Harvard Trauma Questionnaire or HTQ) But others vary

5. A Qualitative Approach to outcome measures • Start with free listing asking: ‘what are the problems of people who have been tortured?’ • List all commonly mentioned problems, then choose priority problems based on frequency and severity. • Key informant interviews to explore those problems.

6. Feeling handicapped • Social injustice. We are not treated equally • Divorce • Poverty • Drinking alcohol • We are not respected as we should be. We feel inferior • We are insulted; especially women and girls are called names, that you are raped. • We regret helping this government. • Social relationships have become weak. • Disappointment. Nothing has been done for us. • Rage • No-one is honest. So we are obliged to do bad things

7. Qualitative Problem descriptions used to: • Other problems are included as separate questions • Add to depression/anxiety/trauma instruments to increase local validity • Thinking too much • Cannot accept person is gone • hating the world

8. Qualitative Problem descriptions used to: • Used for translation • Decide on appropriate interventions

9. Program Monitoring • Monitoring – constantly tracking indicator. • Purpose of monitoring is to identify and address problems as they occur (iterative) • Evaluation – determining if there is a change in the indicator between program beginning and end. • However: Everything that is evaluated is also monitored.

10. program monitoring and evaluation for “effective Implementation”*† Fidelity Availability/Access* Uptake Survivor Compliance/Cooperation Appropriateness/Acceptability* Feasibility Cost Effectiveness* †Dissemination and Implementation Research

11. Fidelity Monitoring

12. Accessibility/Reach Can be defined in various ways: distance cost time opportunity cost.

13. Uptake • How many of those who have access to services and know about them, try them? • Can be defined as: Uptake Uptake

14. Compliance/cooperation • How many survivors who begin treatment complete it? Compliance

15. Appropriateness/Acceptability • Combination of uptake and compliance: • If high uptake and compliance, program is considered acceptable. • If either is low, acceptability is considered low.

16. Feasibility • Requires a vision of who is going to pay for the services for as long as they are needed. • Can this payer(s) afford to pay for the duration? • Is this payer(s) willing to pay for the duration?

17. Effectiveness • Does NOT refer to whether survivors improve. Refers to what would happen to the survivor in the absence of services: survivor would be worse off: effective survivor would be the same: not effective survivor would be better off: not effective and harmful.

18. Reduction of depression symptoms by group Depression Symptom Scores

19. Reduction of depression symptoms by group Depression Symptom Scores

20. Effectiveness: what outcomes? • “Rehabilitation…refers to the restoration of function or the acquisition of new skills required as a result of the changed circumstances of a victim in the aftermath of torture or ill-treatment.” • Ultimate goal is restoration of dignity. • Interpretation: Main aim is the restoration of survivor’s roles in terms of self, family and society.

21. How to operationalize this? Main impact outcomes are the functions that make up locally defined roles. These vary, so there is no single instrument. Replace single instrument with process that has survivors define them locally.

22. Function Assessment • Qualitative methods: • Free listing • What are the activities that men/women normally do (to take care of themselves/family/community? • What do children normally do? • How do you know when a man/woman/child is doing well. • What are the activities that survivors cannot do that they need to do?

23. Iraq (women) • Housework • Cooking • Other manual labor • Caring for family members • Giving advice • Exchanging ideas • Having harmonious relationship with family • Raising children correctly • Contributing to the community • Sympathize with others • Visiting and socializing • Asking for help • Getting help • Making decisions • Taking part in family activities • Taking part in community activities • Learning something new • Concentrating • Dealing with strangers • Attending mosque or religious gathering • Assisting others

24. Holding states accountable • Treat the state as a partner whose priorities must also be met. • Try to help the state meet them. • Typical state priorities: • Increase access to effective health services • Reduce mortality for the whole population • Reduce morbidity for the whole population

25. Meeting State and Survivor Needs • Address priority problems OF torture survivors, not just focus on problems DUE TO torture. • Most problems of torture survivors are shared by many others. • Where possible, support services that deal with these problems for everyone. Survivors should access these same services as others.

26. New Structure in Iraq • Most survivors receive mental health and counseling services integrated into physical health system which addresses these problems for everyone (ie, also non-torture survivors) • Providers are primary clinic staff with little mental health background who are trained to provide effective psychotherapy • Supervisors are mental health professionals based in psychiatric centers and in torture treatment centers. • Referrals (non-torture) to psychiatric centers • Referrals (torture) to TTC. • Advantages: • Government more supportive (supports wider need) • Clients like it better (more anonymity, less singling them out) • Reach and access enhanced +++++ through integration • Priorities better match client priorities and more accessible • Torture survivors who need it still get specialist care • TTC can focus on those who really need them.

27. Apprenticeship Model of Training and Supervision • Key to expanding quality treatment access in low resource countries • Based on research on training: • One-off trainings are ineffective for behavior change. • “Train and hope” approach to implementation does not work (e.g., Kelly et al., 2000) • Ongoing supervision with on-the-job training is critical

29. Purpose of apprenticeship training • Provides real skills development of provider and supervisor by learning while doing • Allows non professionals to really learn to provide treatment while assuring survivor gets quality treatment. • Iterative correction/improvement of survivor and provider and supervisor problems • Cares for providers – monitors and prevents/treats burnout.

33. Impact Assessment

34. Standard evaluation vsExperimental evaluation • Standard evaluation • Post intervention measure only, OR • Pre and post intervention measure • Assesses whether problem improved • Does not assess why. • Research • Pre and post evaluation(s) • Compare with controls • Assesses whether problem improved and whether this was because of the intervention.

35. Uganda • Intervention study: • Pre-intervention - 110/117 (94%) depressed • Post-intervention - 64/117 (54.7%) depressed (p<0.001)

36. Uganda • Control arm: • Pre-intervention - 110/117 (94%) depressed • Post-intervention - 64/117 (54.7%) depressed • Intervention arm: • Pre-intervention - 92/107 (86%) depressed • Post-intervention - 7/107 (6.5%) depressed (p<0.001)

41. Reduction of depression symptoms by group Depression Symptom Scores

42. Reduction of depression symptoms by group Depression Symptom Scores

43. Which one? • Use Standard Evaluation when other factors stable. • Use experimental evaluation when likely effect of intervention is not known and other factors not stable. • Control group is essential to answering the question: ‘was change due to the intervention?’

44. RCT as program evaluation RCTs perceived as: • Unnecessary • Unethical – making people wait. • Too complex/difficult • Too expensive - $,time,effort. • Waste/diversion of resources from ‘real’ aid • Not appropriate for low resource/difficult environments

45. Service capacity=500

46. Service capacity=500

47. What makes RCTs costly/complex? • Design/training/implementation of intervention • Monitoring quality of intervention • Supervision of workers • Monitoring acceptability/feasibility/compliance • Assessing who has priority for services • Reassessing persons after treatment.

48. True costs beyond normal M&E • Randomization (cheap) • Analysis (cheap) • Assessing additional cases (relatively cheap – economies of scale). • Small compared with savings from stopping ineffective programs • Saves money/time/effort of funder and population • More ethical

49. When is an RCT Appropriate? • Existing evidence base poor • Programs vulnerable to cross-cultural variation • Situation unstable • Therefore, uncertainty about local impact of interventions.

50. EFFECT SIZE is a measure of the difference between intervention and control in a way that is comparable across studies