The Quality of Medical Advice: Vignettes (and more)

1. The Quality of Medical Advice: Vignettes (and more) Jishnu Das World Bank

2. Saving Meena: Story of a death foretold

3. From the local to the global

4. Poor Health Life Expectancies (Ethiopia 53; Kenya 54; Zambia 42 in 2007) U5 Mortality (per 1000 born): Ethiopia 127 ; Kenya 114; Zambia 174)

5. Many countries

6. Why? • Infectious diseases: HIV/AIDS, Malaria, Diarrhea, TB • Kenya: TB incidence rate is 62/10,000 (one of the highest in the world) • Partly to do with low incomes • But see Riley on countries that improved health outcomes at low-income levels • Partly to do with access to care • Particularly a problem in rural areas

7. Some quotes There is an obvious difference between rural and urban postings. Working in rural areas involves helping the poor… in urban areas one can learn, have more income, have a good school for one’s children. Doctor in Ethiopia It is Siberia!.Doctor in Addis Ababa There is no plan of development for a doctor in the rural areas; it is as if you are lost. The lack of career development means that it is as if you are punished. Doctor in Rwanda Promotion is as important as remuneration because you cannot stay in the same place forever. Doctor in Ghana Sourced from Serneels and others

8. And yet… In many countries around the world, people use facilities a lot more than they do in the US

9. And yet… In deep rural Madhya Pradesh, one of the states in India with the worst HD outcomes, there are 11.3 medical care providers accessible for representative rural households who do not live close to national highways or close to urban centers

10. And yet… This is not because these villages are huge: for these villages, there are around 20 households per provider

11. New Studies on Doctor Visits • In 5 countries studied, the poor visit doctors almost as much as the rich • Kenya: 60% of poorest quintile seek care when sick relative to 78% of richest quintile • In urban India, new survey methods show that the poor visit doctors more than the rich • The difference arises because of shorter recall periods • In rural India, households visit doctors twice as much as in the United States

12. Do the poor really go to doctors less than the rich? Experimental data from Delhi show that when households are asked on the basis of monthly recall, self-reported doctor visits fall by 65 percent for the poor compared to weekly recall. In fact, the poor go to doctors more than the rich. We don’t know what happens in other countries, because weekly recall questionnaires are very rare!

13. One alternative Health outcomes may be related to the quality of health care that people receive

14. Structural Quality • Quality traditionally defined as structural quality—state of infrastructure, availability of medicines. This is clearly informative BUT • Its not correct when demand is a factor (medicines) • The quality of medical advice may be equally (or even more) important • The relationship between quality of advice and structural quality is weak (India, Indonesia, Tanzania)

15. The Quality of Medical Advice: New evidence • Since 2002, team working on the quality of medical advice at The World Bank and the University of Maryland • Basic Idea • What can be measured, and how? • How do these measurements help us understand the quality of medical advice? • How can the quality of medical advice be improved?

16. Where we are • Quality of medical advice can be decomposed into • Competence: What does a doctor or medical care provider know about how to treat an illness • Practice: What does a doctor or medical care provider do when faced with an illness • We systematically find that the two are very different. This has important policy implications • Improving competence is about training • Improving practice (given a competence) is about getting doctors to exert greater effort • Countries: Tanzania, Urban India, Indonesia, Paraguay • Ongoing: Rwanda, Rural India, Argentina

17. Remainder of Presentation Basic facts about competence Basic facts about practice More interesting facts Lessons learnt (thus far!)

18. What are Vignettes Standardized mix of (in our case) 5 cases One interviewer is `patient’; other is `recorder and observer’ Child with Diarrhea“My child has been suffering from diarrhea for the last two days, and I do not know what to do” History Doc gets 1 point if he asks about last urination (for instance) Examination Doc gets 1 point if he asks about temperature (IF asked, recorded responds `98.8 F’) Treatment Doc prescribes treatment: treatment is graded by independent raters in South Asia and the US

19. Scoring Vignettes • The questions that doctors ask are compared to a checklist of essential procedures • An aggregate “index” is compiled that accounts for the different “difficulties” of different checklist items • We call this index “competence” • We normalize mean = 0 and standard-deviation=1 • Moving from 0 to 2 moves you from 50th percentile to 95th percentile

20. Vignettes: Advantages Advantages Standardized case-load (same 5 cases to all docs) Standardized patient (specify that patient will `comply fully with all medications and tests’

21. What they know

22. Public or Private? An MBBS is the formal Indian medical degree—roughly the equivalent of an MD in the US

23. Distribution of Competence

24. Regression Models • Results very similar in other countries • More competent doctors in urban areas • More competent doctors in richer areas (within urban and within rural areas) • More training increases competence

25. Competence across countries

26. Measuring practice • Sit in the doctors office • Record details about every interaction between the doctor and patient • Time • History taking • Physical exams • Drugs prescribed Photo Credit: Ken Leonard

27. Practice: Some numbers

28. Another look at practice

29. Public-Private Again

30. Public-Private Again

31. What they know, what they do Percentage of Essential Tasks Completed 40% of essential questions asked Private MBBS Public MBBS Private, No MBBS

32. Lost Training: Tanzania Sourced from Ken Leonard

33. Lost Training: India

34. Training or Effort? • Because of lost training • We simulate that the impact of training is very small relative to improvements in effort

35. The big caveat? • Is it that public doctors put in a lot less effort because they see many more patients? • We find more effort in India in hospitals, which typically see many more patients • Two new and incredibly sad pieces of research

36. Case loads and doctor shortage Maestad and others (2009) went and sat in many doctors clinics in rural Tanzania “The average doctor sees 18.5 patients per day and total time use is 5.7 mins per patient. The doctor completes 22% of essential tasks” That’s less than 2 hours a day in an 8 hour day There is no relationship between caseload and doctor effort!

37. The Hawthorne Effect Effort jumps when doctors are observed Doctors significantly increase their effort in Tanzania when they know they are being observed with no detrimental effect on patient (Leonard, 2007) Based on Ken Leonard (2007) in Journal of Health Economics

38. Improving health outcomes • New research suggests that improving effort could also improve outcomes • Bjorkman and Svensson, 2008: Uganda • Rwanda Pay-for-Performance experiment: Ongoing, Rwanda

39. Summary • The quality of medical advice is key and understanding the levels and correlates of quality is an urgent priority • Measuring either competence or practice is a good start • But measuring both vastly increases our understanding of what is going on and where the policy levers may be • Resources exist to help do this • A new website ready soon with all studies and resources in one site • Support from the Chief Economists office and DEC

40. Lessons Learnt: What worked (1) • The overall methodology is sound and important • Sound: Correlates with various characteristics as predicted by common sense • Important: Highlights potential and limitations of different policy measures • The distribution of quality across public/private or rich/poor • The distribution of effort • The know-do gap

41. Lessons Learnt: What worked (2) Regardless of the style of the provider’s training, the type of medications dispensed were very similar • Initial worry that the variation in doctors is too large in India (Allopathic, Ayurvedas, Unani, Homeopaths) for a single instrument to capture quality • Turned out to not be a bit worry because doctors were all treating patients using the same (allopathic) medicines • Therefore, they could be graded on the same scale

42. Lessons Learnt: What did not workVignettes • We chose the simplest cases possible with no complications • Even then, our vignettes are not good at distinguishing bad from very bad doctors (high standard-errors for less than average providers) • Perhaps adding in a simple set of written questions would help • For instance: “If a child is suffering from diarrhea, what should you give the child?”

43. Lessons Learnt: What did not workVignettes • All the 5 cases we chose did not require any treatment at the primary level. • Therefore, the “mistakes” we usually pick up are errors of commission—doctors doing things that they should not have • Definitely consider including cases that require treatment at the primary level (pneumonia?) • This allows us to pick up errors of omission

44. Lessons Learnt: What did not workPractice • The vignettes standardize the case-mix and the patient-mix • Observing real patients does not. This leads to problems • Unobserved patient characteristics could affect inference (for instance sorting) • Use an exit-survey to pick some of this up, if possible • The cases that overlap with the vignettes are limited (rare cases almost never seen) • And may not be perfect overlaps • Possibility of using “simulated standardized patients” • Pilot currently underway: if this works out, it vastly improves our diagnostic abilities

45. Lessons Learnt: What did not workOverall The studies thus far are mostly “boutique” studies We are working on how to mainstream them We are not sure what the cost of deviation from the “boutique” approach would be

46. What I would do (again) Make sure that you keep lots of time for case development Pilot the vignettes until all (>95%) questions that providers ask have a predefined answer Train enumerators until they have memorized the entire vignettes module NEW: use video-recordings of doctor-patient interactions as training material for direct observation (these are being developed)

47. Things to do differently • Post-code treatments (the ultimate nightmare) • Make some changes to the direct observation form • Patient order • Interviewer assessments of patients (to be piloted) • Number of questions that patients ask • Have a clear idea of the timeline and the work program (but that’s for any of this work!)

48. Papers on which this presentation is based