Using a Modified Delphi Technique for Determining Indicators/Services and their Relative Weights

1. Using a Modified Delphi Technique for Determining Indicators/Services and their Relative Weights G FritscheSr Health Specialist/HDNHE International PBF Course, 9 to 21 May, 2011, Mombasa, Kenya

2. What does this picture show?

3. SMART • Specific • Measurable • Attainable • Realistic • Time-bound

4. If you purchase these services, what could occur? • Inpatient care and treatment of complicated bacterial or viral infections other than complicated respiratory infections of children less than five years old • Diagnosis and treatment of uncomplicated diarrhea in children • Treatment of severe malaria

5. If you purchase this service, what could occur? • New curative consultation

6. The cost of services

7. Why Delphi? “Delphi is particularly appropriate when decision-making is required in a political or emotional environment, or when the decisions affect strong factions with opposing preferences. The tool works formally or informally, in large or small contexts, and reaps the benefits of group decision making while insulating the process from the limitations of group decision-making; e.g., over-dominant group members, political lobbying, or "bandwagonism". “ Quote from Alan Cline, Carolla Development, http://www.carolla.com/founder.htm

8. Why Delphi for PBF? • Delphi is appropriate for designing PBF systems: • On the Objectives and Attributes • On the Indicators • On the Weighting, because: • Resource allocation decisions have to be made • Selected indicators have to be chosen from long lists of indicators • The selected indicators will need to be allocated a value (Rwandan HC PBF model) • Time is limited • Many actors (technicians and politicians) with diverging opinions and preferences

9. Delphi Prioritization Procedure* • Pick a Facilitation Leader • Select a panel of experts • Create a straw man criteria list • Rank the criteria (1, 2 and 3; one most important) • Calculate mean and deviation (remove criteria with mean greater or equal to 2) • Re-rank criteria and show to the panel • Repeat the procedure until there is consensus *Drawing from Alan Cline, Carolla Development, http://www.carolla.com/founder.htm

10. Example of a modified Delphi Approach: the Feb 2006 Rwanda National PBF Workshop • Applied a modified Delphi to reach consensus on desired Objectives and Attributes for a national PBF system, reasons: • Three PBF pilots with different approaches • Disagreements as to what PBF meant or ought to do • Confusion on semantics • Strong preferences for own project • Time pressure

11. Example of a modified Delphi Approach: the Feb 2006 Rwanda National PBF Workshop • Process for Attributes: • Assumptions: to create a list of desirable attributes of a Rwandan Performance Based Financing Model. This performance based financing model should have (i) an agency that pays, (ii) agencies that provide health services, (iii) a Monitoring and Evaluation system. • Identify a straw man list of criteria (all participants 10 criteria maximum) • Expert panel ranks the criteria individually and anonymously (1, 2 and 3: one most important) • Ranking compiled in plenary, remove those that are 2 or higher • Re-ranking until consensus reached

12. More Delphi for PBF • Various modified Delphi Techniques have been used also for choosing and prioritizing PBF indicators for HIV, including their relative values (‘indices’) • For instance in Burundi (Sept 2009) • Benin (June 2010)

13. Delphi Group Work Assignment You will be applying a Modified Delphi Technique to (1) reach consensus on a straw man criteria for:list of potential PBF indicators for BPHS (2) create a relative weight for the PBF indicators retained from the first Delphi exercise

14. Delphi Group Work Assignment Round 1 Your group assignment is to derive a list of 12 indicators from the straw man list of indicators for the BPHS for health center level services: • Choose a facilitator • Individually look at the straw man criteria list of indicators and rank these (1, 2 and 3; ‘1’ the most important and ‘3’ the least important) • The facilitator compiles the results in the excel sheet and shows the mean and standard deviation. All those indicators marked 2 or higher are removed from the list • Results are shown to the group, and the group re-classifies the indicators, • Repeat process until complete consensus is obtained • Choose the first 12 indicators and present in plenary

15. Delphi Group Work Assignment Round 2 • Your group work assignment is to derive relative weights (‘indices’) for the 12 chosen indicators: • Choose a facilitator • Each individual group member copies the 12 indicators according to a set order 1-12 • The ‘Curative Care’ indicator is put as number 1 and given a weight of ‘1000’ • Each individual scores the other indicators according to the reference index of ‘1000’, if another indicator is e.g. weighted ‘5000’ then that means that the indicator is weighted five times more than a curative care visit • The facilitator compiles and shows the results • Repeat process till there is consensus

16. Next steps • Use the forecasting tool for inputting the indicators and using their weights, come up with unit values (example Benin) • Show results and discuss unit values, adapt unit values if there is disagreement (plenary) • Frequently, showing the real value, derived from the indices, leads to adapting the relative weights (as the real values depend on the population based volumes and assumptions related to the growth forecasts/targets. • The above method will have determined the MPA indicators and their weights, plus eventually, their real values.

17. Fix the system or I will come and fix youwhen I grow up Malaria Consortium - Sunil Mehra