Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis

1. Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis Training in Clinical Research DCEA Lecture 5 UCSF Dept. of Epidemiology & Biostatistics Jose Luis Burgos, MD, MPH, AAHIVS March 4, 2009

2. Project Map • Think through your research question • Sketch your analysis • Collect data for your model • Adjust model • Run-test your model • Conduct Sensitivity Analysis • Write it up.

3. Objectives • To understand the purposes of sensitivity analysis. • To understand techniques used for sensitivity analysis.

4. Why do sensitivity analyses? • All CEAs have substantial uncertainty • Sensitivity analyses deal with that uncertainty systematically • Convince audience results are ‘robust’ – qualitative findings don’t change with small changes in inputs

5. Sensitivity analysis Prior lectures reviewed how inputs are determined, plus a few simple sensitivity/threshold analyses. This lecture will cover four topics: • 1. Types of uncertainty • 2. Deterministic sensitivity analyses (one-way, multi-way, scenario) • 3. Probabilistic sensitivity analysis (Monte Carlo) • 4. Uses of sensitivity analysis

6. Types of uncertainty • Parameter Uncertainty • What’s are the correct input values? • Methodological uncertainty • Model Structure • How values are combined • or modeled • Model Process • Implicit decisions made by the analyst such as • viewpoints or effects considered

7. Addressing Methodological Uncertainty • Sensitivity Analysis • Scenario analysis with different models to combine costs and estimate effects • Statistical Analysis • Where multiple parameter sets available, can test the fit of different models

8. Addressing Model Process Uncertainty • Standardized CE analysis – difficult and different panels give different recommendations, but key common components are (from Drummond): • The background of the question • The viewpoint for the analysis • The reason for selecting a particular form of analysis • The population to which the analysis applies • The comparators being assessed • The source of the medical evidence and its quality • The range of costs considered and their measurement • The measure of benefit in the economic study (e.g. LY gained, QALYs gained) • The methods for adjusting costs and benefits according to their timing • The methods dealing with uncertainty • The incremental analysis of costs and benefits • The overall results of the study and its limitations

9. Deterministic Sensitivity Analyses How does assigning specific different values to inputs change output? • One-way (‘univariate’): Vary 1 input at a time • Multi-way (‘multivariate’): Vary 2+ inputs at a time • Scenario (variant of multi-way): Tests set of relevant conditions. • Threshold analysis (one-way or multi-way): Input values beyond which cost-effectiveness achieved (or lost).

10. Univariate Sensitivity Analysis • Examine robustness of ICERs to changes in a single parameter: • ‘best’, ‘high’ and ‘low’ estimates (but experts consistently underestimate true variability) • Value +/- 1 SD • 95% CI (based on observed or assumed distribution) • ‘clinical meaningful range • Extremes • Threshold analysis

11. One-way SA – Aneurysm management

12. One-way SA Muennig, 2008

13. One-way SA – LTBI program

14. Peginterferon Model Inputs: Estimated “base case” and range European Journal of Gastroenterology & Hepatology. 2007;19:631-638.

15. LTBI Model Inputs: Estimated “base case” and range Int Jour Tuberc Lung Dis 2008

16. Univariate Sensitivity Analyses: Base case and range of outcomes for 1,000 IDUs Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

17. Automating one-way SAs: • Male circumcision for HIV prevention in South Africa Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

18. Male circumcision for HIV prevention in South Africa • Considering HAART cost averted Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

19. Tornado Diagram

20. Two-way SA:CE of Empowerment program

21. Two-way sensitivity analysis for changes in HIV risk and average cost for managing active TB cases Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

22. Three-Way SA Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

23. Scenario SA Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

24. Low HIV prevalence setting: 25% in CSWs; 16.4% in clients Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($3,864) ($1,824) $2,076 [$509] ($7,426) ($5,386) ($1,486) ($10,989) ($8,949) ($5,059) Medium HIV prevalence setting: 50.3% in CSWs; 33.0% in clients Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 Low FC cost: $0.33 Medium FC cost: $0.66 High FC cost: $1.32 ($6,023) ($3,983) ($83) ($11,203 ($9,163) ($5,263) ($16,386) ($14,346) ($10,446) Multivariate SA on female condom promotion: Net costs by HIV prevalence and key cost inputs for 1,000 CSWs

25. Threshold Analysis: NVP for prevention of vertical transmission of HIV in sub-Saharan Africa Input values needed for $50/DALY

26. Threshold Analysis: NVP $ for prevention of vertical transmission of HIV in sub-Saharan Africa Marseille E, Kahn JG, Saba J. AIDS 1998; 12:939-948

28. Addressing Parameter Uncertainty:Multivariate Sensitivity Analysis • Types of Multivariate Sensitivity Analysis: • Repeat bivariate • Maximize / minimize CE ratio for different parameter combinations • Scenario analysis • Monte Carlo simulations under different assumed distributions for parameters (probabilistic sensitivity analysis’ Gold)

29. Probabilistic sensitivity analysis What is it? What is it good for?

30. The problem with deterministic SAs No estimate of the probability of achieving a particular outcome (Probabilistic SAs are the remedy)

31. Probabilistic sensitivity analyses • Value • Returns the likelihood of attaining particular • outcome or outcome range. • Everything known about each input expressed all at • once. • Particularly valuable when many inputs important. • Drawback • Need to know, or be able to make decent estimates of, the underlying probability distribution.

32. From empirical data to PD (1) Frequency distribution of # of clients reported by 101 FSWs Variable | Obs Mean Std. Err. SD [95% Conf. Interval] -------------+--------------------------------------------------------------- # of Clients | 100 67 3.22 32 60.6 - 73.4

33. From empirical data to PD (2)Graphical Representation of the # of clients reported by 101 FSWs

34. From empirical data to PD (3) Probability distribution of # of clients reported by 100 FSWs

35. From empirical data to PD (4) Graphical Representation of the Prob dist. of clients reported by 101 FSWs

36. From empirical data to PD (5) Probability Distribution of # of clients among 10,000 FSWs Variable | Obs Mean Std. Err. SD [95% Conf. Interval] -------------+----------------------------------------------------------------------------------- # clients | 10,000 68 0.36 33 66.7 - 67.7

37. Triangular Distribution Muennig, 2008

38. Published distributions Henson SJ. Estimating costs of acute gastrointestinal Illness in BC. Int Jou Food Microb. 2008; 127: 43-45

39. Monte Carlo simulation output Crystal Ball output

40. SA For QALYs Gained $226-$504 Treeage output

41. Other uses of sensitivity analysis(the inner teachings) • Planning the analysis • Debugging the model • Documenting relationships between inputs and • outputs • Identifying thresholds • Influencing policy

42. Other uses: Planning the analysis • Program software to permit SAs on likely SA • variables. • SA curves provide a check on integrity of • model. • Identify candidates for more data collection • early.

43. Other uses: Debugging the modelTricks of the trade • One-ways best because simple and intuitive. • Plug in extreme values. • Separate diagnosis of numerator from denominator. • Break outputs down further if necessary • (intervention versus control arms).

44. Other uses: Documenting relationships between inputs and outputs • Distinguish between ‘bugs’ and insights. • Examples of insights: • Slowing disease progression can increase costs. • Higher disease prevalence can mean lower benefits. • Benefits decrease with age - competing mortality risks.

45. Unexpected dynamic uncovered by SA: Female condoms study Marseille et. al. Soc Sci Med. 2001 Jan;52(1):135-48.

46. Other uses: Identify thresholds – Influence Policy Preventing HIV vertical transmission in sub-Saharan Africa • Cost of ARVs to prevent vertical transmission. • Universal versus targeted provision of NVP.

47. Cost per DALY of HIVNET 012 NVP regimen as function of HIV seroprevalence and type of counseling/testing regimen

48. Pegynterferon CE acceptability curve European Journal of Gastroenterology & Hepatology. 2007;19:631-638.

49. C-E Acceptability Curve (QALYs) WTP $0 0.4 $50 0.4 $100 0.5 $150 0.5 $200 0.6 $250 0.6 $300 0.7 $350 0.7 $400 0.8 $450 0.8 $500 0.8 WTP $550 0.8 $600 0.9 $650 0.9 $700 0.9 $750 0.9 $800 0.9 $850 0.9 $900 0.9 $950 0.9 $1,000 0.9

50. Summary • SA is a set of techniques for the explicit management of uncertainty. • Essential part of establishing key findings. • Indispensable for convincing your audience that your results are technically sound and policy-relevant.