Analysis Alongside A Randomized Trial

1. Analysis Alongside A Randomized Trial Todd Wagner, PhD May 2009

2. Objectives • At the end of the class, you should • Understand how to set up your datasets • Be familiar with analytical methods • Want to hear the class on decision modeling

3. Dominance • No need to calculate an incremental cost effectiveness ratio if: • Intervention is more effective and less expensive than control • Intervention is more expensive and less effective than control • But: • Check whether dominance exists within subgroups • Does dominance persists after including uncertainty

4. Incremental Cost-Effectiveness Ratio (ICER) Calculate in the absence of dominance CostEXP - CostCONTROL _____________________ QALYEXP -QALYCONTROL

5. Cost Data • Have costs consistent with the stated perspective • Societal • Health care utilization • Patient costs • Caregiver costs • Intervention costs (direct plus indirect)

6. Common Hurdle • Many of the parameters in the analysis are based on assumptions (e.g., wage rate, mileage costs) • Consider whether these assumptions are biased toward/against the intervention • Ideally want to include uncertainty • One way sensitivity analysis • Statistical methods (bootstrapping, Cooks D)

7. Another hurdle • Include disease-related utilization or all health care utilization? • How do you define disease-related? • Recommend: look at all utilization for the CEA

8. Labor Outcomes • Productivity employment is not in the cost estimate • Anyone remember why? • If labor outcomes are important, still collect them but report them separately.

9. Dataset • You need to create a “long” dataset • You need to have an group indicator for experiment (1) and control (0) • You need to have cost and outcome information • You need subgroup identifiers

10. Data

11. Analysis . tabstat totcost followup if exp==1, by(papres) stat(mean) format(%3.2f) Summary statistics: mean by categories of: papres (initial pap results) papres | totcost followup -----------+-------------------- ASCUS/AGUS | 347.31 0.57 LGSIL | 373.97 0.64 HGSIL | 404.72 0.87 -----------+-------------------- Total | 355.45 0.61 -------------------------------- . tabstat totcost followup if exp==0, by(papres) stat(mean) format(%3.2f) Summary statistics: mean by categories of: papres (initial pap results) papres | totcost followup -----------+-------------------- ASCUS/AGUS | 74.92 0.32 LGSIL | 73.98 0.30 HGSIL | 104.94 0.43 -----------+-------------------- Total | 77.28 0.32 --------------------------------

12. Hypothesis Testing . reg totcost exp Source | SS df MS Number of obs = 348 -------------+------------------------------ F( 1, 346) = 292.64 Model | 6728200.2 1 6728200.2 Prob > F = 0.0000 Residual | 7954890.66 346 22991.0135 R-squared = 0.4582 -------------+------------------------------ Adj R-squared = 0.4567 Total | 14683090.9 347 42314.3829 Root MSE = 151.63 ------------------------------------------------------------------------------ totcost | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- exp | 278.1664 16.26051 17.11 0.000 246.1845 310.1483 _cons | 77.27982 11.62933 6.65 0.000 54.40675 100.1529 ------------------------------------------------------------------------------ Logistic regression Number of obs = 348 LR chi2(1) = 28.42 Prob > chi2 = 0.0000 Log likelihood = -226.30785 Pseudo R2 = 0.0591 ------------------------------------------------------------------------------ followup | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- exp | 3.225974 .7243906 5.22 0.000 2.077419 5.009538 ------------------------------------------------------------------------------

13. Conclusion • Intervention is more costly and more effective • Next Steps: • Include uncertainty • Check for subgroups and • calculate the ICER

14. Intervention Group papres | totcost followup -----------+-------------------- ASCUS/AGUS | 347.31 0.57 LGSIL | 373.97 0.64 HGSIL | 404.72 0.87 -----------+-------------------- Total | 355.45 0.61 -------------------------------- Control Group papres | totcost followup -----------+-------------------- ASCUS/AGUS | 74.92 0.32 LGSIL | 73.98 0.30 HGSIL | 104.94 0.43 -----------+-------------------- Total | 77.28 0.32 -------------------------------- ICER=(355.45-77.28) (0.61-0.32) ICER=982.1837 Is that good or bad? ICER Calculation

15. Uncertainty • You need to calculate the confidence regions around this parameter estimate Variable | Reps Observed Bias Std. Err. [95% Conf. Interval] ---------+------------------------------------------------------------------- cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) | 787.438 1369.206 (P) | 787.3935 1366.588 (BC) ----------------------------------------------------------------------------- N = normal, P = percentile, BC = bias-corrected

16. Confidence Regions • Ratios are complex to interpret

17. More effective and more expensive Less effective and more expensive Less effective and less expensive More effective and less expensive

18. Acceptability curves show the information based on willingness to pay for the outcome. Shape of the curve is dependent on the bootsrapped estimates Allows decision makers with different thresholds to interpret the data Acceptability Curves Source: Henry Glick

19. Threshold Value forCost-Effectiveness Ratio • Rule of thumb: U.S. health care system adopts interventions with ratio of less than $50,000 ($100,000) per Quality Adjusted Life Year. Variable | Reps Observed Bias Std. Err. [95% Conf. Interval] ---------+------------------------------------------------------------------- cerp2 | 1000 982.1837 24.04897 152.2913 683.336 1281.031 (N) | 787.438 1369.206 (P) | 787.3935 1366.588 (BC) ----------------------------------------------------------------------------- N = normal, P = percentile, BC = bias-corrected

20. Adopt this Pap Smear Intervention? • Poll • Yes • No • Don’t know • Don’t care

21. Limitations with RCTs • Proxy outcomes • Length of follow-up • Generalizability

22. Ideal Study • RCT comparing behavioral intervention and usual care control • Follow participants for life • Advantages: know all costs and all benefits • Disadvantages: expensive and possibly useless results at end of study

23. Usual Study • Outcomes measured at end of study • Use models to estimate lifetime costs and benefits in the CEA

24. Effectiveness • Preferred metric for CEA is quality adjusted life years (QALYs) • Most behavioral interventions use an “intermediate outcome” • e.g., receipt of mammography • Few behavioral interventions use QALYs because the study would have to be huge and/or very long in duration

25. CEA with Intermediate Outcomes • Easy and sufficient for publication • Hard to interpret ICER • Can’t compare two CEAs with different intermediate outcomes • Can’t compare CEA to other CEA from another clinical area • Sometimes only feasible approach

26. Intermediate to QALYs • “Translate” intermediate outcome to QALYs • Either build a model de novo or use an existing model • Requires a lot of resources • Most useful, but most challenging

27. Lost in Translation • Gap between ideal and usual study • Models fill the gap • Behavioral models have unique challenges • Partial behavior change is missing from current models • Definition: people who progressed in their “stage of change” but did not successfully change their behavior at the end of the study Wagner TH, Goldstein MK. Behavioral interventions and cost-effectiveness analysis. Prev Med. Dec 2004;39(6):1208-1214.

28. Outcome Responsiveness • Many trials use indirect QALY assessments (HUI, QWB, EQ-5D) • These measures are often not responsive to the intervention effect at the end of the trial. • How do you interpret the results when the main outcome shows an effect and the QALYs do not?

29. QALY Responsiveness • Try to understand why QALYs were not sensitive • Analyze data for subgroup effects • Analyze other outcomes that may be sensitive to change. • Disease specific quality of life • Willingness to pay • If you think there is an effect, ignore the p-value on the QALYs and model the data using the mean and variance estimates

30. Another Approach • Our CEA did not leverage the trial data • We extracted parameters, created confidence regions and then (hopefully) put these parameters into a decision model

31. Net Monetary Benefits • What we did • Collect data to estimate ΔC and ΔE • Calculate an estimate: • ΔC / ΔE (ICER = incremental cost-effectiveness ratio) • Another option: • λ ΔE – ΔC (INB = incremental net-benefit) Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Econ. Jul 2002;11(5):415-430.

32. Data

33. NMB set up • You have cost data and effectiveness data for each person • Use this information in a regular regression framework • You need to estimate λ • λ is the WTP for the outcome • Rerun the analysis for different λ values

34. NMB Limitations • Based on a linear model • Does not necessarily translate into non-linear or latent models • I suspect this is possible if you estimate predicted probabilities for groups and then carry out the analysis for the groups.

35. Generalizability • Recall that RCTs may not enroll a generalizable population • For many RCTs in VA, you can compare participants to non-participants • Generate propensity scores • Use the propensity score as a weight in the RCT analysis– place more weight on people who are a lot like the non-participants

36. Next talk • May 27, 2009How Can Cost Effectiveness Analysis Be Made More Relevant to US Health Care?Paul Barnett, Ph.D.