Thoughts on the Use of Decision Analysis in the Review of New Drug Applications

1. Thoughts on the Use of Decision Analysis in the Review of New Drug Applications October 3, 2007 Todd Durham

2. Outline • NDAs and the nature of the decision • Potential benefits and challenges of decision analysis • An illustration • Learning and opportunities

3. Mental Exercise • Imagine that tomorrow you are diagnosed with a disease from which you will die in exactly 7 days. • If you could take a pill: • That would definitely cure you from this disease, how much would you pay for it? • That would give you a 25% chance of a cure, how much would you pay for it? • That would give you a 25% chance of a cure, how much risk (s%) of a debilitating stroke would you accept?

4. New Drug Applications • Marketing applications for new drugs • FDA reviewed between 20-30 NDAs (for NMEs) per year between 2001-2003 (FDA, Critical Path, 2004) • Data submitted with a NDA • Human evidence of benefit • Human evidence of risk • Manufacturing controls • Animal data on toxicology and carcinogenicity

5. Objective in Reviewing a NDA • Decide if a drug’s benefits outweigh its risks • Evolved historically with various changes in the law to: • Avoid misleading doctors or consumers • Keep dangerous drugs out of the system • What does the law really say?

6. “Substantial Evidence” from FDC Act of 1962 Substantial evidence was defined in section 505(d) of the Act as “evidence consisting of adequate and well-controlled investigations, including clinical investigations, by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved, on the basis of which it could fairly and responsibly be concluded by such experts that the drug will have the effect it purports or is represented to have under the conditions of use prescribed, recommended, or suggested in the labeling or proposed labeling thereof.” (FDA, Clinical Evidence of Effectiveness, 1998)

7. Sufficient Criteria for Demonstration of Efficacy • Choice of Primary Endpoint • Reliably measures a clinically relevant characteristic • Statistically sensitive to treatment • Identified a priori (with corresponding analysis methods) • Results for Primary Endpoint • Treatment effect is “statistically significant” in at least two studies • Magnitude of treatment effect (Δ) is clinically relevant • Results for Secondary Endpoints • Results from secondary endpoints further describe the relevance of Δ (primary endpoint) if results from primary endpoint in the same study are statistically significant

8. The Case of Carvedilol “… the usual two-study FDA paradigm does not make sense under all situations. This much is clear. But I would also suggest, as stated above, that experience has shown the paradigm to be a very useful guideline; exceptions should therefore be relatively unusual, and, when in doubt; one should err on the side of conservatism. Nevertheless, it strikes me as absurd in extreme cases to insist that if one does not meet the original primary end point in a study, that conclusions can never be definitive but only hypothesis generating.” (Fisher, 1999)

9. Criteria Used in Reviewing a NDA • Benefit • Quantity of evidence • Quality of evidence • Typically restricted to one or a few “endpoints” • Leads to a labeled claim consistent with results • Safety • From any number of reported adverse events • Cardiac safety studies (e.g., QTc) • Potentially animal studies (e.g., risk to fetus) • Manufacturing

10. Decision to be Made • Approve the new drug • Reject the new drug • Ask the sponsor for more information (“approvable”)

11. Influences on the Decision • Statistical robustness of the apparent benefit, with appropriate statistical control of the false positive rate • Clinical relevance of the benefit • Excess safety risks, with no control of the false positive rate • Severity of the disease • Availability of other treatments

12. When a Drug is Approved • Can be legally marketed in the U.S. • Doctors have a prescribing option • Patients have a treatment option • Pharmaceutical companies make revenue • Need for education all around • Safety will continue to be monitored • Surveillance has less rigor than RCTs • May be studied further • Expand the label • Clarify the role of the new drug or its effects

13. When a Drug is “Approvable” • Can not be legally marketed in the U.S. • Doctors can not prescribe it • Patients can not take it • May be studied further • Pharmaceutical companies spend more money on research • Time for further research and submission

14. When a Drug is Rejected • Sponsor may withdraw application • Can not be legally marketed in the U.S. • Doctors can not prescribe it • Patients can not take it

15. “Easy” Approval Decisions • A lot of evidence of clear benefit • Clinically relevant • Statistically robust (very unlikely due to chance) • At least moderately sized safety database reflects reasonable risks • No evidence of toxic or carcinogenic effects • No other available treatments or just a few treatments with some toxicities

16. “Easy” Rejection Decisions • Obvious hazards with little benefit • Poor manufacturing controls

17. Decisions are Much Harder When • Mixed results for benefit • Drug which has been shown to have a benefit in some populations but not others. • A lot of studies, only a few of which were successful. • Statistical criteria for “success” are not met. • Some significant trade-offs must be reckoned with.

18. Made Even More Difficult • Changing landscape • Regulatory standards (e.g., emerging concerns) • Medical advances • Changing standard of care • Ex-US medical care • External pressures • Congress • Patient advocates • Pharmaceutical industry

19. Benefits of a Decision Analysis • Transparency of the decision • Many objectives possible (identified, weighting) • Influences for all stakeholders • Role of uncertainties • Which ones make the most difference? • Model that can be applied to many products in the same therapeutic area, but evolve over time. • Dissection of the problem  greater understanding

20. Transparency • Patients • To pharmaceutical company • Within the FDA • Congress

21. Role of Uncertainties • How much do the following uncertainties bear on the consequences? • Quality or quantity of evidence of benefit • Medical need, population affected • Available therapies • How many patients will actually use the treatment? • Don’t need to be accurate but having a grasp on the range of uncertainties can still be instructive (through tornado diagrams)

22. An Evolving Model • Changes in medicine • Changes in how medical expenses are reimbursed • Changes in societal priorities or norms

23. Dissection of the Problem • Factors which most influence the best decision can lead to new priorities • Role of available therapy  compare the new treatment to available therapy • Quantity of evidence  additional information • The safety/benefit tradeoff  patient involvement • Insensitivity of the model to various uncertainties can make decisions easier

24. Challenges of DA for this Application • How to define the safety risks? • All of them? • Control of false positive rate? • How to assess the consequences • By whom? • Using what measure?

25. Consequences • Time • Money • Human lives • Unwanted events • Quality-adjusted life years • Credibility / trust (how to value?) • Quality of information (what is its value?)

26. Basic Decision Tree Approved Approvable NDA Decision Rejected

27. Waiting for More Information Success Approved Outcome Yes Failure Approvable NDA Decision New Study? No Rejected Presumably, success would lead to a greater chance of regulatory approval, but what are the consequences of having made this decision to wait for more information?

28. Illustration: Serious Diagnosis • Advanced cancer that affects 50,000 individuals per year • Current expected life-span (median) is 20 months from diagnosis. • The one available treatment is not tolerated well such that most patients choose not to take it. Loosely adapted from story in New York Times, 2007.

29. Results from Clinical Trials • New treatment compared to placebo • Efficacy: • Treatment effect is ~4 months of survival (benefit) in two studies. • In one study survival had a nominal p-value <=0.050, but it was a secondary endpoint. • Primary endpoint was stopping progression of cancer (failed in both studies). • Safety: • Most common side effect is flu-like symptoms • 1-2% chance of a stroke from new treatment

30. Considerations • DA could address the consequences of a world with (now or later) and without the new treatment • Lives lost in a period of time • New strokes in a period of time • Bouts of flu-like symptoms • Was survival a false positive finding? • Zero survival benefit • What to do with the conventional hypothesis testing interpretation? • Won’t the benefit depend on how many patients might use the new treatment?

31. Could this Ever Be Applied? • Modest proposals: • FDA could conduct an exercise by writing out an influence diagram for approval decisions in one therapeutic area. • Carry out research on how to best communicate risk to patients (both benefit and safety). • Increased emphasis on risk communication to patients. Steiner, 1999 has tremendous insight on the topic. • More difficult proposal: • Conduct focus groups with patients to examine ability to elicit their trade-offs. Howard has written on ways to value life and other outcomes. • Fantasy-land proposal: • Make all drugs available for marketing and change the regulatory paradigm such that regulators verify accuracy of labeling and educate doctors and the public.

32. Learning from Experience • Unexpected clarity, almost profound new understanding of the decision to be made. • Ability to proceed without regret knowing the problem had been understood as best as humanly possible. • Training is important. Even highly intelligent people do a poor job of estimating uncertain quantities.

33. Illustration: What if… • The benefit was only 0-4 months of survival, with a great deal of skepticism that 4 months from the trials was “real”? • Some patients might trade the chance of a stroke for a chance at an extra month or two of life. • But they can’t make this choice unless the drug is made available to them. • We won’t know unless we ask.

34. References US Department of Health and Human Services, Food and Drug Administration, 2004. Challenge and opportunity on the critical path to new medical products. Available from www.fda.gov. US Department of Health and Human Services, Food and Drug Administration, 1998, Providing clinical evidence of effectiveness for human drug and biological products. Available from www.fda.gov. Fisher L. Carvedilol and the Food and Drug Administration (FDA) Approval Process: The FDA Paradigm and Reflections on Hypothesis Testing. Controlled Clinical Trials 1999;20:16–39. Steiner J. Talking About Treatment: The Language of Populations and the Language of Individuals. Annals of Internal Medicine 1999; 130,7: 618-622. Howard RA. On Making Life and Death Decisions. Readings on the Principles and Applications of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic Decisions Group. Howard RA. On Fates Comparable to Death. Readings on the Principles and Applications of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic Decisions Group. Andrew Pollack, “Panel Endorses New Anti-Tumor Treatment,” The New York Times (March 30, 2007).