Probability Elicitation and Calibration in a Research & Development Portfolio A 13-Year Case Study

1. Probability Elicitation and Calibration in a Research & Development Portfolio A 13-Year Case Study • Dimensions of value for R&D projects • Probability of technical success as a metric • Assessing probabilities • A review of thirteen years of data Jay Andersen Eli Lilly and Company June, 2012

2. Project Value R&D Portfolio Model Structure Phase 1 Tech. Feas. Phase 1 Tech. Feas. Preclinical Tech. Feas. Patients Preclinical Tech. Feas. Competition Phase 2 Tech. Feas. Phase 2 Tech. Feas. Phase 3+ Tech. Feas. Technical Feasibility Market Opportunity All Costs Phase 3+ Tech. Feas. Technical Feasibility Capital Costs Cost Timing Preclinical Studies Clinical Studies Development Costs Clinical Costs Regulatory Review The focus of today’s discussion

3. Measuring Technical Feasibility • Qualitative descriptions of uncertainty suffer from vagueness and lack of collective agreement on useful definitions. • A subjective probability represents the degree of belief in an event by an individual. • Quantification of this uncertainty allows other business metrics to be specified (e.g, probabilized NPV, cash flows, expenses).

4. A probability of technical success can be elicited by type of uncertainty Toxicology Results Clinical Efficacy Results Technical Success Clinical Safety Results Regulatory Results

5. A probability of technical success can be elicited by stage of development Preclinical Success Phase 1 Success Technical Success Phase 2 Success Phase 3 & Registration Success

6. Each stage of development can be broken down by type of uncertainty Toxicology Results Tumor Response Rate Phase 2 Success Time to Progression Adverse Events

7. Probability ElicitationProcess Alternatives • Self-assessed by project team • Using a trained facilitator • Independent review board • Independent review board with a trained facilitator

8. Facilitators are trained to deal with bias • Anchoring and Adjustment • Availability • Conditioning • Motivational • Representativeness

9. A historical assessment ofprobability projections • At Lilly, an independent review board (PAG) has been charged with the responsibility of objectively assessing the P(TS) of R&D portfolio projects since early 1997. • These assessments have been partitioned by stage of development: • P(preclinical success) • P(phase 1 success given preclinical success) • P(phase 2 success given phase 1 success) • P(phase 3 & registration success given phase 2 success) • Our database has over 730 probability estimates over the past 13 years. We have been able to couple these estimates with actual success and failures to determine the accuracy of probability assessments.

10. Figure 1 PAG Performance PAG predicted a probability of 0.80 146 times over the last 13 years. 118 of those events were successes, for an observed success rate of 0.81. 0.90 146 0.80 0.70 0.60 Actual Success Rate 0.50 0.40 0.30 0.20 0.10 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 PAG probability assessment

11. Figure 2 PAG Performance, all raw data 2 2 2 1 1 1 1 1 1 1 2 2 1 23 Most of the “high-sample size” observations lie within a +/- 0.10 band about the target line 134 0.90 146 0.80 4 0.70 70 0.60 111 45 Actual Success Rate 0.50 2 2 30 22 0.40 3 18 0.30 23 11 24 0.20 8 0.10 1 3 1 10 1 1 2 1 2 2 1 1 1 2 1 2 1 2 1 1 1 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 PAG probability assessment

12. Figure 3 Collecting Nearby Observations into Buckets, view 1 139 0.90 In this analysis, “nearby” observations were combined together into adjacent intervals centered at the 2½% points. 153 0.80 96 0.70 0.60 117 58 Actual Success Rate 0.50 • For example, there were: • 5 observations at 0.48 (4 successes) • 45 observations at 0.50 (24 successes) • 2 observations at 0.51 (2 successes) • 1 observation at 0.52 (0 successes) • 2 observations at 0.53 (0 successes) • 1 observation at 0.55 (0 successes) • 2 observation at 0.57 (0 successes) • These combined for: • 58 observations at 0.525 (30 successes) 0.40 49 56 0.30 41 0.20 0.10 19 5 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 PAG probability assessment

13. Figure 4 Collecting Nearby Observations into Buckets, view 2 3 140 0.90 In this analysis, “nearby” observations were combined together into adjacent intervals centered at the 7½% points. 173 0.80 0.70 75 0.60 119 Actual Success Rate 0.50 78 • For example, there were: • 3 observations at 0.34 (1 success) • 18 observations at 0.35 (6 successes) • 2 observations at 0.36 (1 success) • 1 observation at 0.37 (0 success) • 30 observations at 0.40 (13 successes) • 1 observation at 0.42 (0 success) • These combined for: • 55 observations at 0.375 (21 successes) 0.40 55 52 0.30 0.20 22 0.10 16 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 PAG probability assessment

14. Figure 5 Collecting Nearby Observations into Buckets, view 3 2 138½ 0.90 In this analysis, intervals were centered about the deciles, and observations at 0.05, 0.15, …, 0.95 were split half/half between adjacent intervals. 164 0.80 84½ 0.70 0.60 118½ Actual Success Rate 0.50 67 0.40 56½ 49 0.30 • For example, there were: • 23 observations at 0.45 (7 successes) • These were split “evenly” between the intervals centered at 0.40 & 0.50. 0.20 33 0.10 17½ 2½ 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 PAG probability assessment

15. Summary • The PAG assessments have been remarkably accurate – no matter how the data are grouped the actual results are usually within 5% of the predictions, and never more than 10% away. • Our experience has shown that a well-planned process for probability assessment can provide executives with reliable measurements of technical feasibility. • A careful consideration of technical feasibility is key to portfolio management. • Probability is an excellent language for quantifying this uncertainty.