Irreproducibility in biomarker research and clinical development: A global challenge

1. Irreproducibility in biomarker research and clinical development:A global challenge Andreas Scherer PhD EATRIS Research Infrastructure for Translational Medicine Institute for Molecular Medicine Finland (FIMM), Univ. Helsinki Spheromics

2. Biomarker Applications • Academic and pre-clinical research • e.g. monitoring validity of a hypothesis • Personalized Medicine • monitoring success of treatment • health status • e.g. Genetic Risk Score to predict coronary heart disease1; • e.g. metabolomics panel to predict 5-year mortality2 • … • Drug development • monitoring success of compound development Medieval “wheel of urine” 1, Tikkanen et al., Arther Thromb Vasc Biol 2013; 33: 2261-2266. 2, Fischer et al., PLOS Medicine, 2014 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

3. Biomarkers in Drug Development 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

4. Advantages of biomarker-guidedtargeted therapy • Increase treatment specificity (target specification, patient stratification) • Decrease size of trials • Improve ability to show value of intervention: decrease overall costs created by identification of non-responders • Patient advantages: improve chances of response, decrease exposure to ineffective therapy • Possibility of therapy/biomarker co-marketing 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

5. However, most compounds still fail during drug development due to… • adverse reactions/side effects/safety • poor absorption, distribution, metabolism, excretion profile • lack of efficacy • lack of understanding of biology/disease • high cost • intellectual property • … 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

6. “To increase R&D success rates we need to move from serendipity to enhanced predictability” Gerd Maass, Vice Pres., Roche Diagnostics 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

7. What do we need to leave the “valley of serendipity”? • Testable hypotheses based on reproducible, trustworthy research findings • Commitment to generate high-quality, reproducible results and reports • Allow timely pipeline GO/NO-GO decisions, e.g. by employing fit-for-purpose marker validation • High-quality reporting • Reliable biomarker data that is scientifically and clinically meaningful to address safety, mechanism of action, patient selection /stratification 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

8. Challenges of biomarkers in drug development • The need to understand precisely what the biomarker can measure (and what it cannot!), and the role it can play in the development or clinical management process • The need for a reliable test or assay which can reproducibly demonstrate the required results • The constant evolution of testing technology which reshapes the boundaries of detection and clinical applicability • The possibility of a novel biomarker emerging, forcing a re-evaluation of the planned development strategy 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

9. Hypothesis • Meaningful pre-clinical and clinical research relies on repeatability and reproducibility of the data • Repeatability The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions(same operator, same apparatus, same laboratory and after short intervals of time). • Reproducibility The closeness of agreement between independent results obtained with the same method on identical test material but under different conditions(different operators, different apparatus, different laboratories and/or after different intervals of time). 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

10. Example from a single lab: Biomarker of clear cell renal cell carcinoma (ccRCC) Goal: establish FFPE samples as sources of transcriptome data information for ccRCC research Source Patients (n=16) Discovery Renal biopsies (4 per patient) Tissue ccRNA (2x) unaff. ctrl. (2x) (n=64) Diagnosis Storage FFPE RNAlater Data RNAseq RNAseq Analysis Phase I Gene expression profiling Biomarker analysis Gene expression profiling Biomarker analysis Comparison Cross-validation Analysis Phase II Confirmation Patients (n=12) FFPE only OysteinEikrem Prof Hans-Peter Marti University of Bergen, Norway PLoS One. 2016 Feb 22;11(2):e0149743. doi: 10.1371/journal.pone.0149743. eCollection 2016. 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

11. Discovery vsconfirmation dataset Correlation of expression levels 0.96, and fold changes (commonly diff.exp.transcripts) 0.93 Cluster by diagnosis, not dataset 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

12. Biomarker candidate:TNFAIP6 Discovery TNFalpha induced protein 6 A: scatterplot discovery datasets B: classifier performance discovery set C: scatterplot microarray dataset (GSE76207, www.ncbi.nlm.nih.gov) D: classifer microarray dataset E: scatterplot FFPE discovery vs confirmation F: classifier performance FFPE datasets Microarrary TNFalpha induced protein 6: Secretory protein, hyaluronan-binding; Diseases associated with inflammation (arthritis); Positively modulates epithelial-mesenchymal transition (EMT) Not reported yet in the context of ccRCC Discovery vs confirmation Eikrem et al., unpublished, submitted 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

13. (Ir-)Reproducibility in Preclinical Research • Systemic and costly inefficiencies in designing, conducting, and reporting preclinical studies Experimental reproducibility Experimental reproducibility Resource identification Treatment description Experimental reproducibility based cumulative error estimates in study design, reagents/materials, protocols, and analysis Freedman et al, PLOS Biology, 2015 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

14. Reasons for irreproducibility https://www.gbsi.org/gbsi-content/uploads/2015/10/The-Case-for-Standards.pdf Leonard Freedman President GBSI 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

15. How to address these issues 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

16. Measures to increase reproducibility 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

17. Use of formal quality systems https://www.gbsi.org/gbsi-content/uploads/2015/10/The-Case-for-Standards.pdf 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany Irreproducibility Absence of a unifying quality control and quality assurance framework

18. Goal: Guideline development for life sciences CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 97 NUMBER 1 | JANUARY 2015 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

19. Advancing best practices and standards by engaging stakeholders 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

20. Ongoing independent biomarker activities Europe {Asadullah et al, Nature Reviews Drug Discovery, Dec 2015} USA 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

21. EATRIS:European Infrastructure for Translational Medicine • 12 member states across EU • Provides “one stop shop”-access to 70+ academic medical institutions across Europe • Unique combination of expertise, facilities and technologies • Access to broad array patient cohorts, including rare diseases • Harmonised legal framework, substantially decreasing transaction time • EATRIS helps you make use of existing technical, clinical, regulatory, and IP expertise in Europe: no need to bring technology in house; saves cost, and time EATRIS member state Negotiations ongoing

22. EATRIS: Current activities in reproducibility/data quality initiatives • SEQC2: FDA-initiative, human genome variant analysis • Global Biological Standards Institute: antibody and cell line characterization • JMAC (Japanese Multiplex-Analysis Consortium): microRNA • Global consortium: high throughput screening technologies • … 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

23. The “Global Consortium”:EATRIS – CDRD – NCATS - TIA EATRIS member state under negotiation “Global collaboration in translational science promises to accelerate the discovery, development, and dissemination of new medical intervention…” ”…Integrating and leveraging the cumulative knowledge of translational organizations on a global level promises to more rapidly advance innovative medicines to patients…” Nature Reviews Drug Discovery, 217-219, vol. 15, 2016 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

24. Conclusions • Challenges to ensurereproducibility for clinicaldevelopment • aretoolarge for individuallaboratories • Setup and management of consortia is workintensebutnecessary • Working in consortiaprovides • - cross-laboratorycomparability of data and protocols • - visibility • - increasedlikelihood of funding • - increased likelihood of acceptance of results in scientific community • (consensus-building) 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

25. If a job is worth doing, it is worth doing twice Jonathan F. Russell. 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

26. Thank you Contact: Andreas Scherer PhD National Coordinator - EATRIS The European Infrastructure for Translational Medicine Project Director - Biomarkers and Molecular Diagnostics, FIMM Email AndreasScherer@EATRIS.eu Andreas.Scherer@Helsinki.fi http://www.eatris.eu http://www.fimm.fi/en/ 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

27. 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

28. Lack of reproducibility of published research on potential drug targets According to studies by researchers from Bayer and Amgen, 65% of published scientific results on drug targets can not be reproduced • Potential reasons: • Inappropriate statistical analysis • Small sample size • Intra-and interlab variability • Insufficient quality of reporting • Variation in cell-line stocks, suppliers, etc. • False numbers • Fraud Prinz et al., Nature Reviews Drug Discovery 10, 712 (September 2011) 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

29. Ways out… • Animal models: carefully characterize the model; experiment design, • Antibodies: antibodypedia.com: 1.8mio antibodies with validation rating Antibodies-online.com: programme for independent labs to perform validation studies (at vendors’ expense): of 275 studies, less than 50% could be validated • Cell lines: Journals ask authors to check their cell lines (e.g. go.nature.com/zqjubh) • Reproducibility Initiative: independent validation of research findings. Submitted experiments are matched with an appropriate, verified “Science Exchange”- lab who reproduces the experimentshttp://validation.scienceexchange.com/#/reproducibility-initiative • Consortia: example: Microarray Quality Control/Sequencing Quality Control (MAQC/SEQC) (FDA-initiated) • MAQC I: test the reproducibility of microarray data across platforms and labs • MAQC II: test analytical procedure to determine clinical endpoints through microarray data • MAQC III (“SEQC”): test the reproducibility of Massively Parallel Sequencing technology across labs and technology platforms 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

30. “Reproducibility crisis”: examples • Animal models of disease: often underpowered, not well characterized, more than 80% of potential therapeutics fail in people, after tested positive in animal studies example: (1) Mouse models expressing a mutant form of the RNA binding protein TDP43 show hallmark features of ALS: loss of motor neurons, protein aggregation and progressive muscle atrophy. TDP43-mutant mice did not progress in the paralysis accompanying ALS. (2) TGN1412, anti-CD28 superagonist, dosage and MOA in human different from NHP • Cell lines: false identification of cell lines ( problems already found in more than 400 cell lines) • Antibodies: batch-to-batch variation, some are poorly characterized, non-selective binding • Reporting: • bias, only “positive” results are published • methods are incomplete or false • Analysis errors: insufficient study design, insufficient statistical analysis, lack in statistical training, biased/incomplete reporting) 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany

31. Validity of data: common threats 7th Conference and Expo Molecular & Cancer Biomarkers, Berlin, Germany