scienceintransition.nl

1. www.scienceintransition.nl @SciTransit en @MiedemaF

2. Why and how science has gone wrong Science in Transition started in Januari 2013 Frank Miedema (UMC Utrecht) Huub Dijstelbloem (WRR, UvA) Frank Huisman (UMC Utrecht, Descartes Centre, UU) Wijnand Mijnhardt (Descartes Centre, UU)

3. Science in Transition POSITION PAPER October 17, 2013 ‘Why Science Does Not Work as It Should And What To Do about It’ http://www.scienceintransition.nl/wp-content/uploads/2013/10/Science-in-Transition-Position-Paper-final.pdf International Symposiums @Royal Academy of Sciences, Amsterdam, 7 and 8 November 2013 and 1 December 2014

4. 1971

5. 1979

6. 2000

7. Dutch 2010 English 2012

8. October 2013

9. NRC 2 nov 2013

10. Cycles of Credit ‘ Volkskrant 9 Nov 2013

11. Science in Transition is part of a large international chorus 2013 San Francisco Declaration on Research Assessment: stop using bibliometric indices for evaluation of researchers NIH: Reproducibility initiatives in life sciences and psychology NEJM: Transforming Academic Health Centers for an Uncertain Future Economist: How Science Goes Wrong, ‘Trouble at the lab’ 2014 Lancet: increase value and reduce waste in biomedical research (8 papers, 9 January) NIH, F. Collins & Tabak in Nature, January 2014: increase reproducibility and change academic incentive system Nobel prize winners Schekman and Brenner point to impactfactormania Alberts, Varmus et al, Rescuing US biomedical science from systemic flaws, PNAS, March, 2014

12. Today, annually ≈ 1.5 million scientific publications 3% annual growth

13. 2006: 1,35 miljoen publicaties

14. Biomedical Research suffers from major systemic flaws Womens Health, CVD and Oncology: only about 25% of published preclinical studies could be validated to the point at which projects could continue (Prinz et al, Nature Reviews Drug Discovery, 2011) Amgen: of 53 landmark oncology papers in only 6 (11%) cases scientific findings were confirmed (Begley and Ellis, Nature, 2012) Several papers and 20 authors and an Editorial by Richard Horton, Lancet, January 9, 2014 85% of research investment is wasted—equating to $200 billion of the investment in 2010 Significant problem with reliability of the literature because of publication bias.

15. Lancet January 9, 2014: reduce waste, increase value Much biomedical research does not lead to worthwhile achievements. 85% of research investment—equating to $200 billion of the investment in 2010—is wasted Significant problem with reliability of the literature because of publication bias. 1575 reports about cancer prognostic markers published in 2005, 1509 (96%) detailed at least one significant prognostic variable. However, few identified biomarkers have been confirmed by subsequent research and few have entered routine clinical practice Ioannidis, Altman, Chalmers, Glaziou, Horton a.o

16. Science in Transition* • Major developments in science over the past 30 years: • The scale of the enterprise has increased in numbers of people involved in many countries • Big Science: larger, international and often multidisciplinary teams and consortia obtain funding • science is increasingly capital intensive, with rapid depreciation of expensive equipment, etc • The business cycle is much shorter: technologies, methods, break through concepts arise faster; resulting in much faster ‘creative destruction’ • Management has changed in response to this to become more business-like (NPM) *Miedema, Science 3.0, AUP and Chicago University Press, 2012 ‘ Volkskrant

17. Problems of the system (1) • Hypercompetition • Short- term, high-rate article production • because of careerism and economics • Individual goals not always in ‘sync’ with overal aim of science • Quality control suffers: • because of the rapid increase in scale and • competition reviewing is poor, replication not done; • all this because of lack of reward and time ‘ Volkskrant

18. Problems of the system (2) • Problem choice is determined by risk reduction and risk avoidance • Relation with society: because of science ideology and overselling, image and understanding of science becomes problematic • Problem choice in democracy: who is settting the agenda? • The mores and behaviour of administrators, scientists and students have changed. Quantity dominates over Quality and CONTENT in research evaluation (New Public Managment) ‘ Volkskrant

19. Science in Transition:to understand and begin to correctwe need a systems approach to science Volkskrant

20. A Toolbox for Science in Transition

21. Bruno Latour and Steve Woolgar (1979)Laboratory Life

22. Science is a sociological ‘field’ ‘ Pierre Bourdieu, Science of Science, 2004 Hessels et al, Science and public policy, 2009 Volkskrant

23. A Toolbox for Science in Transition Science in Transition: a Science of Science Project No overselling: Stop telling the myth of a perfect scientific method The evaluation of research impact needs to change: from Internal Scientific to Societal Impact: involve potential users Target Societal Problems Enable more long term, high risk, multidisciplinary work by changing granting schemes From PI system to composite teams PhD Talent Management and Education Rethink the Idea of the University, teaching versus research

24. A Toolbox for Science in Transition Reflections 1. Science in Transition: a Science of Science Project • Scientific research on the current system of science. • Multidisciplinary theoretical and empirical work, mainly on the sociology and economy of science, • Historical analyses of the changes that science has gone through since World War II. 

25. A Toolbox for Science in Transition 2. Impact The evaluation of research impact needs to change. The simple use of journal impact factors in funding, appointment and promotion must be abandoned, see the San Francisco Declaration on Research Assessment. Meaningful Metrics needed for both ‘science for science’ and ‘science for society’ (societal impact); parameters of value attributed by potential users in society must be developed with stakeholders outside academia. The use of Case studies must be developed

26. A Toolbox for Science in Transition 3. Incentives and Rewards • Grants from government, EU, charities, etc should explicitly be awarded based on evaluation of Composite Impacts. This must result in a mix of basic and targeted research, where also basic science is judged in a wider context. • Research management will have to take into account teamwork, the various types of academic impact, including teaching and ‘academic duties’ and correspondingly provide career opportunities for the different types of researchers* *CWTS Leiden, Authorship in Transition, February 2015

27. A Toolbox for Science in Transition 4. Dealing with Risk Avoidance • High profile personal grants tend to favor researchers based on the usual metrics, who produce knowledge for the internal science market mainly (Science for Science). • This promotes risk reduction in grant writing and review procedures; too much emphasis on short term production of high impact publications. Long term sustainable funding schemes are required. • This results in limited diversity of leading investigators in the institutesand too little investment in staff to build a sustainable knowledge base.

28. A Toolbox for Science in Transition 5. PhD Talent Management and Education • Our graduate school ‘PhD factories’ provide poor prospects for PhDs and post docs that want to make a career in academia. • We have to rethink to better coach our students for careers in and out side science: How much PhD’s should we produce? Do MD's in high numbers need a PhD? What are optimal criteria for PhD theses • We have to promote scientific literacy among our Master and PhD students. Paula Stephan: How economics shapes Science, 2012

29. A Toolbox for Science in Transition 6. Targeting Societal Problems • Trust in science and scientists is not only about how trustworthy and careful we do and report our science, ………… • it is as much about the question whether we do research on the right questions. • We have to relate more with stakeholders in society to orient our research, both basic and targeted research, this has already started in anticipation of EU HORIZON 2020 which targets the so-called Grand Challenges. 

30. A Toolbox for Science in Transition 7. Communication • Stop telling the myth of a perfect method performed by individuals with high moral values without any bias or interests. • Be honest to tell about science being a job and how the economy of science shapes science and the content of our science. That scientists have personal beliefs that shape their scientific ideas. • Explain how science does make objective facts, but that a lot of uncertainty remains in particular regarding complex real world problems.

31. A Toolbox for Science in Transition 8. Problem Choice in Democracy • Finally the problem of agenda setting in a democratic society presents it self. How are we organizing this process designated by Philip Kitcher the 'ideal deliberation’.* • How do we tutor the interests of weaker public parties that compete with powerful private and government advocy for research money • Is Mode-2 science the right direction for academia?** Will there be enough room for ‘free’ basic science, if that exists at all? * Kitcher, 2001 and 2011 **Nowotny et al Minerva 2003

32. Launching conference at Royal Academy of Sciences and publication of ‘position paper’ (nov 2013) After debates at all levels and an avalanche of publicity, personal invitation of Science minister to discuss current problems (March 2014) European Commission recognizes relevance of Dutch debate in “Science 2.0: Science in Transition” analysis (July 2014) Science in Transition participates in Responsible Research and Innovation conference in Rome (Nov 2014) Dutch government report “2025 Vision for Science: choices for the future” promotes rethinking scientific quality, advocates stakeholder interaction and proposes a National Research Agenda (Nov 2014) Dutch association of universities signs San Francisco Declaration on Research Assessment at 2nd Science in Transition conference (Dec 2014) Science in Transition milestones

33. The REWARD Statement we set the right research priorities we use robust research design, conduct and analysis regulation and management are proportionate to risks all information on research methods and findings are accessible reports of research are complete and usable METRICS is a research to action center focused on transforming research practices to improve the quality of scientific studies in biomedicine and beyond

34. Impact of Science in Transition in The Netherlands Standard Evaluation Protocol for universities drops “quantity” as a separate category of quality

35. Onderzoek Nederland, 1 mei 2015

36. Science in Transition reduced to practice: UMC Utrecht strategy ‘Connecting U’ Multidisciplinary research programs Involve stakeholders in prioritizing research New indicators for scientific quality Diversify talent management Evaluation with hybrid panels Educate students about the science system

37. UMC Utrecht strategy 2015-2020 “Connecting U”Societal impact of research programsMultidisciplinary research and innovation Innovation Fundamental research Translational research Clinical trials Safety & Efficacy studies Experimental & Top clinical care Basic care Brain Infection & Immunity Circulatory Health Personalized Cancer Care Regenerative Medicine & Stem Cells Child Health Patient Patient Supporting Technologies Care Delivery Organization Clinicalneed

38. Societal stakeholders involved in research agenda setting and evaluation ‘UMC Utrecht in forefront of patient involvement’ ‘Organize structural involvement of stakeholders’ ‘Start using indicators that show societal impact’

39. Case study Societal Impact Organoids in Cystic Fibrosis >1/3 of all Dutch Children with CF is treated in UMC Utrecht The UMC Utrecht chaired the development of national treatment guidelines for CF The UMC Utrecht stimulated and activated the Dutch Cystic Fibrosis Foundation in the role of medical advisorship and as chair of a National Research Network on CF initiated by the patient organization. This resulted in active participation of patients in setting the research agenda on CF and in strong endorsement The UMC Utrecht played a leading role in organizing pilot trials (Chopin study), advising and enabling governmental decision to initiate a national Heel Prick Screening program on CF in 2011 of patient participation in trials The organoid model has been patented and the UMC Utrecht currently provides this model as a drug development testing model for important pharmaceutical developers like Pfizer and Galapagos. Dekkers et al Nature Med 2013

40. Concept-indicators for impact

41. Launching conference at Royal Academy of Sciences and publication of ‘position paper’ (nov 2013) After debates at all levels and an avalanche of publicity, personal invitation of Science minister to discuss current problems (March 2014) European Commission recognizes relevance of Dutch debate in “Science 2.0: Science in Transition” analysis (July 2014) Science in Transition participates in Responsible Research and Innovation conference in Rome (Nov 2014) Dutch government report “2025 Vision for Science: choices for the future” promotes rethinking scientific quality, advocates stakeholder interaction and proposes a National Research Agenda (Nov 2014) Dutch association of universities signs San Francisco Declaration on Research Assessment at 2nd Science in Transition conference (Dec 2014) Science in Transition milestones

42. The REWARD Statement we set the right research priorities we use robust research design, conduct and analysis regulation and management are proportionate to risks all information on research methods and findings are accessible reports of research are complete and usable METRICS is a research to action center focused on transforming research practices to improve the quality of scientific studies in biomedicine and beyond

43. Impact of Science in Transition in The Netherlands Standard Evaluation Protocol for universities drops “quantity” as a separate category of quality

44. Onderzoek Nederland, 1 mei 2015

45. @SciTransit en @MiedemaF www.scienceintransition.nl