Enhancing Research Rigor & Reproducibility Webinar

1. WELCOME TO: FASEB Webinar on “Enhancing Research Rigor & Reproducibility” THE WEBINAR WILL BEGIN IN A FEW MINUTES Click the join link in your Confirmation email. If you don't have your link, go to https://www.gotomeeting.com/webinar/join-webinar and enter WebinarID: 632-015-435 To use your telephone to hear the audio, call 631.992.3221and enter Access Code 346-707-600 If you experience trouble joining the webinar, please contact Customer Support at https://care.citrixonline.com/gotowebinar/join

2. Enhancing Research Rigor & Reproducibility Yvette Seger, PhD

3. Agenda for Today’s Webinar What is FASEB? Introductionof FASEB’s involvement in research reproducibility Overview of FASEB’s 2016 recommendations Recent FASEB initiatives Questions & Answers

4. To Ask A Question Type your question in the white box Click “Send” (gray button)

5. FASEB, who we are • A federation of 30 societies • Representing over 130,000 researchers • Our Mission… • Advance health and welfare by promoting progress and education in biological and biomedical sciences through service to our member societies and collaborative advocacy.

6. Our FASEB Societies

7. How FASEB got Involved

8. Making (Unwanted) Headlines

9. NIH Signals Action

10. May 31, 2015 FASEB SPC Symposium Symposium Goals: • Explore the landscape of existing initiatives intended to enhance scientific rigor and research reproducibility • Federal Agencies (NIH, NSF) • Private Organizations (GBSI, ATCC, Science Exchange) • Identify potential opportunities for FASEB action in anticipation of new NIH policy

11. June 9, 2015 - NIH Policy Released (1) Effective Jan. 25, 2016, applications for research project funding must include the following: • Scientific Premise • Describe general strengths and weaknesses of prior research cited as crucial to support the application • Discussion of rigor of previous experimental design • Attention to incorporation of biological variables and authentication of key resources • Rigorous Experimental Design: • Describe how robust, unbiased results will be achieved in experimental design

12. June 9, 2015 - NIH Policy Released (2) • Consideration of Sex and Other Relevant Biological Variables: • Factor biological variables, including sex, age, weight, and underlying health conditions, into discussion of study design • Justify premise if choose to study only one sex • Authentication of Key Biological and/or Chemical Resources: • Plans for regular authentication of resources to ensure identity and validity • NIH encourages the research community to inform the development of such guidelines

13. Opportunity for FASEB to Act • FASEB as convener of key stakeholders in policy discussions and development of community-based best practices • Complement existing efforts: Increased awareness of issue and training of early-career investigators and cell line authentication e • Unique to FASEB: • Discussion and development of best practices for the use and reporting of mouse models and antibodies in pre-clinical research and engaging other community stakeholders in the development of appropriate resources • Pilot novel approaches to enhance transparency in conference presentations via the FASEB Science Research Conferences

14. FASEB Roundtable Meetings (2015) • August: Mouse Models • September: Antibodies • October: Community Stakeholders • Each society given the opportunity to identify a delegate for each roundtable meeting • Each meeting featured 4-5 invited speakers • Active participation by NIGMS leadership in all three meetings; Additional NIH representation in October; NSF BIO Associate Director also attended October meeting • Society Executives and Public Affairs staff also invited to participate

15. Roundtable Goals Explore common issues that decrease replicability of studies using mouse models and/or antibodies Develop consensus principles to guide researchers in controlling minimizing these effects Exchange information about the outcomes of other organizations’ efforts to address rigor and reproducibility Identify areas of consensus, seek opportunities to bridge areas of disagreement, and discuss possible future actions Identify emerging topics and concerns related to scientific rigor and reproducibility

16. Overall Goal of Roundtables Create a resource to increase awareness of and facilitate compliance with new NIH requirements to address rigor and transparency in grant applications

17. FASEB Recommendations Released January 14, 2016

18. FASEB Recommendations • Organized into four sections: • Overarching (3) • Mouse and Other Animal Models (6) • Antibodies (3) • Strategies for Professional Societies/Organizations (4) Total of 16 recommendations

19. Overarching Recommendations • Use of the same terms and definitions when discussing research rigor and transparency to promote uniform understanding of the issue • Need for stakeholder dialog to develop uniform expectations for describing reagents, laboratory tools, and protocols in scientific communications • Reinforce good laboratory practices as part of research training (record keeping and reporting, use of standard nomenclature, experimental design, proper use of statistics)

20. Role for FASEB • FASEB will continue to facilitate dialog on enhanced rigor and reproducibility by synthesizing views of research community regarding clarity of the NIH policy, challenges encountered, and resources that could help streamline compliance

21. Follow-On Activities

22. SRC Transparency & Rigor Initiative

23. Timeline Key Question: What are effective ways to provide more transparent communication of research methods and findings in short presentations/posters? • February 2017: Workshop with NINDS and conference organizers to discuss feasibility of integrating icons into conference presentations

24. SRC Transparency & RigorPilot • Introduced at 2017 FASEB Science Research Conferences (SRCs) • Voluntary participation • Tested the use of four icons to communicate specific aspects of experimental design • Post-meeting survey questions • Awareness of NIH Rigor & Reproducibility Initiative • Feedback on icons tested in pilot

25. Summary of 2017 Pilot Findings General skepticism toward broader NIH Rigor & Reproducibility initiative Need for better awareness and socialization of icon pilot Need for uniform guidelines regarding integration of icons into presentations Expand icon options to be relevant to more fields/lines of research

26. Updates for the 2018 Pilot (1) • Improve socialization of the pilot • More detailed description of the pilot to be included in meeting planning and registration materials • Inclusion of a standard, brief overview of the pilot as part of organizer’s remarks during first plenary session of participating meetings

27. Updates for the 2018 Pilot (2) • Enhanced Instructions for Icon Use • Incorporation of icons on relevant data slides, not just the title slide • Provide downloadable image files of icons included in pilot • Offer specific guidance regarding appropriate image size

28. 2018 SRC Icons Consideration of Sex As a Biological Variable Littermate Controls Blinded Analysis Randomization Confirmatory Analysis Preliminary Analysis Database Deposition Positive & Negative Controls Shown

29. Communication Workshop

30. Partnership with NIGMS • Shared interest in improving rigor and reproducibility of basic biomedical research • Desire to identify most effective ways to convey importance of basic biomedical research • Access to different stakeholder groups to effect change SHARED GOAL: Identify best practices for communicating basic biomedical science across the range of potential stakeholders

31. June 2017 Workshop “Responsible Communication of Basic Biomedical Research: Enhancing Awareness and Avoiding Hype” Agenda centered around four key questions: How does science communication affect the biomedical research landscape? How does science communication affect public perception of science? What are the goals and incentives of science communication? How can all stakeholders work together to improve communication of scientific results?

32. Panelists • Biomedical Researchers: • Bruce Alberts, PhD • Angela DePace, PhD • Lee Ligon, PhD • Journalists: • Jocelyn Kaiser • Sara Reardon • Ivan Oransky, MD • Regina Nuzzo, PhD • Communications Researchers: • Joseph Cappella, PhD • Anthony Dudo, PhD • Marketing Perspective: • Kirk Englehardt • Matt Shipman • Government/Policy: • Erik Fatemi • Industry: • Fintan Steele, PhD Keynote: Erika Check Hayden

33. Challenges Identified • Public understanding of science • Complexity and uncertainty • Length of research process • Pressure to connect results with clinical applications • Varying incentive structures • Uncertainty of when/how to communicate findings

34. Recommendations (1) • Effective Communications • Provide communications training and resources to scientists throughout their careers • Tell a good story that conveys excitement and piques audience’s sense of curiosity • Use metaphor, imagery, and illustrations to explain complex concepts or techniques • Make use of the full range of communication platforms to reach different audiences (e.g., YouTube, Instagram, Twitter, blogs, and online whiteboards)

35. Recommendations (2) • Incentives of Communication • Emphasize scientific process rather than single research studies • Develop and follow a logic model to outline both long-term and short-term communication goals and establish appropriate tactics • Consider which news outlet(s) can best reach the intended audience

36. Recommendations (3) • Reconsider Press Releases • Share leads on work in development • Identify scientists who can comment on stories in the news • Embrace and help others understand the uncertainty of scientific inquiries

37. Meeting Materials FASEB Website

38. FASEB BioAdvances

39. FASEB BioAdvances New international peer-reviewed, open access journal In addition to reports of new findings, the journal considers articles with confirmatory results.

40. FASEB BioAdvances FASEB BioAdvances publishes articles that successfully--or unsuccessfully--replicate the methods and results of previously published articlesas Research Reproducibility Reports

41. Websites • Enhancing Research Reproducibility Report http://www.faseb.org/Science-Policy--Advocacy-and-Communications/Science-Policy-and-Research-Issues/Research-Reproducibility.aspx • SRC Transparency & Rigor Initiative http://www.faseb.org/Science-Research-Conferences/Transparency-and-Rigor-Initiative.aspx • FASEB/NIGMS Communications Workshop http://www.faseb.org/Science-Policy--Advocacy-and-Communications/Communicating-Science/Avoiding-Hype-Workshop.aspx • FASEB BioAdvances https://onlinelibrary.wiley.com/journal/25739832

42. Q&A

43. To Ask A Question Type your question in the white box Click “Send” (gray button)

44. Contact Me! Yvette Seger, PhD yseger@faseb.org