Promoting Research Integrity Historical Background & Current Trends

1. Promoting Research Integrity Historical Background & Current Trends Nicholas H. Steneck, PhD Subtitle …from a Misconduct- to an Integrity-centered Universe Workshop on Best Practices for Ensuring Scientific Integrity & Preventing Misconduct 22-23 February, 2007 Mita Conference Hall, Tokyo, Japan

2. http://www.talkorigins.org/faqs/piltdown/pskull.gif http://en.wikipedia.org/wiki/Image:Galileo.arp.300pix.jpg Historical Perspective • Research misconduct is not new • Galileo fabricated & falsified data • Piltdown hox and other scandals • Some things do not change over time • Major changes between 1600 and 2000 • Science ca. 1600 • Privately supported • Limited public benefit • Science today • Publicly supported • Essential to technological growth & public welfare • By 1980, when first cases of misconduct emerge, science (research) is a public activity • Over time, scientists (researchers) became public servants!

3. http://www.nsf.gov/statistics/databrf/sdb99357.pdf Major growth since 1950 • Financial support • 1930s, under 1% GDP • 1950 ca. 1% GDP • Currently 2-3% • 1 of 100 public dollars in US spent on health research • Research is a public activity

4. http://www.thebulletin.org/minutes-to-midnight/ Some doubts along with the growth • Science linked with unpopular events and problems: • Cold War links science with “military-industrial complex • Clock, Bulletin of the Atomic Scientist (1947) • War in Vietnam (1960s) • Environmental impact of nuclear power • Concerns about human & animal experimentation • US, Tuskegee experiments • Declaration of Helsinki (1964) • Global energy crisis (1970s) • New worldviews compete with scientific worldview • By 1980 (first major US misconduct cases) public is taking a closer look at the way research is conducted

5. Response to research misconduct, 1980 ff. • Events have been driven by major cases & media • Story breaks in the news • Local institution responds • Pressure for an official/government response • Official response • Gather information ~ committees, hearings, reports • Try to resolve the immediate problem ~ the major cases • Develop policies and procedures to avoid similar problems in the future • Timing • US begins in early 1980s, policy development still in process • Europe, early 1990s, policy development in process • Asia, late 1990s, policy development in process

6. The misconduct-centered universe • First priority ~ major cases • Define misconduct • Assign authority • Develop procedures for investigation • Definitions focus on deliberate reporting false data & information • Careful to not confuse with scientific disagreement • Not the same as waste and sloppiness • Policies protect researchers from improper charges • Working assumption: pursuing individual cases of misconduct is the best way to protect the integrity of publicly supported research FFP

7. Misconduct-based universe rested on 5 assumptions • Serious misconduct in research is rare • Self-regulation keeps improper behavior in check • Research misconduct is difficult to detect • Research misconduct cannot be prevented • Apart from misconduct, standards for integrity in research are high • Assumptions were based on common perceptions, not empirical evidence • All five can be questioned!

8. Research misconduct is rare? • Martinson, Nature (June 2005) • Goal: factors that influence research behavior • Method: • Developed peer-based list of major offenses • Survey to 6,000+ researchers (3,000+ response) • Major question: “have you done … in last three years?” • Results • Major offenses, ca. 0.3% • Questionable Research Practices (QRP) ca. 5-15% or higher

9. Data from other recent studies • JM Ranstam, Control Clin Trials (2000) • Survey, 442 biostatisticians, 37% response • 51% knew about fraud in medical research • 26% involved FF • 31% directly involved in projects with misconduct • Estimates of rate, .69% –> .80% (.25% standard) • Geggie, J Med Ethics (2001) • Survey, 305 new medical consultants, 64% response • 55.7% observed misconduct (FF lower) • 5.7% committed misconduct in the past • 18% would commit in future • 17% had research ethics training

10. Studies continued • Gardner, Contemporary Clinical Trials (2005) • Authors pharmaceutical clinical trials (64% response) • 1% reported target article misrepresented the research • 5% reported fabrication in a study they had participated in over the last 10 years • 17% knew personally of fabrication in a study over the last 10 years • Rossner, Journal of Cell Biology • 11 in 1,100 papers had serious improper digital image manipulation

11. Science & Engineering Indicators 2006, Figure 3-36 Realistic estimates: • Rough approximation: • Evidence ~ 1/1,000+ • Assume ~ 1/10,000 • Cases predicted • US ~ 1,500 • EU ~ 1,000 • Japan ~ 600 • Other OECD ~ 400 • Cases seen • US ~ 20/year • EU ~ 10/year • Lesson #1 ~ policy makers have always under-estimated the amount of misconduct in research

12. Other assumptions about misconduct? • Self-regulation keeps improper behavior in check? • Researchers do not report suspected misconduct (20-40%) • Journals often do not report misconduct • Research misconduct is difficult to detect? • Hwang could not have completed work in the time reported • Sudbø, trial not started, patients did not exists, data repeated • Research misconduct cannot be prevented? • Schön’s co-author/mentor did not check experiments or data • Pohlmann’s MD co-author did not oversee clinical results • Lesson #2: Policy makers did not understand the strengths & weaknesses of self-regulation in research • If they understood, did not honestly and accurately report

13. Integrity in research otherwise high? • Martinson study, self-reported misbehaviors: 15.5% Changing the design, methodology or results of a study in response to pressure from a funding source 12.5% Overlooking others' use of flawed data or questionable interpretation 7.6% Circumventing certain minor aspects of human-subject requirements 6.0% Failing to present data that contradict one’s own previous research 1.7% Unauthorized use of confidential information 1.4% Using another’s ideas without obtaining permission or giving due credit 1.4% Relationships with students, research subjects or clients that may be interpreted as questionable 0.3% Not properly disclosing involvement in firms whose products are based on one‘s own research 0.3% Ignoring major aspects of human-subject requirements 0.3% Falsifying or ‘cooking’ research data

14. Al-Marsouki, Contemp Clin Trials 26(2005) • Practices felt likely to occur and adversely impact research 83% Over-interpretation of “significant" findings in small trials 80% Selective reporting based on p-values 76% Selective reporting of outcomes in the abstract 75% Subgroup analyses done without interaction tests 68% Negative or detrimental studies not published 68% Putting undue stress on results from subgroup analysis 64% Inappropriate subgroup analyses 64% Selective reporting of (i) subgroups (ii) outcomes (iii) time points 60% Selective reporting of positive results/omission of adverse events data 60% Failure to report results or long delay in reporting 59% Post-hoc analysis not admitted 56% Giving incomplete information about analyses with non significant results 54% Analysis conducted by the sponsor of the trial

15. How do researchers behave? • Lesson #3. Significant gap between ideal (high standards) and actual standards for integrity in research Misconduct ~ 0.1% <–> 1% QRP ~ 10% <–> 50% Research behavior High standards ??

16. How should researchers behave? Integrity-centered Misconduct-centered

17. Slow change to an integrity-centered universe • More emphasis on prevention and improving integrity • First major change ~ improve education/training • US, efforts to require training/education • 1989 Institute of Medicine Report called for training • 1990/92, National Institutes of Health required for trainees • 1997, National Science Foundation required for trainees • 2000, Public Health Service proposed general requirement • Strong objections raised by research community • Requirement has been suspended • Global initiatives • Finland, national requirement for graduate students • Elsewhere, growing number of courses and resources

18. Training covers more than misconduct Nine Core Areas 1. Data acquisition, management, sharing, and ownership 2. Mentor/trainee responsibilities 3. Publication practices and responsible authorship 4. Peer review 5. Collaborative science 6. Human subjects 7. Research involving animals 8. Research misconduct 9. Conflict of interest and commitment • Areas developed over time: • 1994 Training Grant Requirement • Conflict of Interest • Responsible authorship • Policies for handling misconduct • Data management • Human & animal subjects • 2000 HHS RCR requirement • 9 areas • Other areas could be added: • Grant and financial management • Workplace rules • Laboratory safety & special rules (biological materials…)

19. Abundant resources for teaching Textbooks Web pages Train-the-trainer programs No standards for content or approach Coverage depends on the instructor Training not integrated with research Minimal testing or follow-up Little assessment of effectiveness Major challenge: Research is global; research teams and laboratories are international RCR training is local, inconsistent, and for the most part inadequate Current status of “RCR” training

20. Second change ~ Better understanding of behavior • Misconduct-based universe ~ cause of misconduct? • Too few cases/too much variation to draw “scientific” conclusions • Suggested some areas for further attention: • Quality of mentoring, supervision, peer review • Integrity-centered universe ~ many issues to study • Social processes • Authorship - who is listed and why? • Peer review - weaknesses and how to correct? • Data -how do researchers collect and record data? • Institutional role and influences • Policies - how institutions develop and promote policies • Good management - how administrators and committees work • Conflict of interest - how institutions manage their own conflicts • Much better prepared to implement effective policies

21. Four crucial challenges  Establish clear objectives • “Misconduct” policies can have many objectives: • Establish procedures for responding to misconduct in research • Detect and eliminate/correct fraudulent information • Protect public from consequences of flawed research findings • Maintain/restore public confidence in research • US Policy (OSTP, 2000) has three general objectives • Protect “reliability of the research record” • Maintain public “confidence in the research record” • Achieve policy unity • Reason for objectives • Clear understand of goals to be achieved • Standard for measuring success and improving policy

22.  Fair, effective procedures for responding • Establishing national policies is a starting point • Comprehensive definitions that cover all serious misbehavior • Assure reporting and accountability • Protect informants and insulate from bias and conflicts of interest • Global harmonization and communication are next step • Research is no longer local or national • Laws of nature and scientific methods are not local • Standards for reporting, investigating, and judging misconduct should be global • Some accommodation for differences in law and government as long as good research practices are not compromised • The globalization of research requires the globalization of policies and best research practices

23.  Take steps to make policies more effective • Detect and eliminate/correct fraudulent information?? • Investigating 1 of every 100 cases has little impact on the reliability of the research record • Some FFP has is trivial • Current policies have minimum impact on the research record • Steps that would eliminate/correct current system • Improved education, emphasizing professional responsibility • Clearer rules for data management, mentoring, and peer review • Random audits of publications and supporting data • Institutional climate surveys to assess reliability of self-regulation • Extend responsibility for reporting to journals • Investigating major cases is essential but has little impact on the overall reliability of the research record

24.  Take a serious look at QRPs • Viewed from a public perspective, questionable practices are more significant than research misconduct • Occur more frequently ~ 10x or more • Have serious impacts • Poor literature reviews have led to harm of research subjects • Bias reporting/duplicate publication impact health-care decisions • Ineffective public decisions due to improper statistical analysis • How would you explain the following to the public? • 25% of researchers reported recording results in loose-leave notebooks • 40% of abstracts misrepresent findings reported in the article • Funding makes researcher 3-5 times more likely to report results favorable to the funding source • Protecting the integrity of the research record requires more than simply responding to cases of misconduct

25. ORI.HHS.GOV for more information nsteneck@umich.edu