Real and Ideal Research Record Keeping Practices

Real and Ideal Research Record Keeping Practices Kenneth Wilson, Alan Schreier East Carolina University David Resnik NIEHS/NIH And Amanda Drozdowski East Carolina University 2009 RESEARCH CONFERENCE ON RESEARCH INTEGRITY NIAGARA FALLS, NY May 16, 2009 The ideas and opinions in this presentation do not represent the views of the NIEHS, the NIH or the US government

Research Records Definition IncludesAll Media Hypotheses Paper Protocols Electronic Original Data Photographic Processed Data Other Methods Analyses Future Plans Manuscripts/Articles

Overview of Our Project:Goals • What are the problems? • From the Research Misconduct Perspective • What are the current standards or best practices that will prevent problems? • How well do active scientists adhere to these best practices?

Overview of Our Project • Phase I – Interview Officials who conduct research misconduct investigations • Findings: • In almost half of the cases where research records were examined, there were problems with the research records. • Almost 80 percent of the officials who reported problems with research records had at least one case where the problems were so serious that some issues were never resolved. • Over 10 percent of the officials who reported problems with research records reported that the problems with the research records in their typical cases were so serious that some issues were never resolved. • “Research Records and the Resolution of Misconduct Allegations at Research Universities” by Kenneth Wilson, Alan Schreier, Angel Griffin and David Resnik Accountability in Research 2007 14:57–71

Overview of Our Project • Phase II – Conduct focus groups • Identify best practices • The best practices differentiate and provide separate standards for three different levels within the university: • the individual researcher, • the research group leader, • the department/institution. • The authors believe these best practices constitute a “snapshot” of the current normative standards for research records within the academic research community.. • “Academic Research Record-Keeping: Best Practices for Individuals, Group Leaders, and Institutions” by AA Schreier, KR Wilson, and D. Resnik Academic Medicine 2006 81:42-47.

Overview of Our Project • Phase II – Conduct focus groups • Develop phase III survey instrument • These two phases have been funded by NSF.

Overview of Our Project • Phase II – Develop phase III survey instrument • Person assigned to manage research records and data. • Process to assure the smooth transfer and retention of research information • I set standards for good record keeping. • A written document that explains how records are to be kept • New members were clearly told who owns the research records. • New members were clearly told who has access rights to the collected data . • New members were clearly told how the research records are to be recorded. • New members get models of good record keeping practices. • Workers provided with record keeping materials • A common storage area • Supervisors review the research records • Steps taken to see that record keeping plan followed. • Periodically backup data and other research records. • Copies of the supporting records were stored with the manuscript. • Documenting and sharing important basic research information • Plan to store the research records for at least 3 years. • Emails and conference calls documented and saved • Able to reconstruct what leavers had done and understand the data • Most research records would survive a disaster • These two phases have been funded by NSF.

Overview of Our Project • Phase III- Conduct a national survey of 600 federally funded researchers to document their record keeping practices (Funded by NIH --Completed results presented today) • Phase IV- Survey students/post-docs of the Phase III researchers (Funded by NIH – in the field) to assess the “intergenerational” transmission of research values and ideals • Phase V- In five years, repeat Phase III & IV to determine changes.

THE PHASE III TELEPHONE/WEB SURVEYS • Focus groups strongly recommend a combination of web surveys and telephone interviews. • From a random sample of senior faculty at universities who were awarded NIH grants we completed 688 interviews (over 50% response rate).

THE PHASE III SURVEY • Respondents were asked about • their professional characteristics • Problems they experienced finding research records in the past 3 years • Their actual research practices • Their “ideal” research practices (those they would recommend to junior colleagues just starting their own research) • Names of advanced graduate students and post-docs working with them

THE PHASE III SURVEY • Real and Ideal research practices were based on the best practices we had developed in Phase II. • Since we were interviewing researchers in many different disciplines, they were formulated at the level of the research group leader. • For example, all group leaders need to set standards but the nature of the standards they set will be different if they are doing biological research or social research. • The focus group told us that at the level of the group leader, the practices were or should be universal.

Characteristics of the Sample • Respondents were in the • Biological Sciences (59%) • Social/Behavioral Sciences (20%) • Clinical Sciences (11%) • Physical Sciences (10%) • The average length of time since their first academic appointment was 19 years.

RESULTS OF PHASE III SURVEYPROBLEMS • Think about those times in the last 3 years when you had to go to your research records to locate important pieces of information for a final report, a grant application, a manuscript for publication or for planning a new study. Please indicate if the following problems have 1) almost never; 2) rarely; 3) sometimes; 4) usually; or 5) almost always happened. • Answers recorded as these numbers (1 to 5). • Asked about 12 different problems – for example

RESULTS OF PHASE III SURVEYPROBLEMS • I found the information I needed quickly and easily. • I found the information I needed after a significant search. • I know that the information was recorded somewhere but I could not find it. • Another person had the records and it took a while to get them. • I found some of the records had been altered with no explanation.

RESULTS OF PHASE III SURVEYPROBLEMS • 84% of respondents "usually” or “almost always” found the information they were looking for quickly and easily. • Only 37 percent of our respondents reported that they “almost never” or “rarely” had to conduct a significant search to find a piece of information. • 36% of the respondents reported that someone else had the records (sometimes, usually or almost always). • 25 percent reported that they found some critical detail that had not been recorded (sometimes, usually or almost always).

RESULTS OF PHASE III SURVEYPROBLEMSAlpha = .80

RESULTS OF PHASE III SURVEYReal Practices • How often have the following happened in research that you have participated in during the last 3 years? Would you say that these 1) almost never; 2) rarely; 3) sometimes; 4) usually; or 5) almost always happened? • There was a written document that explains how records are to be kept. • Most of my research records would survive in a useable form if a disaster (like a fire, flood, or computer crash) were to strike my research space. • The people working on a research project were provided with record keeping materials (notebooks, binders, specialized software, etc.). • There was a plan to periodically backup data and other research records.

RESULTS OF PHASE III SURVEYReal Practices • Mean ratings for real practices ranged from 2.81 to 4.61. • 7 of the 19 practices had mean rating of 4.00 (usually) or higher. • Only 1 (There was a written document that explains how records are to be kept) had a mean rating less than 3.00 ( sometimes).

RESULTS OF PHASE III SURVEYReal PracticesAlpha = .90

RESULTS OF PHASE III SURVEYIdeal Practices • When answering these questions, imagine that young faculty members approach you for advice. They are new to your department and they are about to begin work on their first project where they supervise graduate students and/or staff members. Which of the following would you recommend? Please indicate if it is a practice that you would 1) almost never; 2) rarely; 3) sometimes; 4) usually; or almost always recommend.

RESULTS OF PHASE III SURVEYIdeal Practices • The mean rating for the 19 ideal practices ranged from 3.86 to 4.79 • 18 of 19 ideal practices have average ratings between "usually"(4) and "almost always"(5) • Only 1 (There was a written document that explains how records are to be kept) had a mean rating less than 4.00 (usually). • 13 of 19 have average ratings closest to "almost always" (above 4.5).

RESULTS OF PHASE III SURVEYIdeal PracticesAlpha = .91

RESULTS OF PHASE III SURVEYIdeal Practices • These results show that the best practices are widely shared among top researchers.

RESULTS OF PHASE III SURVEYReal vs. Ideal Practices • By comparing what researchers report actually happens in their research (ratings of real practices) with the standards they would recommend to new researchers (ratings of ideal standards), we can identify those standards to which researchers routinely adhere and those that are more problematic. • All 19 ideal practices were rated higher than the corresponding real practice.

RESULTS OF PHASE III SURVEYReal vs. Ideal Practices

RESULTS OF PHASE III SURVEYReal vs. Ideal Practices • The smallest differences were found for: • The people working on a research project should be provided with record keeping materials (notebooks, binders, specialized software, etc.). • There should be a common storage area (physical and/or virtual) where people can store the records they are currently using. • There should be a plan to store or archive the research records in a secure manner for at least 3 years.

RESULTS OF PHASE III SURVEYReal vs. Ideal Practices • The largest differences were found for: • There should be a written document that explains how records are to be kept. • Your standard practices should allow you to reconstruct what research workers had been doing and understand the data they collected even if they leave abruptly.

RESULTS OF PHASE III SURVEYReal vs. Ideal Practices • These results show that while best practices are widely shared among top researchers they have a difficult time living up to their ideal standards.

Other Interesting Findings • There is a significant negative correlation between Real Practices and having Problems (r = -.302***). • The correlations between Ideal Practices and having Problems is NOT statistically significant. • There is a significant correlation between the Difference between researchers’ Real and Ideal practices and their Problems (r= 3.51***). This remains significant when the level of Real Practices are controlled.

Other Interesting Findings • There is no significant relationship between the year of the researcher’s first faculty appointment and: • Their real record keeping practices • Their ideal record keeping practices • The difference between their real and ideal record keeping practices • The problems caused by their record keeping practices

Other Interesting Findings • While all disciplines have similar standards, there are significant differences between the primary academic disciplines and: • Their real record keeping practices • Their ideal record keeping practices • The difference between their real and ideal record keeping practices • The problems caused by their record keeping practices

Other Interesting Findings

Conclusions • The Best Practices that we identified during our Phase II focus groups are widely shared by federally funded researchers . • “Academic Research Record-Keeping: Best Practices for Individuals, Group Leaders, and Institutions” by AA Schreier, KR Wilson, and D. Resnik Academic Medicine 2006 81:42-47. • While most elite researchers share an understanding of the best practices, they are not able to live up to their own ideal standards. • Lower levels of Real Practices and greater deviations from Ideal Practices create additional work for many researchers.

Conclusions • Lower levels of Real Practices and greater differences between a researcher’s real and ideal practices created time consuming problems for researchers. • While all disciplines have similar standards, the standards in the physical sciences are lowest and the standards in the clinical sciences are highest. • The physical sciences report the most problems created by research record keeping while the social/behavioral sciences report the fewest. • Length of time that researchers have been practicing does not influence their real or ideal practices or the problems they encounter.

We would like to thank the National Science Foundationand the National Institute of Health for funding this research For more information please contact • Ken Wilson PhD • Department of Sociology • East Carolina University • Greenville, NC 27858 • wilsonk@ecu.edu For a copy of the Best Practices , contact • Al Schreier PhD • Research and Graduate Studies • East Carolina University • Greenville, NC 27858 • schreiera@ecu.edu

Real and Ideal Research Record Keeping Practices