SCRUTINIZING PEER-REVIEW

1. SCRUTINIZING PEER-REVIEW SUSAN HAACK NATIONAL INSTITUTE OF JUSTICE CONFERENCE ON SCIENCE, TECHNOLOGY AND LAW September 2005

2. "Peer reviewed" in the fields I know is an antonym for such expressions as "earmarked" (for pork-barrel awards made by Congress), ... "idiosyncratic," "biased," and "closed." -- PAUL R. GROSS Peer review should be regarded as an untested process with uncertain outcomes. --TOM JEFFERSON et al. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no argument too circular, no conclusion too trifling or too unjustified, and no grammar or syntax too offensive for a paper to end up in print. -- DRUMMOND RENNIE Peer review, out of control, makes science a jungle where politics rules and fraud is tolerated .– CHARLES McCUTCHEON

3. 1. Peer-Review: Its Meaning(s), Its History Review of work (or proposed work) by others in the field: --- To evaluate clinical performance --- To decide what projects are funded; a.k.a. "grant peer review," sometimes "merit review" --- To decide what papers are presented at conferences --- *** To determine what work is published in professional journals and books from academic presses (and what revisions are required before publication); a.k.a. "refereeing"

4. Continued… * Earliest examples: first volume of Royal Society of Edinburgh Medical Essays and Observations (1731); Royal Society of London, Philosophical Transactions (1752) * Not standard practice in medical or scientific journals until after World War II * Major Reason: shift in ratio of papers offered to space available -- as many more papers were offered for publication, editors were no longer scrambling to find material to fill their pages, but looking for a way to decide which to accept

5. 2. Roles of Peer-Reviewed Publication: --- In the sciences (and other disciplines): to decide what work is published in certain journals and books --- In the universities: peer-reviewed publication as part of the promotion and tenure system (& peer-reviewing as "service") --- In the publishing business: that their publications are peer-reviewed supports the market position of publishers of certain journals and books vis a vis academic libraries, and their pricing policies --- In the legal system: "peer review and publication" as one of the Daubert factors: Publication (which is but one element of peer review) is not a sine qua non of admissibility; it does not necessarily correlate with reliability ... and in some instances well-grounded but innovative theories will not have been published... . But submission to the scrutiny of the scientific community ... increases the likelihood that substantive flaws in methodology will be detected. The fact of publication in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion will be premised. -- Daubert (1993), 594

6. 3. Peer-reviewed publication as indicator of reliability: * The epistemological rationale for peer-reviewing work submitted for publication is that this procedure for rationing limited publication opportunities helps ensure (i) that the best work is published; (ii) that readers of scientific journals, etc., aren't obliged to waste time sifting through vast quantities of weak and dubious stuff. * BUT it is hard to test how well peer-review does what it is supposed to do, since there is no easy way to compare peer-reviewed publications with rejected papers -- nor a checklist of what studies are in fact reliable! * AND how good peer-review is at screening out the unreliable, the methodologically flawed, etc., presumably depends among other things on: -- How rigorous the field in question is -- The quality of the particular journal in that field -- The expertise and honesty of referees/the time they can give to the work

7. Moreover: -- Not all work in so-called "peer-reviewed" journals is peer-reviewed (some is invited) -- Sometimes peer-review is a formality: authors may be asked who they wish to have referee their paper, or well-known people may be asked to referee work in which they are criticized -- Genuinely blind peer-reviewing may be impossible, especially in narrow, specialized fields; professional rivalry may skew the process -- Reviewers may be more concerned with, or more able to judge, the readability of a paper or the interest of its topic or results than the reliability of its claims -- AND OF COURSE (romantic myths to the contrary) reviewers DO NOT routinely repeat the experiments/studies described in a paper -- Vast amounts of time and energy go into the peer-review process itself

8. 4. Limitations/Failures of Peer-review: * Lobo, Cha & Wirth study, supposedly showing that intercessory prayer by strangers influences the success of in vitro fertilization, was peer-reviewed and published in the J. of Reproductive Medicine (2001). Lobo, listed as the senior author, and a member of the editorial board of the journal, subsequently withdrew his name, admitting he didn't contribute to the study. (Wirth subsequently pleaded guilty to unrelated fraud charges) * Haller and Benowitz paper, reporting adverse cardiovascular effects associated with Ephedra (& cited in litigation), was peer-reviewed and published in the NEJM (2000). In a table of 11 cases, patient #7 is classified as having "no preexisting conditions"; but in the accompanying text, he is reported to have had "mild cardiomegaly with four-chamber dilation and coronary heart disease with narrowing of 50 to 75 percent in four vessels" * Chan et al. (JAMA, 2004): in 122 published articles, 50% of efficacy and 65% of harm outcomes were incompletely reported * Martinson, Anderson and de Vries (Nature, 2005): more than 10% of 3,247 scientists polled admitted withholding details of methodology or results in papers or proposals; more than 15% admitted dropping observations or data points; more than 27% admitted keeping inadequate records of research projects. • Ioannidis (JAMA, 2005): of 45 published studies claiming effective medical interventions, 15 were later contradicted in whole or part by other studies AND: innovative work may be turned down precisely because it is innovative: * Frederick Lanchester’s circulation theory of how wings lift (1894), Chandra Bose’s photon statistics (1924), Enrico Fermi’s theory of beta decay (1933), Herman Almquist’s discovery of vitamin K (1935), Hans Kreb’s nitric acid cycle (1937), Raymond Lindeman’s trophic-dynamic concept in ecology (1941) were all turned down at least once. (And don’t forget Mendel!)

9. 5. Peer review as a Daubert factor -- some concluding observations: * "Peer review" has been misunderstood by some experts and some courts (e.g. in Havvard one fingerprint examiner's checking another's identification is taken to constitute "peer-review" under Daubert ) * A requirement of peer-reviewed publications has sometimes been imposed where it is inappropriate to the field (e.g. expert on police training techniques in Berry; not subsequently precluded by Kumho) * Judge Pointer's National Science Panel (on silicone breast-implants) relied on unpublished letters and dissertations, as well as peer-reviewed published articles

10. Comments concerning the appropriateness: (a) of peer-reviewed publication as an indication that proffered testimony is reliable enough to be admissible * DOUBTFUL: peer-reviewed publication is at best a very weak indication of reliability, and even worse in weak fields or poor journals -- but it is unclear how reliable is reliable enough for this purpose. (b) peer-reviewed publication as an indication that testimony is credible * NO: (despite some unclarity here too about how reliable is reliable enough) peer-reviewed publication is just too weak an indication of reliability. (c) lack of peer-reviewed publication as an indication that proffered testimony is too unreliable to be admissible * DOUBTFUL: peer-reviewed publication inappropriate with respect to some kinds of expertise; even where it is appropriate, there are many questions: was it submitted and rejected, or never submitted? if rejected, on grounds of quality, or lack of interest to readers, or unsuitability to the journal, or because it is too new, or ?