Peer Review: Challenges Raised by OMB’s Draft Guidelines

1. Peer Review:Challenges Raised by OMB’s Draft Guidelines Leslie Hushka, Ph.D., DABT September 30, 2003

2. ExxonMobil Biomedical Sciences, Inc.

3. What Guideline Elements Are Addressed • Bulletin Goals • Information Covered / Not Covered • Selecting Peer Reviewers • Peer Review Process • Information Access and Public Comments • Updating Agency Guidelines • Summary

4. Dr. Richard Belzer, President Regulatory Checkbook Scholarly peer review  Government peer review (including ownership, objectives, selection, procedures, interests, accountability, etc) No evidence that peer review is an appropriate remedy for the "problem" government peer review is supposed to solve Too much attention is devoted to conflicts of interest and not enough to coincidence of interest Dr. Steven Lewis, Distinguished Scientific Associate, EMBSI Peer Review  Stakeholder Dialogue Conflict of Interest  Bias Peer Review  Sound science What is Not Covered in This Section SRA Forum on Peer Review - May 29 2002

5. Bulletin Goals ... Ensure that agencies conduct peer reviews of the most important scientific & technical information ... Peer reviews are reliable, independent and transparent Questions: • Are these the proper goals? Are these aligned with info quality? • What is reliable? • Independent? From Agency only? Transparent how & when? • What criteria will be used to judge success in meeting goals?

6. Information that is Covered / Not Covered by Guidelines ...Regulatory information means any scientific or technical study that is relevant to regulatory policy . . . used by regulatory bodies ... Peer review undertaken by a scientific journal may begenerally be presumed to be adequate Questions: • Does journal peer review meet the amended PR - IQG standards? • What if the Agency re-analyzes, summarizes, interpretes original findings? • Does establishing a rebuttable presumption for published articles free agencies from requirement to substantiate the quality of information?

7. Selecting Peer Reviewers ... Selected on the basis of necessary scientific and technical expertise ... Broad a range of expertise as is necessary ... Do not possess real or perceived conflicts on interest Questions: • Will this create additional burden or discourage qualified scientists? • Is a broad range of expertise really necessary? Does it dilute skills & decrease rigor of review? • Conflicts of interest and bias - Does the 'appearance' of either of these affect selection of reviewers? • How will the agency measure 'perceived' conflicts of interest?

8. Peer Review Process ...Select peer review mechanism based on the novelty and complexity of the science to be reviewed, the benefits and cost implications, and any ’controversy’ regarding the science Questions: • Are 'controversies' on the science a valid criteria for selecting a peer review method? Is ‘uncertainty’ a more appropriate term? • Agency must provide reviewers with information on those potential sources of controversy . . .What and how? • If Agency uses other firm(s) to conduct review, does it establish some dependence upon the Agency?

9. Information Access and Public Comments ...Provide an opportunity for other interested agencies and persons to submit comments ... provided to peer reviewers ....Disclose names, qualifications of peer reviewers ...Include a certification explaining how agency has complied Questions: • How will the Agency address public & peer review comments on information products that have not been 'disseminated'? • How will agencies set up an administrative records for information products that are years away from use in rulemakings? • What criteria will OMB use to review ‘certifications’ for PR/IQG? Should these criteria be the same to evaluate if a review is successful? Can criteria be outcome-based and not output-focused?

10. Updated Agency Guidelines ...Supplement or amend IQGs to incorporate requirements ...Guidelines for... entanglements that preclude an individuals ... Confidentiality in peer review Questions: • Performance standards or detailed processes? Is this appropriate? How will performance be measured? • Should separate agency guidelines / standards be developed by each agency? • Any criteria for excluding peer reviewers? • What, if anything, is confidential in the peer review process?

11. Data “Quality” Contains Multiple Elements Objectivity • Present info in accurate, clear, complete and unbiased manner • Identify sources, supporting data and models • Ensure transparency, reproducibility, and use “best available science” Utility • Agency must consider how the public uses the information and ensure appropriate transparency for those uses Integrity • Ensure that information is not compromised through corruption or falsification

12. Summary Many questions for agencies to address in developing or modifying peer review programs Overriding questions: • Will peer review improve the quality of information disseminated? • If so, will peer review also improve the underlying quality of science used for regulatory purposes? • If not, what will? • Are there other models that will better fit these needs?