The Economic Benefits of Economic Peer Review

1. The Economic Benefits of Economic Peer Review Richard A. Williams, Ph.D. Director, Division of Market Studies Office of Scientific Analysis and Support Center for Food Safety and Applied Nutrition, FDA

2. Disclaimer The views presented in this talk represents the authors views and are not an official position of the Food and Drug Administration.

3. Economic Peer Review Process Inter-Agency Economic Peer Review (IEPR) - Voluntary review process for federal economists and analysts who review RIAs and RFAs prior to OMB clearance. -The IEPR group strives to improve the quality of economic analyses in regulations and other documents. For more information, contact: Angela Ritzert, Ph.D Center for Food Safety and Applied Nutrition, FDA Angela.ritzert@cfsan.fda.gov Ph: 301-436-1763

4. Benefits Model Benefits of Peer review = Probabilities of • (Error in Analysis) • (Error supports faulty policy) • (Value of Policy) • (Error detected) • (Error corrected) • (Policy corrected)

5. Probability of an Error Committed • Ignorance • Strategic

6. Probability of an Error Committed - Ignorance Poorly trained economists, non-economists doing analysis or poor supervision of analysis. Theory Lack of funding, lack of knowledge about existence of data Data

7. Probability of an Error Committed: Ignorance Very small benefits likely unless: Theoretical errors - infrequent Data – peer reviewers know Better fixes Better trained economists, supervisors Invest more in data

8. Probability of an Error Committed: Strategic Errors Data – May leave out data Omission Commission Data – May use poor data or ignore good data Theory – May use poor theory, assumptions, options

9. Benefit-Cost Analysis and Risk Assessment • Both organize facts in a useful way • Neither science, both inform decision-making • Both are analysis • Both should be separate from risk management

10. Probability Error Supports Faulty Policy If change in error would not affect policy – peer review has no benefits Otherwise – size of potential change in policy has direct bearing on benefits

11. Value of Policy Higher valued policies affected by error will generate greater benefits.

12. Probability that an error is detected Data – Unlikely the peer reviewers will detect data error unless specialists in field Theory – Extremely likely to detect errors with poor theory

13. Probability that an Error is Corrected Probability should be higher when process is public.

14. Probability that Policy is Corrected Probability will be higher when the peer review process is transparent and the policy was based on faulty analysis.

15. Reassurance Benefits Type 1 error – If the null hypothesis is that the analysis is correct, poor reviewers may conclude it is incorrect and pressure may force agency to change it. Type 2 error – Poor reviewers may fail to pick up data or theoretical errors and falsely conclude analysis is correct.

16. Summary • Benefits of peer review are likely to be highwhen: • Ignorance of theory not known by analyst. • Strategic manipulation of theory; close relation of error to policy; value of policy high; process is transparent to public, peer reviewers well qualified; independent of agency.

17. Summary 2 • Benefits are likely to be low when • Shop is known for poor quality analyses (better option is to invest in improvements). • Data problems unless peer reviewers chosen for knowledge of data in area • Error not related to policy, reviewers not qualified or process not open and transparent.

18. Conclusions 1. There are benefits and costs associated with peer review. 2. The benefits depend on multiple factors including agency, study and peer reviewer factors. 3. Any rules established for peer review should be evaluated by examining the benefits and costs.