Peer Review of ARS Research Projects

1. Peer Reviewof ARS Research Projects A Brief Overview of the Process Office of Scientific Quality Review Steven Huber, Scientific Quality Review Officer

2. OSQR Web Sitehttp://www.osqr.ars.usda.gov • National Programs • Schedules of Reviews • Forms and Resources (TIPS!) • Reviewer Information • Peer Review status • Manual (online comments soon) • Staff

3. Topics to Cover • Background • Role of the Research Leader • The Prospectus & Project Plan • General writing tips • The panel process • Outcomes

4. New Mandate for this process • The Agricultural Research, Extension and Education Reform Act of 1998 (public law 105-185): • ARS projects be evaluated by panels composed predominantly of non-ARS scientists • With oversight from the REE Advisory Committee

5. Intent of Peer Review • To obtain constructive, independent, expert feedback and advice on ways to improve the scientific and technical merit of each project plan • NOT funding issues

6. Fundamental Questions • Panelists are asked to evaluate: • MERITANDSIGNIFICANCE (Relevance to NP) • APPROACHESANDPROCEDURES • LIKELIHOOD OF SUCCESS (feasibility) • Suggestions for improvements or solutions to problems are solicited

7. Steps in the Process 16 wk (24 wk) RL and Scientists develop Prospectus NPS RLs Prepare Project Plan with LSs Submit Plan for approval AD 18 wk (14 wk) OSQR 6 wk Peer Review Response and Revision Submit revised project plan 6 wk AD/NPS 6 wk OSQR Re-review?

8. Scientist’s Responsibilities • Communicate with the NP Team. • Scope of the project. • Coordination with other projects. • QUALITY of the Project Plan. • Many are ‘poorly’ written.

9. Project Plan Template • Title and Investigator(s) • TOC (coming soon!) • Project summary • Objectives (< ½ p) • Need for research (1p) • Scientific Background (5-6 p) • Approach & Procedures (6-12 p) • Milestones & Outcomes ( 1 p) • Literature Cited • Past Accomplishments • Issues of Concern statements • Appendices 15 - 25 pages

10. Scientific Background • Focus on the pertinent literature • Show linkage/coordination with other similar projects in the NP (or other NPs) and with minority-coded projects in the MU.

11. GENERAL TIP • Use illustrations (figures, schemes, etc) to enhance the Plan. (Up to 2 pages does not count against page limit). • Preliminary data • How your project fits with others • Working models • Experimental design/treatments

12. Approach and Procedures—Tips • Lack of necessary detail is the most common criticism. • Tell what will be done, by whom, and what will result. Identify staff involved with key portions of the project. • If an objective is supported by a grant, mention that.

13. The Panel Panelists Primary reviewers Secondary reviewers OSQR Officer Provides orientation Observes Debriefs Receives results Panel Chair Ad-hoc reviewers may be used

14. Panel Chair Selection SUGGESTIONS for Non-ARS Panel Chair from NPS, ADs, etc. OSQR Officer Selects Chair OSQR Officer Consults inside and outside ARS

15. Panelist Selection SUGGESTIONS for Panelists from NPS, ADs, RLs, Lead Scientists, University Scientists, Customers, etc. OSQR Officer Approves Panel* Panel Chair (PC) Makes recommendations to OSQR Officer *Based on PC recommendations and diversity requirements

16. Suggested Panelists Use this form!

17. The Panel Meeting Process Panel Discussion Primary reviewer presents the plan input from secondary reviewer, panelists, and ad hoc reviewer(s).  1 hour Turn in to SQR Officer Panelists and Panel Chair assign an action class to the project. Turn in to SQR Officer Primary reviewer modifies the reviewrecommendation on disk Panel Chair Approval On breaks, evenings

18. Action Classes* • Each panelist individually provides overall ‘Action Class’ assignment: • No revision required. Minor revision might be made. • Minor revision required. Objectives fit the NP action plan; approaches to all objectives are sound. Project is feasible. • Moderate revision required. Moderate revision of an objective and/or one approach needed. Project is feasible. • Major revision required. Should be sound and feasible after major revision. (May be re-reviewed). • Not feasible. Deficiencies in expertise or facilities or major flaw. *Action Class Matrix will be provided

19. After the Panel Meeting • If the composite Action Class is: • No revision required • Minor revision required • Moderate revision required • Major revision required • Not feasible Scientists solicit comments and revise the Plan within 6 weeks Revised Plans will be revised AND re-reviewed Revise for immediate re-review; delay revision until major resource changes (e.g., personnel) are made; or terminate

20. Responding to the Review Recommendations Project Title:_____________________________________________________________ CRIS #_______________________ National Program__________________________ Lead Scientist:_________________________________ Reviewer #_________________ PEER REVIEW OF ARS RESEARCH PROJECT 2. Adequacy of Approach and Procedures: Please comment on the following: Are the hypotheses and/or plan of work well conceived? Are the experiments, analytical methods, and approaches and procedures appropriate and sufficient to accomplish the objectives? How could the approach or research procedures be improved? Please list and number each significant recommendation being made. Be sure to briefly state the rational or basis for suggestions made or questions raised. Each recommendation can include several specific questions you believe should be addressed by the lead scientist. 1. Objective 1--The hypothesis being tested is only one of several that should be considered. For example…… ARS Response:

21. ARS Responses to Review Recommendations • ARS scientists must respond to each major recommendation. • TONE of the response: should be receptive, not defensive or condescending. • CONTENT: make all reasonable efforts to accommodate suggestions made. Lack of adoption must be justified. Should ‘stand alone.’ The completed forms go back to the panelists!

22. ‘Re-Review’ of Project Plans Purpose: 1) independent evaluation of response, and 2) maintain credibility with community.

23. NP108, NP302, NP202 Panel Statistics 7 Panel Chairs and 44 Panelists (51 total) • 1 ARS employee • 33 employed by universities • 7 employed by private industry, trades groups, or self-employed • 10 Other federal agencies (U.S. and Canada)

24. NP108, NP302, NP202 Panel results Number of Projects Action Class

25. Comments from Panelists • Think the peer review process is important for the agency • Appreciated the positive approach of the ARS peer review process. • Enjoyed the experience (opportunity for them to discuss science with peers). • Hope the review recommendations will be useful and incorporated.

26. Benefits of Peer Review • Better short- and long-term thinking and planning of research (if done properly!). • Increased communication among scientists, ADs and NPLs. • Increased collaboration (especially among ARS labs). • Increased quality—perspectives from peer reviewers. • Greater ability to compete for competitive funds.

27. OSQR Web Sitehttp://osqr.ars.usda.gov/ • Peer Review Status • Manual • Staff • National Programs • Schedules of Reviews • Forms and Resources • Reviewer Information