Mike Strauss Office of Scientific Quality Review

1. Surviving the Path to Peer Review Success Mike Strauss Office of Scientific Quality Review

2. Why OSQR Review? It’s not our fault! 1998 Farm Bill • ARS research peer-reviewed every 5 years • Most review panelists external to ARS • Satisfactory review before beginning research

3. 1.Stakeholder Workshop Congressional Mandate 2. National Program developed. Input Input 8. Retrospective evaluation. You are here 3. Program Priorities set (PDRAM). Assess Plan 4. Project Plan prepared. 7. Annual progress reviews NPL Validation Implement 5. OSQR Review and Certification. 6. Research initiated

4. NPL Validation • The objectives in this PrePlan are those provided in the PDRAM or subsequently approved by the Office of National Programs and the approaches are suitable for achieving the objectives • The objectives in this PrePlan are those provided in the PDRAM or subsequently approved by the Office of National Programs and the approaches are suitable for achieving the objectives Some advice Be early Be proactive Communicate! A Warning OSQR Due Dates are FIRM

5. What is OSQR Review? A Dialogue And an… External Review Scientific Review Prospective Review Peer Review Competitive Review

6. Who are the Reviewers? Panelists are your colleagues. They read your peer-reviewed papers. Panelists are active scientists. Most are academics (per the Farm Bill). Panelist often know your work. And are often familiar with your excellence. Panelists take their task very seriously. The devote many hours to each review. They don’t want to give low scores!

7. Who are the Reviewers?

8. How is a Panel Selected? OSQR Receives suggestions/nominations from NPS, Areas, others. All potential chairs are screened for COIs. Coordinator Interviews potential candidates and makes a recommendation to SQRO. SQRO approves Chairs. Chairs work with Coordinator to develop a balanced, proposed list of panel members. SQRO reviews and approves final list of panelists.

9. What do Reviewers Want? A coherent story… What are the problems you are researching? Why are they important? What do you hypothesize? (or why a hypothesis is not appropriate) How will you test your hypothesis? (or accumulate and manage data) Why are you the best group to do this? (DON’T say this, demonstrate it)

10. What is a Research Plan? Tells the “story” of your research An integrated and logical presentation A clearly presented argument A “Marketing” brochure A

11. Project Review Criteria Adequacy of Approachand Procedures Probability ofSuccessfullyAccomplishing the Project’s Objectives Merit and Significance

12. Review Products • Action Class Score • Consensus review comments

13. Reviewers NEED to know… What is the problem? Why is it important? Where are you going with it? How are you going to get there? And how will you know you have arrived? Don’t make them hunt for this!

14. Like a Manuscript Review - “Editor” = Scientific Quality Review Officer (SQRO) (Currently Thomas (Ed) Cleveland, SRRC - Three outcomes 1. Publish as presented (no revision) 2. Publish after revision as monitored by the “editor” (SQRO). Reviewers clear on what researchers are planning (minor gaps in info). (minor, moderate revision) 3. Publish after revision and reexamination by both reviewers and SQRO. Reviewers not at all clear about what researchers are planning (major gaps in info). (major revision, not feasible)

15. Remember You are composing a narrative… …not completing a form! Your plan is not reviewed by bureaucrats. Reviewers are active scientists in your field. They often know you and your work. Reviewers devote, on average, 6 to 8 hours reviewing each plan BEFORE the panel meets.

16. Document Outline Title and Investigators..………….page 1 Signature Page……………...........page 2 Table of Contents……….………….page 3 Project summary (250 words)...page 4 Objectives...…………..................page 5 Need for research (1-2 p) Scientific Background (5-7 p) Approach & Procedures (6-15 p) Prior Accomplishments (2 p) Literature Cited Milestone Table (1-3 p) Past Accomplishments of Project Team Members Issues of Concern statements Appendices (letters plus other material) 15 - 30 pages These are not boxes…they are guides to your narrative flow.

17. Your Plan is a Marketing Tool…and the reviewers are your customers By page 7 reviewers should know: The subjectof your research Why it is important What it will produce How you are going to get there The rest of the document will “flesh out” this but if the reviewers don’t know it by page 7, they probably won’t get it easily from the rest of the document.

18. Title and Investigators..………….page 1 Signature Page……………...........page 2 Table of Contents……….………….page 3 Project summary (250 words)...page 4 Objectives...…………..................page 5 Need for research (1-2 p) Scientific Background (5-7 p) Approach & Procedures (6-15 p) Prior Accomplishments (2 p) Literature Cited Milestone Table (1-3 p) Past Accomplishments of Project Team Members Issues of Concern statements Appendices (letters plus other material)

19. Project Plan Components Project Summary – 250 words The Appetizer! Write this in active voice. State the essential problem and why it is important. What have you done to date (1-2 sentences)? How will you address the issue? Why is this important? This is where you capture the interest of the reviewer. Make it compelling!

20. Project Plan Components NEED FOR RESEARCH: 1-2 pages Where are you going? Express need scientifically AND in the context of NP Action Plan. Be concise in statement of research purpose. Discuss potential benefits and anticipated products. Identify relevant customers and stakeholders. Briefly note the principal methods you will utilize (e.g., …using microarray technologies we will elucidate…” Build upon, don’t repeat, the overview!

21. Project Plan Components Objectives: 1-2 pages How does all this fit? Why are all these pieces here? How do they relate? Are there closely allied plans that bear on this work? A figure can help! Should reflect your plan and be a guide. - Objectives and sub-objectives - Personnel - Outcomes - Related projects Your plan can include 4 total pages of figures. Use them!

22. Project Plan Components Scientific Background: 5-7 pages Why are you going there? Highlight knowledge gaps. Literature demonstrates understanding and gaps. Not an exhaustive. Show a rationale for the objectives. How will this fill knowledge gaps? Limit to 1/3 of project plan length Note similar projects within and outside ARS and how your past work prepares for or leads to this (provide details in the Prior Accomplishments section…but say enough to convince reviewers you know the area). Cite preliminary data from your projects, if available

23. Project Plan Components Prior Accomplishments: 1-2 pages What have you done before? Highlighted briefly in the Background. Name prior project terminated within two years Major objectives and accomplishments Prior project investigators Impact of prior work (science, technology, users) Pertinent publications A table or chart of past data can be very helpful.

24. Project Plan Components Approaches & Procedures: 6-12 pages How are you going to get there? Set out your Experimental design. Formulate REAL, testable, hypotheses! Describe approaches and methods any why they are appropriate. Discuss advantages and limitations (important if methods are “risky”). Illustrate how objectives can be achieved. Who will do what, how, and when (including collaborators and SCAs!) Describe nature and extent of collaborations, including SCAs Letters in Appendix need to confirm what you say! For SCAs, a copy of the agreement is sufficient. Include management, evaluation and contingencies. What is your path to success? How will you monitor it?

25. Project Plan Components MILESTONES AND OUTCOMES Summarizes the project Dynamic over the project lifecycle

26. Readability and narrative flow • Connection between parts (a diagram) • Roles for all members of the team • Appropriate expertise on team or from collaborators • Grammar/spelling/proofreading • Clear Hypotheses • Appropriate detail in Approach • Clear, proper, milestones • Real contingencies • Does the plan instill confidence in this team’s abilities? Project Plan Checklist OSQR

27. Real Hypotheses—Are they testable? Hypotheses are not REQUIRED but don’t use a general goal where a hypothesis is better! Lack of connection--How/why do the parts of your plan relate? Or if part does not, why is it there? Uneven presentation—If different people wrote different parts, it will show! Context of plan—How does this fit with other similar work within and outside ARS? Statisticallysound—Are replicates sufficient? How will you analyze…don’t just throw out jargon (“Data will be analyzed using ANOVA.”)…are replicates sufficient? When you’re done, will you be able to know if you arrived? “But we’ve always done it this way” is not sufficient. Some Recent (frequent) Criticisms

28. How will it get done?—Who does what? What other resources are there? (postdocs, technicians, students…include in human and physical resources) Vagueness that prevents real analysis—Ifthe information is confidential say why you can’t tell them but say enough to allow some level of analysis. Risk without justification--Risk can be good but ONLY if it’s apparent you are aware of the challenge and have justified it. Data accumulation without analysis—It’s not enough to gather data, what will you do with it? Very large plans—Need to provide both a “30k ft” overview AND “on the ground” description. Where the overview involves several projects, consider a common description for all in the appendix and brief summary in the plan. …More frequent Criticisms

29. To keep in mind… The reviewers need to see the logical “thread” through your work. Don’t make readers “search” for what you are doing! Be clear, accurate, and correct. Don’t assume reviewers know you and your work…(a poor plan may not be saved even if they do!) A NONTECHNICAL reader should be able to understand the general goals of your research. A TECHNICAL reader should be confident you understand the problem and can accomplish the work.

30. Some hints to success… Proofread Your Plan Ask a nontechnical person to read your plan Ask someone who hasn’t seen it to read and proofread your plan Ask a highly critical colleague to read it thoroughly. Are collaborations documented appropriately? Check hypotheses… Treat this the same care you would a competitive proposal. The reviewers will!

31. What if I have several SYs? Don’t follow this recipe for failure… • Ask for a portion from each • Assemble those in order • Consider the plan done… Write a flowing, logical, and consistent narrative. Settle scientific differences…then write! You may not like a colleague… (don’t let the panel know that!) You may not like some of the work… (don’t let the panel see that)

32. Hypotheses • Most plans should have hypotheses. • Hypotheses must be falsifiable and testable. • Hypotheses are not restatements of objectives. • A hypothesis may not be appropriate for work like breeding or germplasm characterization…but explain that! • Ask your Area statistician to review your hypotheses

33. Hypothesis Problems • Hypotheses that are too complex, i.e., these are statements with “and” and “or” that essentially make the hypothesis a compound hypothesis, rendering it very difficult if not impossible to really test and reject because part might be rejected and part might not. • Wiggle words. A hypothesis with “may” or “might” or “could” cannot be rejected; it’s true no matter what result you get. • Misdirectedhypotheses about the researchers themselves. These say things like “Discovering the mechanism behind X will enable us to…….” This tests the abilities of the researchers to take information and do something with it. Instead, the hypothesis should focus on the experimental system itself.

34. Hypotheses that are statements of the obvious, or are scientifically trivial. “Disease results from expression of genes for virulence in the pathogen and genes for susceptibility in the host.” • Too global. “Quantifying X will provide significant increases in income for the industry.” Can any 5-year project plan in ARS really test this?

35. Sometimes a GOAL is preferable to an Hypothesis. We recommend: Nabel, Gary J., 2009 The Coordinates of Truth Science, 326 (5949): 53-54

36. Hypothesis Test(Courtesy, Don Knowles) Hypothesis: Babesia equi requires a specific set of genes for tick infection and transmission. What’s wrong with this? -vague (which genes?) -compound (infection & transmission

37. What Happens After Review? • No, Minor or Moderate Revision • Lead Scientist responds to comments. Scientific Quality Review Officer certifies compliance with recommendations. • Major Revision or Not Feasible • Lead Scientist revises and responds to comments. • Panel performs a second review assessing response to their comments and assigns a new Action Class Score. If still Major or Not Feasible, project is returned for administrative action. No further review. • Projects are reviewed no more than two times • (There are no page limits for revised plans)

38. What Happens After Review? The Impact of Vertically Striped Voles (VSV) on Wheat, Rye, and Egg Production R. U. Kidding 1321-38000-123-00D 1/5/2006 Frontiers of Vole Biology and Relativity Theory

39. Can I disagree with the panel? This is a dialogue If you really disagree…put it away for a few days! Then… Honestly consider panel opinions. Be polite but if you disagree say why DON’T skip changes to plan DON’T insult or impugn panelists DO provide justification for your alternative view Panels are NOT perfect…they are fellow scientists

40. How not to disagree Q: The panel does not see any [expertise] in this plan. A: “I disagree.” [no explanation] Q: Can you provide some preliminary data to support this idea. A: “Yes, we have preliminary data but can’t/won’t show it to you.” Q: The panel suggests you try this approach. A: But that’s just too difficult. A: We’re not allowed to alter this project in any way. [not true!] A: We’ve done it our way for [x] years and see no reason to change. Q: The panel suggests you alter the direction/wording of this work. A: This work is “set in stone” and we are not permitted to change it. Q: This is not a hypothesis. Fix it or change to a goal statement. A: I looked at Tom’s plan and Bill’s and their panel didn’t make them do this so I don’t think I should have to do it. Q: Did you do a power analysis? A: No we did not, but we’ve always done it this way before. A. Yes. It said we needed more so we ignored it.

41. Advice from a first time project leader • Lead SY and RL spell out process in detail • Train all in MS-Word, including Compare and Combine • Ideal if all have same reference management program • Assign one person to do all references to make consistent • START EARLY!!! • Merging several sets of edits can be disastrous. Manage review in an orderly fashion. • Start with the PDRAM and add subobjectives, then hypotheses or goals (goals are allowed, but they should be clear and not unjustified “fishing expeditions). • Consult statistician early and then just before internal review. • Get everyone’s input.

42. Last Words Proofread Seek Review then proofread and seek more review And lastly Proofread and Seek Review However…

43. …Correct Grammar and Spelling are Important -- but not enough `Twasbrillig, and the slithytoves  Did gyre and gimble in the wabe;All mimsy were the borogoves,  And the momerathsoutgrabe. Lewis Carroll Or as Indiana Jones learned, The path to success is really clearer than it may initially appear…