1 / 40

Revising and Resubmitting: Practical Considerations Based on the Psychology of Re-Reviews

Revising and Resubmitting: Practical Considerations Based on the Psychology of Re-Reviews. Marc I. Rosen, M.D. Revising and Resubmitting: Lecture Structure. Deciding W hether to R evise and R esubmit Suggestions for R evising and R esubmitting Example Moral. Review Group Actions.

willow
Download Presentation

Revising and Resubmitting: Practical Considerations Based on the Psychology of Re-Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Revising and Resubmitting: Practical Considerations Based on the Psychology of Re-Reviews Marc I. Rosen, M.D.

  2. Revising and Resubmitting: Lecture Structure • Deciding Whether to Revise and Resubmit • Suggestions for Revising and Resubmitting • Example • Moral

  3. Review Group Actions • Discussed and Scored with Impact Rating (score and percentile) • Triaged/Not Discussed with no Impact Rating but criteria scores (lower half) • Deferral; Not Recommended; Abstention

  4. Reading the Critiques • Read critiques carefully and calmly • Even if you are angry • Assume you got a good-faith, intelligent review • Hard to do if you are angry • Let colleagues and mentors read the reviews for reality testing, support, and input

  5. Deciding Whether to Revise and Resubmit: Get More Information • Contact Program Officer • -Ask about study section discussion • -Ask about NIH Institute • interest in area? • advice? • Talk through reviews with co-investigators and peers

  6. Deciding Whether to Resubmit: Keep Perspective • Reviewers assess your submitted material • Reviewers are never totally wrong or right • Extremely competitive process: • Resubmission is common • Avoid WYSIATI (what you see is all there is)---other talented people out there

  7. Deciding Whether to Revise and Resubmit: Prospect Theory • Sunken Cost Fallacy • Staying to the end of a boring movie hoping to recoup loss of spent money • Using a fitness plan even when it’s painful • It’s a fallacy • Loss aversion: It’s not a rejection if you don’t give up • Kahneman and Tversky • Thinking Fast and Slow (Daniel Kahneman)

  8. Deciding Whether to Revise and Resubmit: Psychology • Overconfidence: Excessive Optimism • Only 5% of U. Chicago MBA students predict they are in bottom 50%; most predict second decile • 90% of drivers think they are above average • Entrepreneurs say success rate for new business is 50% but predict personal success rate of 100% • Few newlyweds expect to be among 50% who eventually divorce Nudge, Thaler and Sunstein

  9. Deciding Whether to Revise and Resubmit: Overconfidence? • Test of Overconfidence: 90% Confidence Interval for • Weight of earth in tons • 6.0 X 1024 • Percentage of world’s population who are Native English speakers (per CIA World Factbook 2009) • 4.83%

  10. Reviews that Should NOT Make You Overconfident • “This grant addresses an important topic” • “Yale has superb facilities for this research” • “The investigator is qualified” • Only the first reviewer was critical of the application and the grant was un-scored • Mild praise and the grant was un-scored

  11. Meta-Critiques that May Not be Answerable • ”There are already a lot of grants in this area” • “Not innovative” • “Not significant” • “Not exportable”

  12. Deciding Whether to Revise and Resubmit • Do you have something better to work on for two-plus months?

  13. Revising and Resubmitting: Lecture Structure • Deciding Whether to Revise and Resubmit • Suggestions for Revising and Resubmitting • Examples • Moral

  14. Consider the Person Behind the Review • Look into research interests of people on the committee • CRISP search of what committee has funded in the past • Talk to people who have been on the committee • Talk to your project officer

  15. Consider the Person Behind the Review • Reviewers want to avoid cognitive dissonance • Cognitive dissonance • Inner drive to hold our attitudes and beliefs in harmony • Drive to avoid dissonance between them • Examples • The Fox and the Sour Grapes • The review group that found fault with your grant • So, don’t say the reviewer was wrong

  16. Consider the Reviewer’s Perspective The reviewer who likes your application has to justify your response to a committee of 12+ very smart people • Make your response easy-to-follow • Use tables for complicated concepts (the reviewer can say to the committee—“He’s got a table laying that out.”)

  17. Consider the Reviewer’s Perspective • If the reviewer likes the application, he/she is more likely to agree with your justifications • How juries decide: • They do not weigh the evidence • They do arrive at a narrative that appears to fit the data • Answer meta-critiques

  18. Revising and Resubmitting:Content Issues Address any grant-killing meta-questions early in your response emphatically and clearly

  19. Examples of Meta-Critiques Critique: “They’ll never be able to pull this off – the project is not feasible.” Answers: -Pilot data -Bring in collaborators who can pull it off

  20. Examples of Meta-Critiques Critique: “This was written by a slob who just does not know this topic well.” Answers: -Emphasize how much the application has been cleaned up -Consider adding expert who would have caught all your mistakes the first time.

  21. Examples of Meta-Critiques Critique: “This was written by Kathy Carroll’s (Stephanie O’Malley’s, RajitaSinha’s, Marc Potenza’s…) go-fer and is not really an independent application.” Answer: -Spell out what is yours and what is not

  22. Examples of Meta-Critiques Critique: “[zzzz’s] always make a hash out of [yyyy] research– it requires someone in my field of specialization. “ “The application would be strengthened by the involvement of a biostatistician.” Answer: -Include someone with the recommended expertise

  23. Examples of Meta-Critiques Critique: ”The study design is from hunger.” Answer: -Table and/or figure justifying and explaining the study design

  24. Revising and Resubmitting:General Content Issues • Write introduction explaining changes • Address all criticisms thoroughly • Update preliminary studies • Indicate method of highlighting changes (e.g., bold, italics in text) • If not ready to submit at next deadline, DON’T • Reviewers generally need a reason to improve your score

  25. Revising and Resubmitting:General Content Issues • “Thank you for the careful review of our proposal to […] We appreciate the praise for […] and the careful, thoughtful critiques” • Main criticism and response • List more minor criticisms (in italics) and responses • “Thank you for reconsidering our application”

  26. Revising and Resubmitting:Process Issues • Respond constructively and positively • Accept the help of reviewer comments • Be thankful and responsive, not argumentative or arrogant • Avoid defensiveness or making reviewers defensive • No more than 1-2 areas of disagreement, but justify decision thoughtfully and respectfully • Let experienced investigator or grant reviewer read introduction and change what they tell you

  27. Revising and Resubmitting:The Best Responses • Pilot Data • Re-analysis of your own data • Literature

  28. Revising and Resubmitting:Weaker Responses • Logic • Your opinion • “In my clinical experience…”

  29. Revising and Resubmitting:Don’t, Don’t, Don’t • (Usually) don’t answer questions that were not raised • Don’t malign the review process or the reviewer • Don’t spend much effort pointing out that one reviewer liked what another reviewer critiqued • If the reviewer asks for something that was already in the application, be humble, e.g. “The information is presented more clearly this time in the methods as follows…” • Don’t get personal (no jokes, personal opinions, etc.)

  30. Revising and Resubmitting:Don’t, Don’t, Don’t • Don’t repeat every critical word from a review • Summarize criticisms (it was bad enough the first time) • Don’t over-answer minor criticisms by writing a long essay that makes the criticism seem more major than it is

  31. Revising and Resubmitting: Lecture Structure • Deciding Whether to Revise and Resubmit • Suggestions for Revising and Resubmitting • Example • Moral

  32. Example of Grant Review 4-year clinical trial to test computer-delivered counseling to improve engagement in work

  33. Grant Review • Lousy score of 270, 67th percentile • Program officer tells me they liked it, wanted to see it back • Reviewer response: • Reviewer one liked • Reviewer two mixed • Reviewer three (statistician) gave it terrible score

  34. Reviewer’s Potentially Grant-Killing Responses “However, no data exists whether veterans would actually use the intervention.” Summary Statement Recommends “Further conceptually develop and pilot test the internet-based intervention. Provide that data as a part of the proposal.”

  35. Planned Response • Agree with everything reviewers say and propose three-year, pilot-type, therapy development study to address it

  36. Revising and Resubmitting: Lecture Structure • Deciding Whether to Revise and Resubmit • Suggestions for Revising and Resubmitting • Example • Moral

  37. Morals • It helps to enjoy the process • Doing your best • Advocating for something you believe in • Promoting yourself • Your CV lists grants • No lasting harm from unfunded application

  38. Moral • “At the length, truth will out” • Shakespeare (Merchant of Venice) • “In the long run, we are all dead.” • John Maynard Keynes

  39. Thank you

More Related