Office of Extramural Research, Education and Priority Populations OEREP Grantsmanship Session

1. Office of Extramural Research, Education and Priority Populations (OEREP)Grantsmanship Session Overview: Francis Chesley, MD � OEREP Director Prepare a Grant Application: Debbie Rothstein, PhD, Senior Advisor to the Director - OEREP AHRQ Peer Review Process: Kishena Wadhwani, PhD, MPH � Div. Scientific Review Director, OEREP Human Subjects Protections and Inclusions: Patrick McNeilly, PhD � AHRQ IRB Official, OEREP

2. Preparing a Grant Application: Steps to Success Debbie Rothstein, Ph.D. Senior Advisor, Extramural Research Office of Extramural Research, Education and Priority Populations Agency for Healthcare Research and Quality September 9, 2008

3. Getting Started Internet is funding gateway AHRQ and NIH use essentially the same application processes and grant mechanisms Keep abreast of open solicitations and Agency research priorities

4. AHRQ�s Website � www.ahrq.gov Research Findings -- Web 1� dwn R. Activities� past issues, funded grants, other anouncmnts key word search minority health patient safety long term care, rural health, etc Research Findings -- Web 1� dwn R. Activities� past issues, funded grants, other anouncmnts key word search minority health patient safety long term care, rural health, etc

5. AHRQ and NIH: Grant Application Similarities Application forms: SF 424(R&R), PHS 398 Application submission dates Use of internet as the funding gateway to keep abreast of research priorities and open solicitations, staff can provide technical assistance Announcement publication: NIH Guide for Grants: http://grants1.nih.gov/grants/guide/index.html and Grants.Gov: www.http://grants.gov A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts

6. AHRQ and NIH: Grant Application Similarities (cont�d) Receipt and referral of grant applications Application review procedures Grant mechanisms � e.g., R01, R03, R13, R36, K01, K02, K08 Transition to electronic submission of applications Some common Policy Notices and Funding Opportunity Announcements A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts

7. AHRQ and NIH: Grant Application Differences Detailed budget vs. modular budget Conference grants � AHRQ has small and large opportunities AHRQ doesn�t allow multiple PI�s Budget limitations Large grants (R01, R18) - $300K vs. $500K Small grants (R03) - $100K vs. $50K Fewer grant mechanisms used at AHRQ A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts A) www.nih.gov, B) grants, C) guide-- e.g., grnt anoucmnts, othr get 398 --1) inst offcs spnsrd rsch 2) grantsInfo@nih.gov. 3) AHRQ Pubs clrnghse -- Tel: 1-800-358-9295 4) http:// grants.nih.gov/ funding/phs398.html Pg 21 & DSCUS�N 2 Folw on oth pts

8. AHRQ Grant Opportunities AHRQ announces availability of grants in a Funding Opportunity Announcement (FOA) FOA�s are published in the NIH Guide for Grants (http://grants.nih.gov/grants/guide/index.html) and posted on Grants.gov (http://www.grants.gov/ Program Announcement (PA) Describes broad research interests Open over a period of time Request for Applications (RFA) More well defined area, specific program objectives Single application receipt date and amount of funds

9. Common AHRQ Grant Opportunities R01: large research grants Feb 5, June 5, Oct 5 application receipt dates up to $300K/yr R18: research demonstration grants Jan 25, May 25, Sept 25 application receipt dates up to $300K/yr R03: small research projects Feb 16, June 16, Oct 16 application receipt dates up to $100K in total costs Large and small conference grants (R13) Large: Apr 12, Aug 12, Dec 12 receipt; up to $100K/yr in total costs Small: Feb 20, Apr 20, Jun 20, Aug 20, Oct 20, Dec 20 receipt; up to $50K

10. AHRQ Grant Opportunities: Training/Career Development Pre and Postdoctoral Training Institutional Training Programs (T32) Individual Pre-doctoral Fellowships to Promote Diversity (F31) Individual Post-doctoral Fellowships (F32) Dissertation Grants (R36) Career Development Awards Mentored Scientist Awards (K01) Independent Scientist Awards (K02) Mentored Clinical Scientist Awards (K08) Detailed information at: http://www.ahrq.gov/fund/training/trainix.htm Send questions to: training@ahrq.hhs.gov

11. AHRQ Staff Involved in Grants Process Referral Officer Review Staff � Scientific Review Officer and Grants Management Specialist Program Staff � Project Officer (PO) Grants Management Staff

12. Electronic Application and Submission Transition to Electronic Receipt began Dec, 2005 Vast majority of all competing applications to AHRQ are now submitted electronically using SF 424 (R&R) application All major mechanisms used by AHRQ have transitioned except for the training grants

13. What is Grants.gov? Federal government�s single, online portal to electronically: Find Grant Opportunities Apply for Grants A cross-agency initiative spanning 900 grant programs from the 26 grant-making agencies, and over $350 billion in annual awards.

14. Electronic Submission of Grant Applications With this new process, it�s critical to read and follow all instructions in the announcement and application guide Applicant must prepare for e-submission by completing mandatory registrations Grants.gov: registration for organization ERA Commons: registration for organization and the PI

15. SF424 (R&R) Application Form Provides a consistent electronic submission process through Grants.gov Consists of common data elements, arranged in components Not all components will be used for every Funding Opportunity Announcement (FOA) Each FOA will have the appropriate application package attached From data files, application image is generated Applicant should view and verify the application image

16. Electronic Application and Submission - General Information and Help Links SF424 (R&R) Application and Electronic Submission Information: http://grants.nih.gov/grants/funding/424/index.htm General information on Electronic Submission of Grant Applications: http://era.nih.gov/ElectronicReceipt/ Finding Help: http://era.nih.gov/ElectronicReceipt/support.htm

17. Paper No More, Use 424 (R&R) OLD NEW Process Description Applicant assembles application with authoring tools Word Processor for documents Custom presentation for structured data entry Each Service provider has a separate interface Grants.gov has PureEdge forms Word processing file transformed into PDF Systems assembles a transaction of XML structured data PDF attachments Package is transmitted to NIH and verified NIH rebuilds the image Format the XML data into either 398 or 424 Concatenates PDF without changing their content Adds header, footer page, numbers, table of contents, bookmarks Generates one single PDF �image� for both structured data and documentsProcess Description Applicant assembles application with authoring tools Word Processor for documents Custom presentation for structured data entry Each Service provider has a separate interface Grants.gov has PureEdge forms Word processing file transformed into PDF Systems assembles a transaction of XML structured data PDF attachments Package is transmitted to NIH and verified NIH rebuilds the image Format the XML data into either 398 or 424 Concatenates PDF without changing their content Adds header, footer page, numbers, table of contents, bookmarks Generates one single PDF �image� for both structured data and documents

18. What Determines Which Awards Are Made? Scientific merit Significance and originality Methods Program / Agency considerations What is uniquely AHRQ Existing research portfolio balance Anticipated IMPACT of research Availability of funds

19. Ingredients of a Successful Grant Application Good Idea Good Science Good Application Fits Agency Research Priorities If U respnd clrly & well to th seqnc of ?s in th aplct�n U will be on th way towrds dvlpng a well crafted applct�n. PRACTICE: statng spcfc obj in 1-2 sentences & how hypth (small in #) link to OBJ decrb how analys pln is designed to ans hypoth Hndout--cmprhnsve �schlrly _/ lst + rvwrs -- SEE ROSTER (+ HRDI) If U respnd clrly & well to th seqnc of ?s in th aplct�n U will be on th way towrds dvlpng a well crafted applct�n. PRACTICE: statng spcfc obj in 1-2 sentences & how hypth (small in #) link to OBJ decrb how analys pln is designed to ans hypoth Hndout--cmprhnsve �schlrly _/ lst + rvwrs -- SEE ROSTER (+ HRDI)

26. Purpose of Peer Review To evaluate the scientific and technical merit of grant applications, providing information (Recommendations) used by the Agencies/Institutes and Centers to make funding decisions.

27. Reviewers consider Scientific and technical merit of proposed research/training Qualifications of Principal Investigator(s) and research team Availability of resources (Labs, Institutions)

28. Reviewers also consider (Cont.) Reasonableness of requested budget for work proposed Other factors (e.g., human subjects, animal welfare, inclusion policies/plans)

29. Reviewers do not consider Program relevance (Except if stated in FOA) Policy issues Funding levels Anticipated budget reductions Comparisons with other applications

30. Review Criteria For all R01, R03, R15, R21, and P01 subprojects: Significance and Originality Approach � Methods and Data Innovation Organization of the Project Investigators Environment � Facilities and Resources

32. Also... Reviewers evaluate Protection of human subjects, the environment, and animal welfare

33. Research Involving Human Subjects Important Considerations that must be addressed in the application because they impact on priority score - considered to be part of the Approach Are there any risks* to the human subjects? Are the protections adequate? Are there potential benefits to the subjects and to others? What is the importance of the knowledge to be gained? Are the plans for inclusion of minorities, both genders and children adequately addressed? Is the proposed study exempt from human subject review? No page limits

34. AHRQ Requirement:Inclusion of Priority Populations Inclusion of Children? - Rationale? Inclusion of Elderly? - Rationale? Inclusion of Rural? - Rationale? Inclusion of Inner City? - Rationale? Inclusion of Low Income? - Rationale? Inclusion of Disabled? - Rationale? Inclusion of Chronic Care? - Rationale? Inclusion of End of Life? - Rationale?

35. Review Criteria (Cont.) For FOAs (RFAs or PAs) , modified or additional criteria may be specified For other mechanisms, specific criteria apply

36. Peer Review Process Each application assigned to 3 reviewers for written comments Streamlined Review may be used Group discussion of each application, including budget recommendation Recommended score range Reviewers privately assign priority score

37. Peer Review Group Actions Unscored (Streamlined Review) Scientific Merit Rating (Priority Score)

38. Priority Scores/ Percentiles Priority scores range from: 100 (Best) to 500 (Worst) [100 x (1.0 � 5.0)] In Streamlined Review, those in lower half are eliminated (unscored) Percentiles calculated to normalize scoring behavior across review groups

39. Priority Scores Assigned by Reviewers SCORED (stronger) 1.0 ? 1.4 (Outstanding) 1.5 ? 2.0 (Excellent) (25%) 2.0 ? 3.0 (25%) UNSCORED (weaker ) 3.0 ? 4.0 4.0 ? 5.0 (50%)

40. HINTS: PREPARING AN APPLICATION Start early Read and follow instructions Clarify any confusing instructions early Do not assume staff or reviewers �will know what you mean� Include well designed tables and figures Format - consider the reviewers Proofread and check before sending

41. PREPARING AN APPLICATION (cont.) Explicitly state the purpose of the proposed work Refer to the literature thoroughly but thoughtfully Present an organized, lucid write-up Be mindful of the review criteria to be used Get advice from colleagues

43. 43 Strategies to Strengthen Applications: The Short List Ask scientist with AHRQ support to critically review your application prior to submission � more than once Talk with an Agency (AHRQ, etc.) program officer(s)

45. Help Reviewers See the Merits Think like a reviewer � Learn as much as possible about The System (e.g., figure out the likely review group) Preempt criticism Include collaborators who can compensate for your deficiencies Learn as much as possible about what research projects in your area that have been funded by the NIH/AHRQ/HRSA/CDC/etc. CRISP Database: (http://crisp.oit.nih.gov)

50. Human Subjects Protections Overview for Applicants Patrick McNeilly, Ph.D. September 2008

51. Background Belmont Report - Respect for Persons - Beneficence - Justice Common Rule (45 CFR 46) http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm Three Guiding Principles: 1. Respect for persons = Requirement for Informed Consent 2. Beneficence = Weighing Risk/Benefit 3. Justice = Adequate subject selection; adequate distribution of burdens Belmont Report principles codified in regulation at 45 CFR 46 Called Common Rule � adopted by 17 D/AThree Guiding Principles: 1. Respect for persons = Requirement for Informed Consent 2. Beneficence = Weighing Risk/Benefit 3. Justice = Adequate subject selection; adequate distribution of burdens Belmont Report principles codified in regulation at 45 CFR 46 Called Common Rule � adopted by 17 D/A

52. Common Rule Requires IRB Review Informed Consent Evaluations of all applications and proposals involving human subjects (45 CFR 46.120) Common Rule requires that each application be reviewed Criteria: 1. Risks to the subjects, likelihood & seriousness 2. Adequate protection from risks 3. Potential benefits to subjects and others 4. Importance of the knowledge to be gained Common Rule requires that each application be reviewed Criteria: 1. Risks to the subjects, likelihood & seriousness 2. Adequate protection from risks 3. Potential benefits to subjects and others 4. Importance of the knowledge to be gained

53. Peer Review vs IRB Peer review - Evaluate the application - Identify human subjects concerns in any portion of the application - Justification of exemption IRB - Initial and continuing review of research - Approve modifications to research Not expecting them to be an IRB or as knowledgeable as an IRB. First protection of human subjects is good science. Acceptable for study section to comment on human subjects protections just as it is appropriate for IRB to question the science involved. IRB reviews from the perspective of the subjectNot expecting them to be an IRB or as knowledgeable as an IRB. First protection of human subjects is good science. Acceptable for study section to comment on human subjects protections just as it is appropriate for IRB to question the science involved. IRB reviews from the perspective of the subject

54. Human Subjects Concern Any actual or potential unacceptable risk, or inadequate protection against risk, to human subjects as described in any portion of the application. - Captured in Summary Statement - Administratively coded Study section are experts in the field of study and know what is acceptable within the field. You know where the risks lie SRO will capture the discussion and write summary statement SRO will determine the codingStudy section are experts in the field of study and know what is acceptable within the field. You know where the risks lie SRO will capture the discussion and write summary statement SRO will determine the coding

55. Common Peer Review Issues No human subject protection plan Unidentified subjects Unidentified risks Informed consent issues Some typical issues that might be raised during peer review would include: The applicant failed to include any plan for the protection of human subjects, even though it was requested in the application. The peer reviewers may identify human subjects which were not considered by the applicant. (e.g., surveying providers, but not having and informed consent process for them) The reviewers may also identify risks to the subjects which the applicant had not mentioned. There is also the possibility of the peer reviewers identifying specific issues related to the informed consent document or process.Some typical issues that might be raised during peer review would include: The applicant failed to include any plan for the protection of human subjects, even though it was requested in the application. The peer reviewers may identify human subjects which were not considered by the applicant. (e.g., surveying providers, but not having and informed consent process for them) The reviewers may also identify risks to the subjects which the applicant had not mentioned. There is also the possibility of the peer reviewers identifying specific issues related to the informed consent document or process.

56. Resolution of Issues Written response from Principal Investigator to all human subjects concerns AHRQ Program Official will consult with human protections administrator on adequacy of response For those applications which will be funded, the PO is responsible for ensuring that issues related to human subjects are addressed before the application is funded. The PO should be in contact with the applicant and make sure that a written response is provided which addresses the peer-reviewer�s comments. The PO should consult with the Human Protections Administrator (HPA) regarding the adequacy of the applicant�s response to the human subjects issues. The HPA is the only the person who can change the human subjects code to allow the funding of the application. It is important that the PO request a change of the HS code. If you don�t request the change, it may not occur and funding could be delayed.For those applications which will be funded, the PO is responsible for ensuring that issues related to human subjects are addressed before the application is funded. The PO should be in contact with the applicant and make sure that a written response is provided which addresses the peer-reviewer�s comments. The PO should consult with the Human Protections Administrator (HPA) regarding the adequacy of the applicant�s response to the human subjects issues. The HPA is the only the person who can change the human subjects code to allow the funding of the application. It is important that the PO request a change of the HS code. If you don�t request the change, it may not occur and funding could be delayed.

57. Exemption from HHS Regulations Six categories of exemption Not all health services research is exempt Is there an appropriate justification of exemption? Most likely to be exempt under 1, 2, or 4. Just because it was exempt before doen�t mean it will still be if something changes. Providing list if exemptions and a decision tree for exemptions Most likely to be exempt under 1, 2, or 4. Just because it was exempt before doen�t mean it will still be if something changes. Providing list if exemptions and a decision tree for exemptions

58. Inclusion of Women & Minorities Women and members of minority groups must be included Unless a clear and compelling rationale to exclude http://grants.nih.gov/grants/funding/women_min/women_min.htm

59. Priority Populations Inner-city; rural; low income; minority; women; children; elderly; special health care needs Studies should consider including one or more AHRQ priority populations AHRQ specific requirement http://grants.nih.gov/grants/guide/notice-files/NOT-HS-03-010.html Special health care needs include those with disabilities, need chronic care, or need end-of-life health care.Special health care needs include those with disabilities, need chronic care, or need end-of-life health care.

60. Example 1 The proposed study will provide a screening tool in the form of a Bruising Clinical Decision Rule (BCDR) for discriminating bruises caused by physical child abuse vs. accidental trauma. Success will result in a BCDR to function as a screening tool to identify children and infants with bruising who are at high risk for physical abuse and require further evaluation. Study involves human subjects & study identifies risks to physicians and children in the study Physician participation is voluntary and informed consent requested. Parental permission was requested to be waived by the IRB Identified abuse will be reported as required by law Subject data will be de-identified for the analysis Reviews questioned: It was not clear why physicians were being asked for consent if unit of study was the patient. Concerned that the risks may change with the study � subjects at increased risk of falsely identified as having non-accidental trauma Study involves human subjects & study identifies risks to physicians and children in the study Physician participation is voluntary and informed consent requested. Parental permission was requested to be waived by the IRB Identified abuse will be reported as required by law Subject data will be de-identified for the analysis Reviews questioned: It was not clear why physicians were being asked for consent if unit of study was the patient. Concerned that the risks may change with the study � subjects at increased risk of falsely identified as having non-accidental trauma

61. Example 1 (cont�d) The proposed study is a prospective observational study of bruising characteristics in children over 4 years of age. Data on bruising characteristics will be collected by pediatric emergency medicine physicians and child abuse experts on 1,000 children with bruising.

62. Example 2 In an application focusing on the differences in treatment of prostate cancer for whites compared to non-whites.

63. Example 2 (cont�d) Reviewers Commented: �Given the application�s focus, prostate cancer, the exclusion of women is appropriate. However, the low minority representation and poor recruitment strategies for minorities (85% White, 7% Hispanic, 8% African American) would make it difficult for comparisons of treatment methods by racial and ethnic categories to be made .�

64. Questions?