PROPOSAL REVIEWER TRAINING Florida Learn & Serve 2008-09

1. PROPOSAL REVIEWER TRAINING Florida Learn & Serve 2008-09 May 2007

2. SESSION GOALS • Participants will learn about the importance of a Request For Proposal (RFP) competitive proposal review and why it is an essential part of the grant funding process. • Participants will gain an understanding of the roles and responsibilities of grant proposal reviewers. • Participants will learn about the review process and be sufficiently prepared to serve as an informed, fair, and proficient grant proposal reviewer.

3. This Session includes…. • The Proposal Review Process • Program Office Role and Functions • Proposal Reviewer Role and Functions • Grants Training & Development Role and Functions • Proposal Reviewer Hints and Help

4. The Significance of the RFP in the Proposal Developmentand Submission Process

5. Request for Proposal (RFP) The RFP • Is a Notice • Released by the “grantor” agency • Announcing an opportunity to compete for funds • Includes criteria for eligibility, proposal structure, proposal content, and submission

6. An RFP Should… Instructions Directions Procedures • Be comprised of clear, concise requirements • Dictate the format for the proposal application • Include both qualitative and quantitative outcomes • Set scoring criteria for each section • Be followed, just as you would follow any: Refer to the RFPs: Florida Learn & Serve Pilot Projects and Florida Learn & Serve Renewal Projects: www.fsu.edu/~flserve

7. The release of an RFP starts the “official” clock ticking for qualified entities interested in submitting a competitive proposal application. Once all submissions are received (and meet certain preliminary criteria), they must be evaluated and scored (rated) by reviewers. Proposals earning enough points to equal or exceed the “cut off” score are considered eligible and may be recommended for funding.

8. Why is this significant to the proposal reviewer?

9. Because…. • Proposals can only be as good as an RFP allows! • The applicant has no control over the RFP design. • The task of the proposal reviewer is to make sense of the proposal and score it as fairly and objectively as possible. Therefore… If an RFP is disjointed and confusing, the proposal may reflect the same issues! ...and a disjointed proposal is much more challenging for the reviewer to score.

10. Role of Program Administrator during Review Process S/he is considered the Proposal Review Leader and will: • be “on-call” • manage instruments and supplies, if necessary • answer technical and procedural questions • not comment on individual applications nor provide opinions related to same • check completed score sheets for correct addition, completeness, adequate scoring comments, and legibility • prior to formally ending the review process, provide all score sheets to Grants Training and Development (GTD) for review • follow-up with individual reviewers as needed

11. Recommended Proposal Review Methodology • The Program Administrator selects qualified volunteers to participate as the reviewers. • A “Reviewer Training and Orientation” is convened, via a face-to-face session, a web-based event, or a conference call. • Each reviewer is assigned a specific number of proposals to evaluate and score. • The Conflict of Interest statement is signed by each reviewer. • An agreed-upon deadline is set for completing the reviews. (All reviews are expected to be completed by the pre-selected date and time.) • Reviewers score each proposal individually, without input from other reviewers.

12. Upon completion of the individual work, reviewers return the completed score sheets to Joe Follman. • The Program Administrator and Grants Training and Development (GTD) staff review all score sheets to ensure the scores “add up” and that all reviewer comments are legible, on target, constructive, precise, and provide adequate justification. Reviewers may be contacted for clarification or additional information. • The Program Administrator conducts a debrief of the proposal review process. (This can be accomplished via an email, a conference call, a participant feedback form, etc.) • Upon completion of the debrief, Joe Follman will send an email to all reviewers stating the review is complete and formally closed. • The reviewers are asked to complete a Grant Proposal Review Process evaluation and send it, as an email attachment, to GrantsTrainingandDevelopment@fldoe.org within three (3) working days after the review closes.

13. On the one hand, reviewers are not involved in the actual funding decisions.On the other, reviewer scores and comments provide critical input to those making the award recommendations. The Role of reviewer is an IMPORTANT one….

14. Reviewers are expected to: • Refrain from possible conflicts of interest • Apply a fairness standard to all • Expect a comparable level of effort from each applicant • Score solely on the written information in each application • Provide expertise as reflected in constructive comments supporting each score not awarded the total possible points • Score using only whole numbers (no factions, no decimals) • Maintain confidentiality (today, tomorrow, and beyond)

15. CONFLICT OF INTEREST • A conflict of interest occurs if (or when) a reviewer has a conflict that in any way might create a biased response. • Reviewers must sign a Conflict of Interest Statement after ensuring all eligible proposals present no conflict of interest. Each Reviewer must FAX, ASAP, the signedConflict of Interest Statementto Michael Murphy FAX # (850) 922-2928. At the end of the review, the signed original must be returned in the metered pre-addressed envelope provided to all Reviewers.

16. IMPORTANT INSTRUCTIONS FOR THE REVIEWERS • READall proposals through (before scoring any of them) to get a feel for the proposals individually and in comparison to each other. • EVALUATEthe quality of EACH response in EACH proposal, at the beginning of the scoring process by considering the following: • Is the response…. • Relevant and responsive to qualitative criteria? • Comprehensive and well thought-out? • An effective, logical, and realistic approach to the problem?

17. RATE each applicable section by assigning a point score. • ALWAYS JUSTIFY, with written comments, ALL ratings of less than full points. • 5. Comments explaining why a narrative component received a “Perfect Score” are strongly recommend. Generic comments, such as “Meets all criteria” state the obvious and add no value. • 6. Comments that support a professional assessment are important for several reasons and can include, but are not limited to: concrete examples, professional constructive criticism, suggestions for improvement, highlighting noted strengths, and/or emphasizing challenges.

18. The score sheet contains guiding statements for each criterion section The guiding statements do not carry equal weight. Rather, the statements are prompts to help the reviewer understand what the proposal should address. Please do not divide the points available for the section and allocate an even number for each criterion. To accurately reflect the reviewer’s impressions and expertise, scoring decisions should be based on an overall assessment of the entire question, taking all parts into consideration.

19. Written Comments are ESSENTIAL! • Comments (feedback) justify each score! • If there are questions about scoring, reviewers’ comments should be sufficient to defend the scores without further explanation (Even a year or two after the review!). • Write comments that clearly explain, to anyone reading them, why a particular score was given. • Always try to provide concrete suggestions to guide applicants in future efforts. • Comments should be positive, constructive versus critical, professional, helpful, and impartial. • Strength-based comments are important to the applicant and help anyone else reading the scored application understand why the scores were given.

20. Reviewer Hints If you can not locate a required item, it is better to say that you can’t find it than to say it isn’t there.Another method is to say the response isn’t thorough regarding . . .If an answer is located other than in the section of the narrative where it belongs, please make a note and then credit the answer in the correct section on the score sheet. Refer to the Handout “A Typical Source of Scoring Criteria (EDGAR)

21. THE FINAL SCORING PROCESS • The Program Manager will notify the reviewers their portion of the process is complete when all pertinent documents have been received and reviewed. • After the review is formally closed, the Program Office will compile all scores for each proposal. The highest and lowest score earned by each proposal will be dropped. The remaining three scores are averaged, becoming the final score for each proposal.

22. Different Approaches Some reviewers…. - Never met a proposal they liked - Never met a proposal they did NOT like - Subtract points from a maximum - Add points to a minimum (“zero” based) - Have a predetermined, personal scale Consistency and ability to support scoring decisions must be fundamental to all approaches.

23. What about…. Should proposal reviewers be concerned about these issues? Form? Grammar? Readability? Neatness? Opinions can and do differ; one acceptable approach may be to suggest that reviewers overlook these issues UNLESS the state of the proposal brings into question the applicant’s commitment to the project.

24. Be mindful of the possible tendency to utilize the “Benefit of the Doubt” strategy, i.e. assigning a higher than usual score in topic areas where a reviewer is not entirely familiar… We can’t all be EXPERTS in every area! If a reviewer is aware of this happening, it is recommended that s/he review that component again (and possibly adjust some scores) tomake sure all scoring is as consistent as possible.

25. All comments and other notes should be entered directly on the score sheet. If not, reviewers are reminded to mail all handwritten notes not included on the score sheet in the metered pre-addressed envelope provided.

26. The RFP, the entire Grant proposal packet and all documents and notes generated by reviewers during the evaluation process are public documents. Reviewers’ names, addresses and other personal information are confidential and should only be provided through Public Record requests. • One of the reviewer’s tasks is to provide strength-based feedback intended to help the applicant when preparing future proposals. • It is very important to make sure written comments are professional, clearly stated, constructive, and useful. • Applicants are encouraged to obtain copies of completed scoring packets. Reviewer scores and comments provide feedback that may help the next time the applicant develops a proposal. Reviewer comments count! Success rates improve in the next competition among those who request the review packet. CAUTION: See H-7 Grant Investigation

27. The Final Steps • The Program Administrator will not close the review until all score sheets are assessed for accuracy, completeness, and quality and appropriateness of comments. • All score sheets, other reviewer notes, and comments are routed to the Program Administrator, electronically or via mail. • Following internal review and final approval of the score sheets by DOE, the reviewers are notified that they have • completed their portion of the competitive process. • The reviewers complete a Grant Proposal Review Process Evaluation form and email it to: • GrantsTrainingandDevelopment@fldoe.org. • Reviewer feedback is IMPORTANT and • helps to improve the RFP Competitive process.

28. Contact Information Grants Training and Development Bureau of Contracts, Grants, and Procurement Florida Department of Education325 W. Gaines Street, Room 344Tallahassee, Florida 32399-0400 Main (850) 245-0499 SunCom 205-0499 Fax (850) 245-0730 SunCom 205-0730 Susan Howell Susan.Howell@fldoe.org

29. THANK YOU! Your valuable time and expertise are greatly appreciated!