1 / 20

In Depth Panel Review Training

In Depth Panel Review Training. Activity: Mock Panel Review. To evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: A concise problem statement that identifies the current condition(s) to be addressed by the project

licia
Download Presentation

In Depth Panel Review Training

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. In Depth Panel Review Training

  2. Activity: Mock Panel Review To evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: • A concise problem statement that identifies the current condition(s) to be addressed by the project • Supporting information or data detailing the scope and nature of the problem • Current challenges standing in the way of addressing the problem • A clear description of the community to be served and who the intended beneficiaries are What score, out of 10 possible points, did you give the application narrative? Why?

  3. Panel Review Roles Review Director: oversee the entire panel review process Logistics Contractor: handles payment and ARM support Program Area Manager (PAM): An ANA representative with the responsibility to approve the PSR Subject Area Manager (SAM): An ANA representative charged with assisting the panel on the process and the editing of the PSR Reviewer: A member of the panel charged with evaluating the application and documenting their evaluation by scoring and commenting on the application strengths and weaknesses Facilitator: A member of the panel charged with managing the panel discussions and writing the PSR

  4. Panel Review Structure

  5. Approval Process of PSRs

  6. Calendar of Events Pre-Panel (April 1st – April 27th) Reviewers and facilitators are identified and trained Session 1: P&M, EMI, and ERE (April 25th – May 9th) Reviewers and facilitators evaluate and panel applications Session 2: SEDS and SEEDS (May 9th – May 23rd) Reviewers and facilitators evaluate and panel applications Post Panel Payment is sent to reviewers and facilitators

  7. The Application Review Module (ARM) • Do • Enter the Score First • Use the ARM text box to write your comments • Spellcheck • Save every 20-30 minutes • Do Not • Use Google Chrome • Click the ARM support link on the ARM login page Email ANAReviewer@acf.hhs.gov with all ARM questions!

  8. Three Principles for Reviewing Applications • Evaluate the extent to which the application responds to the FOA Evaluation Criteria • The FOA Evaluation Criteria is the only measure used to evaluate the application • Do not compare applications against each other

  9. How Do You Determine the Significance of a Weakness and a Strength?

  10. Examples of Good Weakness Comments The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach) The applicant did not describe anywhere in the application how community input was used in the development of the project, making it unclear if the project proposal is truly community-driven. (page 25, Approach) The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be evaluated, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected)

  11. Not everyone agrees at the beginning Structured and focused Initial scores vary. Sometimes by 20 points or more What is a panel discussion like? Hearing the other reviewers point of view made me reconsider my score We spent about an hour talking about each application

  12. Panel Group Norms • Attend all meetings • Keep Focused • Listen and show respect Focus the meeting by creating a meeting plan The FACILITATOR sets group norms, focuses the meeting, asks probing questions, shares information, and documents the discussion • Probing Questions • What criteria did you use? • How did you determine this score? • What documentation did the application provide that substantiates the approach? Share initial reviewer comments Document the discussion to help write the Panel Summary Report

  13. Requirements of a Panel Summary Report (PSR) • Factually Accurate • Adheres to the FOA Evaluation Criteria • Consistent • All comments justify the score • Polished and Well-Written

  14. Example of an Inconsistency in a PSR Strength Comment The applicant clearly showed how the current staff will manage and complete the project on time. (page 75, Objective Work Plan) Weakness Comment The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)

  15. Writing a PSR that Represents the View of the Panel • Create an initial PSR • Take notes during the panel discussion • Draft the final PSR • Submit the final PSR to the PAM/SAM for comments and approval • Share the final PSR with the reviewers

  16. Tips on Writing a PSR that will be Approved • Reference Application Page Numbers • Use the wording of the relevant FOA evaluation criteria • Make it grammatically correct

  17. Example of a Good PSR Comment The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be measured, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected) FOA Text in Outcomes Expected Evaluation Criteria To evaluate the projects intended impact, reviewers will consider the extent to which: • The condition(s) identified in the problem statement will be addressed • The lives of community members and beneficiaries will improve To evaluate the impact indicator(s), reviewers will consider the extent to which: How impact will be measured with at least one impact indicator using the same measure at three points in time, baseline (beginning of project), end of project, and three years post-project.

  18. PSR Approval No Yes Is the PSR factually accurate? Does the PSR adhere to the FOA Evaluation Criteria? Is the PSR Consistent? Do the comments justify the score? Is the PSR polished and well-written? Approved!

  19. Roles and Responsibilities Recap Reviewer Evaluate the Application Document evaluation by scoring and justifying score with comments Submit score and comments to facilitator through ARM SAM/PAM Responsibility 8. Determine whether the PSR can be approved. If not, back to step 6. Facilitator Responsibility 9. Share final PSR with reviewers to ensure it reflects panel opinion. Facilitator Create initial PSR by compiling all reviewer comments Holds panel meeting with reviewers Revises PSR based on reviewer comments, discussion or SAM/PAM recommendations Submit PSR to SAM through ARM SAM Responsibility 10. Submit PSR to PAM for approval through ARM PAM Responsibility 11. Approve the PSR

More Related