AmeriCorps Grantee Training Evaluation and Research September 11, 2014

1. AmeriCorps Grantee Training Evaluation and Research September 11, 2014

2. Session Objectives • Review policy context for evaluation • Review evaluation requirements • Share examples of grantee evaluation practices • Share resources

3. Federal/State Policy Context • Federal • Evaluating Programs Requirements and Procedures (CFR 45 §§.700-.740) • CNCS finalized these regulations July 8, 2005 • State • Program Design Policy E.3—Evaluation: Competitive programs need to follow evaluation requirements as described in the AmeriCorps Regulations…Formula programs are not required to conduct an evaluation. • Policy adopted 2009

4. Definition of Performance Measurement Performance measurement is the process of systematically and regularly collecting and monitoring data related to the direction of observed changes in communities, participants (members), or end beneficiaries receiving your program’s services. It is intended to provide an indication of your program’s operations and performance. §2522.700

5. Definition of Evaluation …evaluation uses scientifically-based research methods to assess the effectiveness of programs by comparing the observed program outcomes with what would have happened in the absence of the program. §2522.700

6. Comparing Performance Measurement and Evaluation

7. Building Evidence of Effectiveness

8. Evaluation Study Designs and Causal Impact

9. Evaluation Study Designs & Causal Impact

10. Evaluation Study Designs and CNCS Requirements * Fulfills CNCS evaluation requirement for large grantees if a reasonable comparison group is identified and appropriate matching/propensity scoring is used in the analysis.

11. Evaluation Requirements • Competitive Programs • Over $500,000 • Under $500,000 • Formula Programs

12. Evaluation RequirementsCompetitive ($500K +) • Independent evaluation • Quasi- or experimental design study • Cover at least one year of operation • Submit evaluation with any application for competitive funds § 2522.710

13. Evaluation RequirementsCompetitive (less than $500K) • Internal evaluation • Quasi- or experimental design (optional) • Cover at least one year of operation • Submit evaluation with any application for competitive funds § 2522.710

14. Competitive Evaluation RequirementsWhat is Due When?

15. Evaluation RequirementsFormula Programs • State commissions establish evaluation requirements for formula grantees • CV does not require formula grantees to conduct evaluations (per 2014 Program Design Policies)

16. Strengthening the Evidence Base Formula and Competitive (less than $500K) • Select a study design most appropriate for developmental phase of AmeriCorps program (process, implementation, outcome or impact) • Encourage use of experimental or quasi-experimental designs but not required • Assure program design is based on or adapted from a similar program that has evidence from an evaluation

17. Evaluation & Grant Review • 2014 applications scored and placed into one of four tiered evidence levels. Applications with a stronger evidence base received more points. • Evaluation reports submitted are assessed in terms of the quality of the evaluation designs and the studies’ findings. These assessments may be used to inform CVs/CNCS’s consideration of the selection criteria and for the purpose of clarifying or verifying information in the proposals. • Applicants that failed to submit evaluation reports as required had points removed.

18. Research and Evaluation Grantee Panel Stephanie Biegler Chief Program Officer Birth & Beyond/Child Abuse Prevention Council Atalaya Sergi Deputy Director Jumpstart CA/Jumpstart Julie McClure and Sara Sitch Director, Assistant Director CaSERVES Volunteer Infrastructure Program/ Napa County Office of Education Matt Aguiar Chief of Staff Reading Partners CA/Reading Partners

19. Questions?

20. Resources Available • Evaluation FAQs: • Electronic CFR: www.gpoaccess.gov/ecfr • Select Agency List from left navigation bar • Scroll down to CNCS, select 45 CFR Chapters XII, XXV • Scroll down to XXV, select 2500-2599 • Select 2522 • Select Subpart E-Evaluation Requirements • National Knowledge Network: • https://www.nationalserviceresources.gov/evaluation-americorps#.VAowPbdOXcs • CV Program Officer

21. THANK YOU!