Selecting Your Evaluator: A Guide for 21st CCLC Grantees

1. 21st CCLC Grant, Title IV-B: Selecting your Evaluator January 11, 2017

2. Agenda • Purpose & Introduction • Get Started! Finding the Right One • Choosing Your Angle • Four Considerations  Starter Discussion Points • Two Separate Documents • Show Evidence  With DATA • Finishing Strong • Program Description Report • Things to Remember • Resources

3. Purpose of this Presentation Evaluation • This has been created to provide guidance to all Oregon 21st CCLC grantees: • This is based on Critical Element II in the USED State Education Agency (SEA) Monitoring Protocol. • This is to help Oregon Project Directors in identifying, interviewing and selecting the most qualified local level evaluator.

4. What is Critical Element II in the USED State Education Agency (SEA) 21st CCLC Monitoring Protocol? • “Does the SEA require that sub-grantees undergo a periodic evaluation to assess progress toward achieving the goal of providing high-quality opportunities for academic enrichment based on Principles of Effectiveness?”

5. Direct Federal to Local Questions: • 1. What is your process for evaluating your program? • 4202(c)(3)(A) • a. Who conducts your evaluation (program staff, outside contractor, State evaluator)?4203(a)(6) • b. What are the major activities and data sources? 4203(a)(13) • c. How often do you conduct an evaluation? 4205(b)(2)(a) • d. How much does it cost? Non-Regulatory Guidance H-5&6 • 2. What performance indicators and measures do you use to assess the impact of your program on student achievement and/or behavior? 4203(a)(13)(A); 4204(b)(2)(E)

6. Introduction: Local Evaluations & Expectations • A Local Evaluation • helps track program implementation, fidelity and progress • gives data to help you improve program performance • is a team effort that involves your stakeholders • A Local Evaluation • must collect & analyze your program data • must include site visits & interview staff & students • carries out an evaluation plan, develops a Year-End Report for you and your stakeholders.

7. 21st CCLC Grant Application -- 2013

8. Entire 21st CCLC Evaluation Process

9. Let’s Get Started! – Finding the Right One • Stakeholder Team Members: • School Principals • Staff Members • Student Representatives • Business & Community Partners • Parent representatives • Build a strong stakeholder team with representation from all program areas and offers multiple perspectives. • The stakeholder team should meet at least three times during the evaluation to Review & approve the evaluation plan – Review a Mid-Year Report and discuss its implications and review the Final Report

10. Potential Local Evaluator Tasks & Traits • Meet to create an Evaluation Plan with multiple growth & achievement data points plus qualitative data. • ID data to collect for a complete picture of the program and provide a plan for and manage data collection, analysis & reporting. • Does the applicant’s education meet your requirements? • Does the applicant have relevant work experience? • Is there evidence that the applicant has the required skills to conduct an evaluation like this? • Are the applicant’s references credible and relevant? • Are the applicant’s work samples professional and competent?

11. Data Administration Checklist • Do surveys or other assessments need translation for non-English speaking students or parents? • Are rooms available for conducting interviews or focus groups? • Have parents been informed about data collection and been provided any required permission forms? • Has staff been informed of classroom disruptions (e.g., interviews, observations)? • Are procedures in place to secure sensitive data?

12. Deliverables to ODE • The ODE will be requiring TWO separate documents for all programs ending their five-year cycle. • These documents are: • The 5-year Comprehensive Evaluation & • The Program Description Report

13. 5-Yr. Comprehensive Evaluation • The Power of the Picture • Use data to develop a vivid picture of a program to include: • Telling Your Story • Parts of the Story • Setting the Scene: Introduction • Putting the Pieces Together: Program Implementation • Showing the Evidence with Data: Measures & Outcomes • Finishing Strong: Summary & Conclusions

14. Program Description Report • Your evaluator will also need to submit a Program Description Report as a separate item (4 pgs. max) • “Just the Facts” • Introduction • Program Implementation • Measures & Outcomes • Analyze trends • Compare to state, national standards • Determine progress toward achieving goals, youth & family outcomes • Summary & Conclusions • Major Accomplishments • Review & Status from prior year evaluation • Plans to sustain and continue the program.

15. Things to Remember

16. Selecting the Right Evaluator • Questions, Comments and Follow-Up • Req’d Action Steps: 2 Reports by the end of Year #5 • The 5-year Comprehensive Evaluation & • The Program Description Report • Recommended: Begin Local Evaluation Yr. #4

17. Thank You! Contact • Pete Ready • 503.947-5785 • pete.ready@state.or.us