Local Evaluation of the SS/HS Initiative

1. Local Evaluation of the SS/HS Initiative Steven Rider, Ph.D. Pacific Research & Evaluation, LLC Meg Small, Ph.D. Penn State University Jennifer Cross, Ph.D. Colorado State University

2. Overview of Presentation • Complexity of the SS/HS Initiative • Role of the Evaluator • Top Ten Traits of an Effective Evaluator • Lessons Learned

3. Complexity of Initiative • Broad Focus • Three Federal Agencies • Education • Health & Human Services • Justice • Six Project Elements • Safe School Environment • AOD & Violence Prevention and Early Intervention • School and Community Mental Health Services • Early Childhood Services • Supporting & Connecting Schools and Communities • Safe School Policies

4. Complexity of Initiative • Wide Variety of Stakeholders • Differing goals • Different populations served • Varying political agendas • All (or at least most) want sustainability

5. Complexity of Initiative • Many Programs Under One Umbrella • Evaluate programs separately? • Evaluate impact of SS/HS project as a whole? • Evaluate quality of the local SS/HS partnership?

6. A Project Director’s perspective

7. An evaluator’s perspective

8. Selected Activities of Cohort One and Two Evaluators • Setting up and sustaining services & evaluation activities • Developing research agendas • Carrying out trend analyses • Facilitating partnerships among agencies • Developing newsletters • Presenting findings to administrators

9. Recommendations from Cohort One Project Directors • Collect, analyze and report data that are more program-specific • Evaluation activities should be more closely linked to decision-making

10. General “Role” Categories • Objective Outsider • Testing a new program • Fully Integrated Collaborator • Technical assistance on substantive issues • Engaged Partner • Improve program management • Manage relationships between stakeholders • Communicate findings • Develop sustainability strategies

11. Top Ten Traits of an Effective Evaluator

12. 10. Asks, “What data do you need?” • Talks to key stakeholders about the data they want. • Asks school principals/teachers what data they need.

13. 9. Is Flexible • Knows what matters and what doesn’t. • Creates evaluation strategies that work for the project. • Bends on how and when the data comes in, but not what the data is.

14. 8. Watches the Clock • Knows what the decision deadlines are. • Plans to collect and report data in time for use in decision-making. • NEVER, EVER misses a deadline set by the Federal Project Officer.

15. 7. Is an Outsider • Uses her position as an outsider to: • Collect sensitive data • Challenge beliefs/actions/decisions • Create the greatest benefit for the project

16. 6. Rigorous Not RIGID • Balances research standards with practical concerns • Priority is collecting quality data for the benefit of the project • Thinks about what programs could be published studies and contribute to knowledge about “evidence-based programs”.

17. 5. Dares to Be Different • Evaluation plans each year should change • Builds on research from year to year for sustainability • Has different standards for different programs

18. 4. Knows What’s Goin’ On • What collaborations and partnerships are working? • What programs are stumbling? • Looks for ways to improve implementation and or processes.

19. 3. Knows the audience • Write reports for different audiences • Federal Project Managers • Project Director • District Administrators • School staff • Program staff • Parents • Community stakeholders

20. 2. Your New Best Friend • Meets with project director once a week. • Meets with management team regularly. • Teaches you to LOVE data.

21. Asks, “How can I help?” • Starts every conversation by asking how he can help. • In every interview, asks what the people need. • Looks for every opportunity to provide, collect, or interpret data.

22. Lessons Learned • Show Stakeholders that Evaluation is Valued • Evaluator should be around early and often • The support of the Project Director for evaluation is essential • Get relevant data back to stakeholders in a timely manner

23. Lessons Learned • Project Director and Evaluator Teamwork is Imperative • Different roles and areas of expertise • Should complement each other • Value the perspective and contributions of the other • Shared goals of: • Providing helpful programs and services • Learning about these programs and services • Sustaining worthwhile programs and services

24. Lessons Learned • A Strong Start is Crucial • Long project • Broad scope • Many stakeholders • Lots to do • You don’t want to fall behind!