Our Evaluation Evolution! Santa Fe Mountain Center’s Journey to Evidence-Based Research and Evaluation

1. Our Evaluation Evolution! Santa Fe Mountain Center’sJourney toEvidence-Based Research and Evaluation 2008 REAP Symposium Santa Fe, NM Sky Gray, MS, CTRS Jenn Jevertson, MS Michael Gass, PhD Santa Fe Mountain Center Santa Fe Mountain Center University of New Hampshire sky@santafemc.org jenn@santafemc.org mgass@cisunix.unh.edu

2. Overview • History of SFMC Evaluation • Anti-Bullying Initiative Evaluation Design • Results from ABI Research and Evaluation • Q&A

3. SFMC Evaluation History • 1979 • Since inception, Santa Fe Mountain Center has engaged in various research and evaluation. • 2005 • Began learning about new trends in evidenced based research in New Mexico, at AEE and beyond. • 2005-2006 • Created new evaluation tool to measure outcomes with assistance from Dr. Susan Carter • 2006 • Created the Experiential Adventure Based Resiliency Model and published our new Program Manual with assistance from Project Adventure

4. Why Bother? I have a organization and program to run!!!

5. External Motivation • Following suit with New Mexico's Behavioral Health Redesign • Following local and national trends in the evidence-based evaluation and research world • Satisfying the requirements of our largest funding source of the SFMC • Contributing to the body of knowledge in the Experiential Adventure Based profession

6. Internal Motivation • Evaluation tool outdated and ineffective (therefore not embraced by staff) • Recognition of the value and importance of good evaluation and research • Desire to contribute to the field • Desire to create stronger program outcomes • FUNDING and SURVIVAL!!!

7. Learnings • This process was at times intimidating for the organization, and there was a need for staff training at all levels • We consciously entered into new territory and had to ask for help from academic institutions (UNH) and other related experts • Deepened our understanding of Evaluation and Research and other related principles • Joining forces with the Symposium and now with CORE has pushed us further into our learning curve. • How to conduct quasi-experimental evaluation designs within our program… Anti-Bullying Initiative

8. Anti-Bullying Initiative An Experiential Adventure Based Approach to Increasing Resiliency and Decreasing Bullying Behavior

9. Anti-Bullying Initiative • Intended Outcomes • Classrooms are safer and more productive learning environments. 2. Students possess skills and strategies to deal with being the target of bullying behavior. 3. Students possess skills, strategies, and confidence to intervene appropriately in bullying situations.

10. Anti-Bullying Initiative • Program Design • One identified at-risk elementary school • Low-income, predominately Hispanic, high percentage of recent immigrant children • Entire 5th grade level • 5 classrooms; total of 82 children • 2 bilingual classrooms • Dosage throughout school year • Intro/Closing Session • 8 two-hour experiential sessions @ School • 3 full-day experiential adventure programs @ Santa Fe Mountain Center ropes course

11. Demographics anonymous Internal Assets adapted from the California Healthy Kids Survey: Resilience Module (WestEd). goals and aspirations, problem-solving, empathy, and self-efficacy. corresponds to the Search Institute’s 40 Developmental Assets Attitudes and beliefs about violence and bullying adapted from the Safe Haven Survey (WestEd). assess students’ beliefs and attitudes about violent and bullying behavior in several categories: communication, school environment, and conflict management. Violence and bullying experienced by respondent adapted from Bullyproofing Your Schools program. assesses students’ direct of experience of violent and bullying behavior in the past month in the classroom. Violence and bullying witnessed by respondent adapted from Bullyproofing Your Schools program. assesses students’ observation of violent and bullying behaviors experienced by other students in the past month in classroom. Evaluation Tool Developed by Dr. Susan Carter and Jenn Jevertson, 2006

12. Year 1 Pre/post surveys (356) 72 Tx Group 87 Comparison Group Client Satisfaction (post only) Student Essays Teacher post surveys (post only) Year 2 Follow-up pre/post surveys in Fall Focus Groups (teachers/students) in Fall Evaluation Design

13. A Step Farther…Research • Collaboration with UNH • Jesse Beightol, Master’s student • Dr. Michael Gass

14. Resilience “a class of phenomena characterized by patterns of positive adaptation in the context of significant adversity or risk” (Masten & Reed, 2002, p.75)

15. Why Resilience? School Performance • Low Assets = Low API Scores(Hanson & Austin, 2003) • Resilient Students perceive a more positive learning environment and have fewer challenges with schoolwork(Padron et al., 1999) Bullying • High resilience = Less risk taking behavior • Low resilience = More risk taking behavior and twice as likely to be a victim of bullying (Donnon & Hammond, 2007)

16. RYDM Framework

17. Current Research • Does participation enhance resilience compared to a control group? • Is there a sustained effect on levels of resilience at 4-months post-treatment? • If so, what aspects of the program may have contributed to this resilience enhancement? • Do levels of resilience vary according to gender or ethnicity?

18. QUAN Data Collect QUAN Data Analysis QUAN Results RYDM Frame- work Compare and Contrast Interpretation QUAN + QUAL QUAL Data Collect QUAL Data Analysis QUAL Results Research Design Modified Triangulation Design: Convergence Model (Adapted from Creswell and Plano Clark, 2007, p.63)

19. Demographics • Treatment Group • 52 Students • 26 Male • 25 Female • Control Group • 55 Students • 29 Male • 25 Female

20. Quantitative Measure • The Anti-Bullying Initiative Survey (Carter & Jevertson, 2006) • Goals and Aspirations • Problem Solving • Empathy • Self Efficacy

21. Focus Groups, Interviews, and Program Observation • Focus groups with students and teachers • Semi-structured interviews with SFMC facilitators and school principal • One day of program observation at the Santa Fe Mountain Center

22. Goals and Aspirations Significant increase for the treatment group from pre-test to follow-up (p = .010), with a small to medium effect size (r = .25). Significant increase from post-test to follow-up, (p = .031), with a small to medium effect size (r = .21).

23. Goals and Aspirations • “One of my goals is to go to college and become a doctor…because, like, the Mountain Center taught us…to help people out.” • “My goal was to go to college and be a nurse because the Mountain Center taught me to help people.” When I asked how the Mountain Center taught her that she wanted to help people, she said, “To be like, have more responsibility and be more friendly to the other person.”

24. Self Efficacy Significant increase for the treatment group from pre-test to post-test (p = .008), with a small to medium effect size (r = .26).

25. Self Efficacy • “Yeah my uncle really picks on me and I don’t do much about it” to a few months later “Hey I used this stuff with my uncle that picked on me and it worked.” • “Before they came…I saw people getting bullied but I didn’t do anything, like I just walked away. Cause I thought like if I tried to do something they would start bullying me. So I didn’t do anything…they told me about all this stuff, like the HAHASO and all that, um, I started helping those people out.” • “They gave you the option of going down, but they also said ‘Are you sure you want to try another step’ and then when you did you realized you thought different, like you didn’t want to go down yet, and you ended up going up to the top.”

26. Average Resilience by Gender Treatment group females higher Average Resilience scores than males 4 months after treatment.

27. Goals by Gender Significant increase in treatment group females from pre-test to follow-up (p = .009), with a medium effect size (r = .37). Significant increase from post-test to follow-up (p = .017), with a medium effect size (r = .34).

28. Female Goals by Group Significant difference between females in Goals and Aspirations at follow-up.

29. Self Efficacy by Gender Significant difference in treatment group at follow-up (p=.028), with a medium effect size (r = .31). Significant increase for females in the treatment group from pre-test to post-test (p = .025), with a small to medium effect size (r = .32).

30. Comparison to Aggregate Treatment Control Aggregate Average High 94.2 90.9 65 Medium 5.8 9.1 33 Low 0 0 2 Goals and Aspirations High 94.2 92.7 83 Medium 5.8 7.27 17 Low 0 0 1 Problem Solving High 80.8 83.6 38 Medium 19.2 12.7 55 Low 0 3.6 7

31. Comparison to Aggregate Treatment Comparison Aggregate Empathy High 75 70.9 44 Medium 21.2 23.6 50 Low 3.8 5.5 6 Self Efficacy High 98.1 92.7 NA Medium 1.9 7.27 NA Low 0 0 NA

32. External Assets - Physical and Emotional Safety -Unique and Consistent Definition of Resilience Enhancing External Enhancing External Assets at School Tools and Common Language Experience and Development Continuation The Resilience Cycle Successful Experiences in a Variety of Activities Increased Responsibility Improved Outcomes Courage to Implement Tools in Real Life Situations Consistency and Continuity Internal Assets Transfer of Lessons Conceptual Model from Qualitative Data

33. Ideas to Consider • Reliability and Validity of Survey • Subscales, Age Appropriateness, Ceiling Effects • Gender Differences • Complexity of a Topic like Resilience • Importance of a Holistic Program

34. Select References Carter, S., & Jevertson, J. (2006). Anti-Bullying Initiative Survey. Constantine, N., & Benard, B. (2001). California Healthy Kids Survey: Resilience Assessment Module Technical Report. Berkeley, CA: Public Health Institute. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage. Donnon, T., & Hammond, W. (2007). Understanding the Relationship Between Resilience and Bullying in Adolescence: An Assessment of Youth Resiliency from Five Urban Junior High Schools. Child and Adolescent Psychiatric Clinics of North America, 16, 449-471. Masten, A., & Reed, M.G.. (2002). Resilience in Development. In Snyder, C. R. and Lopez, S. Handbook of Positive Psychology. Oxford University Press: New York, 74-88. Technical Report: 5th Grade. (2006). WestEd and California Safe and Healthy Kids Program Office.