Austin Youth Program Quality: A Snapshot

1. Austin Youth Program Quality: A Snapshot Dr. Raphael Travis – Texas State University School of Social Work

2. Overview • Five waves of data collection and analysis • Sixth wave coming soon • 219 assessments entered and analyzed • Only possible from a dedicated few volunteering time, energy and resources … you!

4. A Snapshot • Average levels of program quality areas across programs • Successes and Opportunities • Analyses across each wave of data • Distinctions between self-assessments and external assessments • A spotlight on what predicts high youth engagement (our most desirable indicator of quality)

5. The Pyramid of Program Quality Plan Make choices Engagement Reflect Lead and mentor Be in small groups Partner with adults Interaction Experience belonging Encouragement Reframing conflict Supportive Environment Skill building Session flow Active engagement Welcoming atmosphere Psychological and emotional safety Safe Environment Program space and furniture Emergency procedures Healthy food and drinks Physically safe environment Youth Voice and Governance Professional Learning Community

6. Action Preparation Contemplation Score Reports: Keep in Mind • Observation scores represent a snapshot –this has limitations and value. • These are aggregates where multiple observations occurred. • The overall story is more important than the actual numbers… But, today we will talk about numbers • What you do with it matters (how you step up to action).

7. Wave 5:National Sample vs. External Assessments

8. Wave 5:Self Assessment vs. External Assessment

9. Wave 4 vs. Wave 5 (All Assessments)

10. Wave 5 Safe Environment: Very High Scores

11. Wave 5 Supportive Environment:Very High Scores – Even Higher (~5.0)

12. Wave 5 Interaction: Big Spread in Programs Occur; Scores Dip Below 2 and 3

13. Wave 5 Engagement:Substantial Variability; But Majority are 3 and Below

14. Predicting Engagement • Certain processes are consistently more important than others in predicting high levels of engagement

15. Safety • SUCCESS: • SAFEB3/4 – Ventilation, lighting, temperature okay • OPPORTUNITY: • SAFEC4/C6 – Appropriate safety and emergency equipment; access to outdoor space is supervised

16. Supportive • SUCCESS: • SUPPORTF2/F3/J1/J2 – Warm tone, friendly, smile; staff actively involved, acknowledge accomplishments specifically • OPPORTUNITY: • SUPPORTK2/K3 – (Conflict) Staff seek input from both youth about cause and solution of negative behavior; encourage youth to reflect on actions and consequences when conflict

17. Interaction • SUCCESS: • INTERACTL3/O2 – Youth identify with program (shared language, gestures, and traditions); staff provides explanations, guidelines and directions • OPPORTUNITY: • INTERACTM2/N2/N3 – Staff use two or more ways to form groups; youth have chances to mentor others and lead groups

18. Engagement • SUCCESS: • ENGR1/R4 – All youth engage in intentional reflecting on activities; staff provide structure for youth feedback on activities • OPPORTUNITY: • ENGP2/Q2 – Two or more strategies used to plan projects; all youth can make at least open-ended choice during program processes

19. Explaining High Youth Engagement • Safety, supportive, and interaction explained half of all the variability in engagement scores, • in addition, even more (58%) when looking only at self-assessments.

20. Significant Predictors of Engagement (All) • Wave 1: SUPPORTH • Wave 2: SUPPORTI, SUPPORTJ* • Wave 3: SUPPORTG* • Wave 4: SUPPORTH, SUPPORTJ, INTERACTN • Wave 5: SUPPORTK • ALL: SAFED, SUPPORTH, SUPPORTI, SUPPORTK, INTERACTN

21. Significant Predictors of Engagement (Self and External Assessments) • Predictors of Engagement (Self) • ALL: SUPPORTG, SUPPORTH, SUPPORTI, SUPPORTK, INTERACTL, INTERACTN • Predictors of Engagement (External) • ALL: SUPPORTH, INTERACTN

22. What Explains Engagement THE MOST? • SUPPORTH: Active Engagement • Youth can create, combine, reform materials/ideas toward tangible products; reflect on what they do • SUPPORTI: Skill-Building • Support youth in building new skills; persistence • SUPPORTJ: Encouragement • Acknowledge what youth do; actively involved with youth; open-ended questions • INTERACTN: Leading and Mentoring • Youth have opportunities to act as group facilitators and mentors

23. So What? What Does This Mean for You? • For assessors • it means continuing to do the great work you are doing. • You have collected high quality, meaningful information – both self-assessors and external assessors. • For methods trainers • it means there may be indicator training areas to highlight even if that is not the focus of your particular workshop • (e.g., active engagement, skill-building, encouragement, leading and mentoring). • These processes seem to help distinguish programs that are highest in engagement from those that are not. • For program leadership and staff • it means continuing to pay attention to how your program operates as much as you pay attention to the results. • Highlight and pay attention to colleagues that promote these influential processes and model effective program facilitation. For others, continue to support the work!