Social Sector Trials Evaluation Summary: Implementation Phase - April 2012

1. Social Sector Trials Evaluation Summary: Implementation Phase - April 2012 2011 2012 2013 Implementation evaluation phase Implementation Phase Evaluation Inputs Tasks Outputs Immediate outcomes Intermediate outcomes Ultimate outcomes File Reference A588415 Date 4 April 2012 The Trials Evaluation Findings: Service Delivery Findings: Young People • Background • Trialling New Approaches to Social Sector Change (the Trials) tests the • ability of a Committed Individual (CI) or lead Non-Government Organisation • (Lead) to use cross-agency and community resources to improve • outcomes for young people aged 1218 in six locations: Levin, Gore, • Kawerau, Te Kuiti, Taumarunui and Tokoroa. • Evaluation of the Trials • The Centre for Social Research and Evaluation, within the Ministry of Social • Development, leads the evaluation in partnership with the Ministries of Justice, • Education and Health, and the New Zealand Police. • The evaluation is divided into three phases: • Establishment Phase Evaluation: reported in July 2011 • Implementation Phase Evaluation: this A3, covering July to December 2011 • Final Phase: reporting progress towards outcomes, to be reported in 2013 Issues for young people Interview participants consistently identified the following factors as compromising young people’s ability to help themselves: What has happened? Trials locations have put significant effort into planning and delivering new and innovative youth services. Each location has engaged members of their community in the Trials, raised the profile of young people and the issues they face, delivered collaborative initiatives and created new roles to work directly with young people. • dysfunctional family environments • drug and alcohol use and addiction • lack of education, training and employment (ETE) • lack of specialist mental health services • lack of transport to out-of-town ETE and specialist mental health services • lack of positive young role models. What has changed? Improved service development and more services for young people Findings from the establishment evaluation indicated that there were opportunities to improve the quality of collaboration between youth service providers. Participants from the current evaluation believed the way organisations were working together had improved in a number of key areas, including: “It is hard for young people to maintain positive change because there may be only one high school and they can’t easily change their circle of influence.” (NGO interviewee) • planning and development of youth services so they are more integrated, collaborative and innovative • better knowledge about what services were available for young people and the best approach to achieve positive change • less duplication of services for young people • increased community buy-in and community participation (including young people) in youth service planning. What has happened for young people? All locations have established, or were in the process of establishing, new youth coordinator roles. These roles were different in each location, and included: • intensive mentoring roles with at-risk young people • School and community-based youth workers • activity-focused youth worker roles • youth participation coordinator (focused on engaging young people in Trials). • “Young people told us there was no one to talk to, they lacked a sense of belonging... The • Youth Coordinators are going to make a remarkable difference.” (NGO lead interviewee) “… This gives responsibility to us and is not dictated from on high so the enthusiasm generated is ours, we are helping our own.” (Local Government interviewee) Figure One: Evaluation Phase Purpose of the Implementation Phase Evaluation The purpose of this phase of the evaluation was to document progress to date (including tasks and outputs), barriers and opportunities encountered, and identify immediate outcomes of the Trials. Many evaluation participants reported that there were already more services available for young people now than before the Trials and this would continue to improve. Other actions to address improved youth outcomes have been implemented in the Trials locations. Examples of these include: “There was little available before the Trials started that wasn’t sport focused but now we are developing a wider approach and more opportunities are presenting.” (NGO interviewee) • a day programme established for 20 young people previously not attending school or alternative education • youth hubs established and in development to provide appropriately supervised venues and events for youth • part-time holiday employment in local businesses secured for 12 young people • a youth mentor employed to provide intensive mentoring with 30 at-risk young people • a system for using community service to enable young people to pay outstanding fines. • Method • Findings reported in this A3 were drawn from: • individual interviews, in four of the six Trials sites (Te Kuiti, Tokoroa, Taumarunui and Levin) with Trial leads and key stakeholders - mainly governance group members • a short pre-interview questionnaire administered to the interviewees • a review of key documents. Barriers • The following barriers to integrated youth services in the Trials sites were identified : • Not all service deliverers were engaged in the Trials (most areas had one key NGO inadequately engaged, or disengaged, from the Trials). • Regional representation from some government agencies was sporadic due to the large geographic area some government representatives covered. • Government agency input was sometimes limited by their internal bureaucracy, for example: they had siloed approaches, and agencies had different geographical boundaries from each other. High Level Findings What has changed? Ratings from the pre-interview questionnaire with evaluation participants showed they had considerably more confidence that the Trials would deliver improved youth outcomes for at-risk young people now than they had before the Trials. Confidence the Trials will make a difference Evaluation participants were largely optimistic about the Trials and the anticipated gains the Trials would bring to the young people in their community. They described early successes that buoyed their confidence that the Trials will make a positive difference. They also recognised that the Trials were aiming to address a lot of deeply embedded difficulties for young people and change would take time. Improved leadership and planning processes Participants viewed the Trials governance groups they participated in as highly motivated and productive. They considered the planning process for youth service delivery to be better informed, more integrated across the youth sector and more innovative now, than it was prior to the Trials. Better understanding of young people New relationships through the formation of the governance groups had increased participants’ understanding of the young people in their community, including what strategies were likely to be effective for sustaining positive changes for them. Positive Enablers • Trials leads were seen as skilled agents of change • Trials leads were described as integral to the success of the Trials. In particular they: • effectively mobilised key people in the community to get involved • actively planned and delivered services for young people • formed extensive networks of people and helped build bridges between them. • Influential governance groups and resources • Having influential decision makers within governance groups was viewed as critical to • gaining traction for youth within the town. Specifically, influential local governance: • gave the Trials status • helped leverage additional resources to allow quick Trials-related actions. • Government and NGO support • All levels of government and NGO support were described as enabling Trials success, • specifically: • Local government helped elicit community engagement and local buy-in to the Trials. • Regional government agency representatives were informative and could mobilise significant resources, including people. • The Trials Directorate and Ministerial championship helped maintain the impetus of the Trials and resolve issues pertaining to central government input. • NGOs contributed key knowledge about what they delivered for young people and how to make a difference for them. Potential Barriers to Longer-term Impact • Issues raised as potentially reducing Trials success in future included: • lead NGOs becoming overstretched and unable to deliver at the same intensity for young people overtime • other government initiatives superseding the Trials and taking the focus away from young people. Potential Enablers of Longer-term Impact No differences between CI and NGO model outcomes No major differences were observed in this evaluation, between what the CI and NGO models were achieving. Some differences in the enablers and barriers each model faced were emerging and will be investigated more closely in the final phase evaluation, for example the importance of collegial support between CIs and how lead NGOs engage other NGOs in service delivery. Interviewees identified the following elements as important for sustaining longer-term change: • ensuring an alignment in each Trials location between the vision of the Trials and what local leaders want for their town • local leaders’ commitment to long-term change for their young people • governance groups made up of people from within the community who were passionate about young people and able to influence improvements for them.