Assessing Impact of Youth Work in Language Revitalisation: Challenges, Opportunities, and Next Steps

1. Evaluating and assessing the impact of working with young people in regional and minority language revitalisationRhys Jones, Elin Jones (Prifysgol Aberystwyth) and Fiona O’Hanlon (Edinburgh University)

2. Strwythur/Structure • Cyd-destunallanol: gwaithieuenctidmewniaithleiafrifol • Cyd-destunmewnol: heriau a chyfleon • Theorinewid • Cymorth a chamaunesaf • External context: youth work in a minority language • Internal context: challenges and opportunities • Theory of change • Support and next steps

3. Cyd-destunallanol/External context • Cyd-destunallanol 1 • Polisiar sail tystiolaeth (Test, Learn, Adapt) • Mae ymyrraethau’nmethuynaml • Mae angencyfiawnhaugwariantcyhoeddus • External context 1 • Evidence-based policy (Test, Learn, Adapt) • Interventions often fail • The need to justify public spending

4. Cyd-destunallanol/External context • Cyd-destun allanol 2/External context 2 • “The strategy recognises... the need to strengthen the evidence base on the impact of youth work across Wales.” • “much more needs to be done to systematically and robustly evidence the contribution of youth work” (both from the National Youth Work Strategy for Wales 2014-18). • Youth Work Reference Group (2015): “to develop and implement a National Outcomes Framework”.

5. Cyd-destunallanol/External context • Cyd-destun allanol 2/External context 2 • “Self-evaluation processes use the triangulation of evidence-based information and data, people’s views and direct observation of practice to inform outcomes and measure impact on young people. In order to plan improvement effectively we need to know how well we are doing” (Scotland’s National Youth Work Strategy 2014-19), p. 17). • Measuring impact as one of 5 principles of youth work

6. Cyd-destunallanol/External context • Cyd-destun allanol 2/External context 2 • Develop efficient and user-friendly ways of collecting and analysing quantitative and qualitative data at local, regional and national level to inform service planning, delivery, assessment and continuous improvement (Ireland’s National Youth Strategy 2015-20, p. 35) • “Undertake and publish research that demonstrates the positive role of youth work in achieving national outcomes and local priorities” (National Youth Council of Ireland’s Strategic Plan, p. 23).

7. Cyd-destunallanol/External context • Cyd-destun allanol 3/External context 3 • “Continue to build on our evidence about the Welsh language and its speakers, as a basis for assessing the effectiveness of our interventions and developing Welsh language policy. • Considering, as programmes are being planned, the most appropriate methods of evaluating their effect. • By combining qualitative and quantitativedata we aim to enhance our understanding of speakers’ linguistic experiences and practices as they move through different periods of their lives” (Cymraeg 2050, p. 77).

8. Cyd-destunallanol/External context • Cyd-destun allanol 3/External context 3 • “In order to track progress on the Plan, Bòrd na Gàidhlig will collate and publish information from a range of sources, including public authorities, Gaelic organisations and community groups in an annual National Gaelic Language Plan progress report. • Increasing the use of Gaelic is at the core of the National Gaelic Language Plan and ways of measuring this will be developed to give a clear picture as to how the strategies for increasing use are working” (National Gaelic Language Plan 2018-23, p. 55).

9. Cyd-destunallanol/External context • Cyd-destun allanol 3/External context 3 • Local Action Plans “supplemented by a language audit process which will identify ongoing strengths and weaknesses and provide time-series evidence of the impact which language-related policies and reforms would have on actual language use” (20 year Strategy for the Irish Language, p.24). • Overall… • An emphasis on the need to demonstrate outcomes and impact • Coming from funders and practitioners • Couched in terms of effectiveness rather than value for money • New methods, synthesis, a focus on time

10. Cyd-destunallanol/External context • Beth yw’rcyd-destunallanolargyfermesurcanlyniadau a thraweffaitheichgwaith? • Beth yw’rcyd-destunpwysicaf (gwaithieuenctidneuwaithiaith)? • Sutmae’rangeniarddangoscanlyniadauyndylanwaduar y berthynasgydanoddwyr? • What is the external context for evaluating the outcomes and impact of your work? • What is the most important context (youth work or language work)? • How does the need to show outcomes influence relationships with sponsors?

11. Cyd-destunmewnol/Internal context • “Urdd Gobaith Cymru is a National Voluntary Youth Organisation with over 53,000 members between the ages of 8 – 25 yrs old. It provides Sport, Cultural, Community and Residential opportunities through the medium of Welsh for children and young people in Wales that will enable them to make a positive contribution to the community. The Urdd has nurtured generations of young men and women to be proud of their country, open to the world and living embodiments of our language and culture, along with the universal values which we cherish in Wales. Its contribution to…the lives and confidence, and mental health of our young people over the years has been immense” (Urdd Gobaith Cymru Annual Report 2017-18)

12. Cyd-destunmewnol/Internal context • “Gael Linn is a national organisation whose principal aim is to foster and promote the Irish language and its heritage as a living language and as an expression of identity at policy and at community level.” • Feachtas is a young Irish-language organisation that was established in 1980. Objectives of the organisations: • a) To promote the Irish language among young people • b) To help the social and personal development of a young person • c) Help to build an Irish-language community • Cumann na bhFiann provides opportunities for young people to meet through the medium of Irish and create networks of Irish speaking friends throughout the country

13. Cyd-destunmewnol/Internal context • ComannnaGàidhligorganises a range of sporting and cultural activities for Gaelic-speaking young people. It aims to: • create opportunities for them to use Gaelic out with the classroom so that young people can use Gaelic in informal settings along with their peers. • increase the young people’s confidence in using Gaelic and their fluency.   • Research shows young people, parents and teachers feel local community initiatives are successful in terms of confidence and language skills (Young 2018) • Fèisean nan Gàidhealtutors young people in traditional music, Gaelic song & drama. It aims to ‘bring our culture to our young people.’ -Around 6,000 young people take part in such tuition each year.   -Research shows participation to be associated with positive attitudes to Gaelic and motivation to use the language (Broad & France 2005).

14. Cyd-destunmewnol/Internal context • Hyrwyddoiaithleiafrifol (gallu ac ynenwedigdefnydd) • Cysylltiadgydahunaniaeth a diwylliant • Creucymunedauieithyddol • Ondhefyd… • Creudinasyddion ‘gwell’ • Gwellaperfformiadacademaidd • Lleihauanhafaliadaueconomaidd/NEETs • Promoting a minority language (ability and especially use) • A connection with identity and culture • Creating language communities • But also… • Creating ‘better’ citizens • Improving levels of academic achievement • Addressing socio-economic inequalities/NEETs

15. Cyd-destunmewnol/Internal context • Beth ywprifamcanioneichmudiad? • I baraddaurydychchi’nasesucanlyniadau a thraweffaitheichgwaith? • Sutydychyngwneudhyn? • Oesarferdda? • Beth yw’rheriauneurwystrau? • Oescyfleon? • What are your organisation’s main aims? • To what extent do you evaluate the outcomes and impact of your work? • How do you do this? • Is there good practice? • What are the challenges or obstacles? • Are there opportunities?

16. Cyd-destunmewnol/Internal context • Capasiti (amser, sgiliau) • ‘Pellter a deithiwyd’ • Treiglamser • Cymdeithasfel system cymhleth • Gofynionasiantaethauariannugwahanol • Heriaupenodolymmaesiaith: • Defnyddiaith • Perthynasgydaffactoraumwyhaniaethol • Ai data meintiolsyddorau? • Capacity (time, skills) • ‘Distance travelled’ • The passage of time • Society as a complex system • The demands of different funding agencies • Specific challenges in relation to language: • Language use • Link to more abstract factors • Can quantitative data capture this?

17. Cyd-destunmewnol/Internal context • Cyfleon • Rydychchi’nadnabodeich ‘cleientiaid’ ynddaiawn • Mae gennych chi storfafawr o wybodaethamdanynt • Mae cyfleilunio agenda gyffredin? • Opportunities • You know your ‘clients’ very well • You possess a large store of information about them • There is an opportunity to shape a common agenda?

18. Cyd-destunmewnol/Internal context • Allbwn, canlyniad a thraweffaith • Allbwn: gweithgareddau, prosesau, cynnyrch • Canlyniad: canlyniad y gweithareddau a phrosesau • Traweffaith: effaithehangach a mwyhirdymor • Output, outcome and impact • Output: actions, processes, products • Outcomes: the results of actions and processes • Impact: broader and longer-term effects

19. Cyd-destunmewnol/Internal context • “While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale” (Andrew Harding, LSE). • Path fath o dystiolaeth mae modd ei gasglu?/What kind of evidence is it possible to collect? • Pa fath o dystiolaeth a ddisgwylir i chi ei arddangos?/What kind of evidence are you expected to produce?

20. Cyd-destunmewnol/Internal context

22. Theori newid/Theory of change • Theorinewid • Deiagramsy’ndangosdylanwadrhaglenneuymyrraethargleientiaid • Amlinelluyrhollweithgareddau a phrosesau (allbwn) • Sutmae’rprosesauaybynarwain at ganlyniadau • Naturtraweffaithehangach y canlyniadau • Theory of change • A diagram that represents the effect of a programme or intervention on clients • Outline all of the activities and processes (output) • How these processes etc lead to outcomes • The broader impact of the outcomes

23. Dull y NWA/The MSC method

24. Theori newid/Theory of change • “We believe that a clear, concise and convincing explanation of what you do, what impact you aim to have and how you believe you will have it is a vital foundation of any programme, and a prerequisite for effective evaluation” (NESTA, Guidance for Developing a Theory of Change for Your Programme).

25. Theori newid/Theory of change • Cymrwch 10 munudiluniodeiagramtheorinewidargyfer un o amcanioneichmudiad • A ydyw’nddefnyddiol? • A ydyw’narwain at gwestiynau/ystyriaethaunewydd? • Take 10 minutes to produce a theory of change diagram for one of your organization’s aims • Is it useful? • Does it lead to new questions/considerations?

26. Camau nesaf/Next steps • Cymorth • CyrffyngNghymrua’r Alban ynbarodihelpu • PrifysgolAberystwyth • PrifysgolCaeredin • WISERD • Pa gefnogaethsyddeiangen? • Help • Institutions in Scotland and Wales are ready to help • Aberystwyth University • Edinburgh University • WISERD • What support is needed?

27. Camau nesaf/Next steps • Camaunesaf • Ysgrifennudogfenar sail y cyflwyniad, yrymchwilsy’n sail iddoa’rmewnbwngennych chi • Parhauigydweithio? • Next steps • Writing a document on the basis of the presentation, the research that lies behind it and your input • Continuing to collaborate?