The whys and wherefores of data in the evaluation of language revitalisation efforts

1. The whys and wherefores of data in the evaluation of language revitalisation efforts Hywel Jones BSc (Hons), DMS, CStat, CSci 26 February 2019 http://statiaith.com/

2. Whys? • ‘Trends experienced in youth work in Wales reflect other areas of public policy, including an emphasis on evidence-based policy and a greater reliance on quantitative indicators to measure impact.’ • http://cwps.aber.ac.uk/wp-content/uploads/2018/07/Evaluation-Impact-and-Outcomes-What-does-it-mean-for-us-Final-Report-1.pdf

3. Wherefores? • ‘5. Recommendations • 5.1 Data to be collected by organisations working directly with young people should, where possible, include: • 5.1.1 Statistical data (e.g. related to participation, membership, attendance, awards, demographics, finances including leverage, etc) • 5.1.2 Skills and competence data for young people … • 5.1.3 Youth worker data (e.g. participation, demographics, qualifications and training, etc) • 5.1.4 Achievement of the organisation’s mission, strategic and operational plans. • 5.1.5 Achievement of external stakeholder objectives.’ • http://www.cwvys.org.uk/wp-content/uploads/2014/06/Towards-an-Outcomes-and-Impact-Approach-for-the-Youth-Sector-in-Wales.pdf

4. 6.6 Given lack of capacity and consistency issues facing some organisations within the sector to deal with data analysis, would some organisations benefit from further guidance e.g. …,sample sizes, frequency, ….? • http://cwps.aber.ac.uk/wp-content/uploads/2018/07/Evaluation-Impact-and-Outcomes-What-does-it-mean-for-us-Final-Report-1.pdf

5. A living language: a language for living - Welsh language strategy 2012 to 2017: Evaluation Framework • (p.9) ‘It is important that organisations providing data are clear about their role, about the purpose of the data, and about its quality and use. It is also important that they are supported adequately in this respect.’ • 130225-welsh-language-strategy-evaluation-framework-en.pdf https://gov.wales/statistics-and-research/welsh-language-strategy-evaluation/?lang=en

6. Consider whether the indicators which follow relate to: • Inputs • Activities • Outputs • Outcomes • Impact

7. Are the indicators?: • Precise/well-defined • Reliable • Valid • Measurable • Practicable

8. http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/EUpresentation.pdfhttp://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/EUpresentation.pdf

9. Strategic Area 2 - Children and Young People 130321-welsh-language-strategy-measuring-indicators-en.pdf 21/3/2013 https://core.ac.uk/display/18491370

10. https://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdfhttps://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdf

11. https://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdfhttps://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdf

12. Strategic Area 3 – The CommunityThe number of people attending Welsh language events, including thoseorganised by recipients of Welsh Government grants

13. https://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdfhttps://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdf

14. https://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdfhttps://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdf

15. https://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdfhttps://gov.wales/docs/dcells/publications/180329-welsh-language-strategy-annual-report-2016-17-en.pdf

16. Data collected by organisations in receipt of Welsh Government grant funding Scale and reach of interventions to promote the use of Welsh are sufficient to bring about meaningful and lasting change

17. de Luna, I., Suberbiola, P. (2008). Measuring student language use in the school context, Teaching through Basque. (ed.J. Cenoz) Multilingual Matters, Bristol.

19. http://statiaith.com Pecyn Mesur Canlyniadau – Fersiwn 1 – Tefynol.pdf

20. ‘Complex social systems of the sort described above are amenable to the techniques of System Dynamics, a set of techniques which evolved in response to the need to understand and manage complex systems (Forrester, 1987). System Dynamics can help explain how outcomes emerge in diverse systems including business scenarios, biological systems, and social policy.’ • https://understandingsociety.blogspot.com/2016/02/causal-diagrams-and-causal-mechanisms.html • Example of use: • JOHN WYBURN & JOHN HAYWARD (2008) The Future of Bilingualism: An Application of the Baggs and Freedman Model, The Journal of Mathematical Sociology, 32:4, 267-284, DOI: 10.1080/00222500802352634

21. ‘Causal mechanisms theory is generally associated with a realist approach to the social world, and generally rejects the Humean theory of causation. The causal diagram approach, by contrast, is premised on the Humean and statistical approach to causation.  A causal mechanisms hypothesis is not fundamentally evaluated in terms of the statistical relationships among a set of variables; whereas a standard causal model is wholly intertwined with the mathematics of conditional correlation.’ • https://understandingsociety.blogspot.com/2016/02/causal-diagrams-and-causal-mechanisms.html

22. Directed Acyclic Graph (DAG) http://dagitty.net/mX8oOKb http://statiaith.com

23. Research • Young People’s Social Networks and Language Use (2006) Research project commissioned by the Welsh Language Board from the Centre for European Research (Wales) and CwmniIaith. 2005 • https://www.webarchive.org.uk/wayback/archive/20120328031232mp_/http://www.byig-wlb.org.uk/English/publications/Publications/3562.pdf

24. Siartrau Iaith / Language charters • Exploring bilinguals’ social use of language inside and out of the minority language classroom • Enlli Môn Thomas & Dylan Bryn Roberts • https://doi.org/10.1080/09500782.2010.544743

25. https://www.gwynedd.llyw.cymru/cy/Cyngor/Dogfennau-Cyngor/Strategaethau-a-pholisiau/Cynllun-iaith/Cynyddu-defnydd-or-Gymraeg-Ysgolion-Uwchradd.pdfhttps://www.gwynedd.llyw.cymru/cy/Cyngor/Dogfennau-Cyngor/Strategaethau-a-pholisiau/Cynllun-iaith/Cynyddu-defnydd-or-Gymraeg-Ysgolion-Uwchradd.pdf

29. Evaluation of the Welsh Language Strategy • Welsh Language Transmission and Use in Families Research into Conditions Influencing Welsh Language Transmission and Use in Families (2017) https://gov.wales/statistics-and-research/welsh-language-transmission-use-in-families/?lang=en • Welsh Language Use in the Community (2015) • https://gov.wales/docs/caecd/research/2015/151007-welsh-language-use-community-research-study-en.pdf

31. Language use surveys

32. https://gov.wales/docs/caecd/research/2014/140429-welsh-language-questionnaire-youth-2014-15a-en.pdfhttps://gov.wales/docs/caecd/research/2014/140429-welsh-language-questionnaire-youth-2014-15a-en.pdf

34. Welsh Language Use Survey, 2013-15: Use of the Welsh language https://statswales.gov.wales/Download/File?fileId=507

36. Confidence intervals: calculation • http://profrobertnewcomberesources.yolasite.com/resources/chcyfrannedd.xls (Welsh) • http://profrobertnewcomberesources.yolasite.com/resources/ciproportion.xls(English) • http://profrobertnewcomberesources.yolasite.com/resources/icproporcion.xls(Spanish)

37. Evaluation guidance • https://www.betterevaluation.org/en/rainbow_framework/describe • http://www.data.cymru/eng/evaluation

38. Other reference • Unintended Behavioural Consequences of Publishing Performance Data: Is More Always Better? • http://ci-journal.net/index.php/ciej/article/view/863 • http://ci-journal.net/index.php/ciej/article/view/863/947

39. Contact details @statiaith http://statiaith.com