Positioning evaluation as responsive to influences and as being influential

1. Positioning evaluation as responsive to influences and as being influential Cheryl Ballantyne NSW Department of Education and Communities Western Sydney Region Cheryl.ballantyne@det.nsw.edu.au Australasian Evaluation Society International Conference 29 August – 2 September 2011 Hilton, Sydney, Australia

2. Aims of presentation To explore the notion of evaluation as an intervention by: • Describing a current evaluation in progress • Identifying influences to date • Outlining future directions for the evaluation

3. Theoretical context • Evaluation as an intervention has an end goal of social betterment (Henry, 2005; Henry and Mark, 2003) • Evaluation as an intervention has potential to influence at individual, interpersonal and collective levels (Henry and Mark, 2003) • Integrated theory of evaluation influence incorporates three dimensions, source of influence, intention and time frame (Kirkhart, 2000)

4. Key claim • Evaluation, when incorporated as a regular intervention within program implementation can increase the program’s effectiveness and its responsiveness to contextual influences.

5. Structure • Description of program • What have been the contextual features that have influenced the program development and implementation? • What influences has the evaluation had to date?

6. Structure 4. Why has this evaluation had these influences? 5. How was the evaluation data generated? 6. What are the implications for developing future evaluation instruments?

7. 1. Western Sydney Ningbo Chinese Volunteer Teacher-researcher Program • Partnership involving DEC Western Sydney Region, Ningbo Municipal Education Bureau, PRC, and University of Western Sydney

8. Program aim • To promote the teaching and learning of Chinese language and culture in WSR schools by generating interest in schools and embedding that interest through a sustained program of support • Long term – development of students’ intercultural competence in Chinese

9. Program features 2008-2012 • 18 participating schools, 10 primary and 8 high schools selected by expression of interest, majority of students are non-background speakers of Chinese • Up to 10 graduates from Ningbo, China arrive at the end of June each year • Volunteer in schools two days per week to support the teaching and learning of Chinese language, culture and society • Additional human resource to participating schools equal to approximately $25,0000 per year

10. Program features 2008-2012 • Volunteers complete Master of Education (Honours) at University of Western Sydney • Research informs and is informed by school experience • Return to China in December the following year

11. 2. Contextual features that have influenced program development and evaluation • Globalisation • Expansion of China in the world • Approx 1.2 billion English speakers in China • Approx 700 billion Chinese speakers in other countries • Implications for students’ future communities and employment

12. Contextual features that have influenced program development and evaluation • Commonwealth Government commitment to increasing Asia literacy of Australian students • 2020 target • National Curriculum • Languages learning area • Mandarin Chinese to be one of two target languages to be taught in NSW • Across curriculum dimension – Asia and Australia’s engagement with Asia

13. Contextual features that have influenced program development and evaluation • NSW DEC memorandum of understanding with China • NSW Confucius Institute in partnership with China • Confucius classroom in Western Sydney Region

14. Contextual features that have influenced program development and evaluation • Western Sydney Region memorandum of understanding with Ningbo Municipal Education Bureau • annual cultural exchange visits • sister schools

15. Evaluation response • 2010 – 2012 evaluation of Volunteer Teacher-researcher Program to inform ongoing program development and future directions • Focus question: What can a Chinese native-speaker volunteer program contribute to the teaching and learning of Chinese language and culture in schools?

16. Analysis of Evaluation influences: • Individual • Interpersonal • Collective (Henry and Mark, 2003)

17. 3. What influences has the evaluation had to date? Individual • Increased the importance individual principals and regional staff attribute to the Program • Renewed individual principals’ commitment to the Program • Shaped Regional leaders’ attitudes towards importance of Chinese language learning and the partnership with Ningbo

18. What influences has the evaluation had to date? Interpersonal • Principals share ideas about their Chinese programs with one another • Principals emphasise importance of Chinese programs to staff and community

19. What influences has the evaluation had to date? Interpersonal • Additional schools have expressed interest in participating in the Program • Inquiries about/interest in the Program from interstate and national levels • Meeting of partners to be held in September 2011 to discuss extension of partnership

20. What influences has the evaluation had to date? Collective • Signing of new Memorandum of Understanding planned for late September 2011 • Increased allocation of funding to program • Regional planning to target outreach programs delivered through Confucius Classroom to high school learners • Planning of professional learning to focus on areas for further program development

21. Evaluation intervention and social betterment • In 21st century Australia, Asia literacy and development of intercultural competence in Chinese has potential to lead to social betterment (increase students’ world view, improve employment prospects). • Evaluation has identified student, parent and teacher interest in Asia literacy through Chinese • Evaluation has revealed that students, parents and teachers regard Asia literacy through Chinese as important • Evidence has influenced further evaluation strategies, to enable a deeper analysis of influence

22. 4. Why has this evaluation had these influences? (evidence)

23. Why has this evaluation had these influences? (evidence)

24. Why has this evaluation had these influences? Primary school surveys • 702 student responses from eight schools • 67% - learning Chinese language important • 80% - learning Chinese language interesting • 63% - learning about China important • 78% - learning about China interesting • 65% would like to continue learning Chinese in 2011

25. Why has this evaluation had these influences? Primary school parent surveys • 117 parent responses from seven schools • More than 80% - learning Chinese language and culture important and valuable for students

26. Why has this evaluation had these influences? High school surveys • 245 student responses from five schools • 178 student responses from one school only • 10 parent responses all from one school

27. Why has this evaluation had these influences? (evidence)

28. Why has this evaluation had these influences? Staff surveys • 50 staff responses representing 12 schools 38 from 6 primary schools 12 from 6 high schools • 95% - learning Chinese language important • 100% - learning Chinese culture important • 93% - Chinese Program has a positive influence on schools

29. Why has this evaluation had these influences? Staff surveys

30. Why has this evaluation had these influences? Principal focus groups • Volunteers’ rapport with students has stimulated interest in Chinese language • Chinese language is accepted as part of the curriculum (no longer exotic – High School Principal) • Planning for Chinese language and culture programs occurring in primary schools and high schools

31. Why has this evaluation had these influences? Principal focus groups • Chinese language has replaced other language programs - two high schools • Links between Chinese language and culture and other learning areas - three high schools • Chinese classroom allocated - one high school

32. Why has this evaluation had these influences? Principal focus groups • Collaboration across learning communities to support transition Year 6 to 7. • System support important to the long term effectiveness of school programs

33. 5. How was the evaluation data generated • DEC annual census information on languages programs • Focus group of Principals term 1 2010 • Surveys of students, parents and staff distributed to 18 participating schools term 3 2010

34. Evaluation influence: source, intention, time frame • Source – process (focus groups) and reporting of evidence (results) • Intention – intended (focus groups, reporting of evidence) • Time frame – immediate, evaluation embedded in program implementation (Kirkhart, 2000)

35. 6. What are the implications for future evaluation instruments? Future directions • Student self-assessment of intercultural competence to enable deeper analysis of Program’s impact on student learning • Parent and teacher assessment of student intercultural competence to enable deeper analysis of impact on student learning

36. 6. What are the implications for future evaluation instruments? Future directions • School self-evaluation (based on program standards) to enable deeper analysis of school programs and influence school planning • Document analysis of school plans, policies and annual school reports to analyse extent to which Chinese programs are valued, planned for and embedded in whole school operation

37. Conclusion • This evaluation (in early stages) has had influence at school, regional, state, interstate and national levels • Further planning of evaluation instruments is intended to deepen influence • This evaluation, as an intervention embedded in program planning and implementation, has potential to make a difference for Western Sydney students

38. Thank you from Cheryl Ballantyne for considering - Positioning evaluation as responsive to influences and as being influential Australasian Evaluation Society International Conference 29 August – 2 September 2011 Hilton, Sydney, Australia