Changing Tests for Teachers: English Language School-Based Assessment in Hong Kong

1. University of Hong Kong Faculty of Education Inaugural Seminar Series English Language School-Based Assessment: Integrating Theory and Practice, Dec 19th 2005 Changing Tests for Teachers: English Language School-Based Assessment in Hong Kong Chris Davison Faculty of Education, University of Hong Kong

2. Curriculum and assessment reform in Hong Kong • The Hong Kong Examinations and Assessment Authority (HKEAA) has recently moved from norm-referenced to standards-referenced assessment, including the incorporation of a substantial school-based summative oral assessment component into the compulsory English language subject in the Hong Kong Certificate of Education Examination (HKCEE).

3. Curriculum and assessment reform in Hong Kong The initiative aims • to align assessment more closely with current English Language teaching syllabus (Curriculum Development Council, 1999) and the new outcomes-based Senior Secondary curriculum. • to assess learners’ achievement in areas not easily assessed by public examinations, in particular speaking and extensive reading. • enhance student self-evaluation and life-long learning.

4. The SBA initiative • Starting in S.4, teachers assess their own students’ oral English language competencies through a range of authentic classroom-embedded activities over 2 years. • Initiative developed by a team of researchers at the Faculty of Education, University of Hong Kong, in collaboration with HKEAA. • Part of an evolving but coherent and systematic programme of research and evaluation, professional development and system-wide support

5. The SBA initiative • SBA is integrated into the teaching and learning process, with teachers involved at all stages of the assessment cycle:  planning the assessment programme identifying and developing appropriate assessment tasks  overseeing the assessment process  making final judgments

6. The SBA initiative But previous studies have found … • Changes to summative assessment did not automatically lead to improvement in learning (Andrews 1994; Cheng 1998; Andrews, Fullilove & Wong 2002). • Assessment innovation was severely constrained by traditional school culture and by teacher, parental and student expectations (Cheung & Ng 2000; Carless 2001; Adamson & Davison 2003). • Wide variation in teachers’ interpretations of student performance and of their role in the assessment process (Yung 2001).

7. The SBA content and structure • Aims • Content • Structure • Assessment criteria • Making judgments • Standardization • Safeguards

8. SBA as research • The SBA initiative has stimulated multi-level and multi-purpose research efforts from a growing research team with more than 12 interrelated research projects now underway, or under review. • In addition, the research efforts have attracted the interest of a growing number of MPhil and PhD students, as well as the attention of the international assessment and English language education research community.

9. SBA as research The particular characteristics of this research that makes it so interesting: • its significance to the educational community • its high-impact and visibility • its concern to address both fundamental theoretical problems in language assessment (basic research) and the needs of school communities (applied research) • its action-oriented collaborative approach

10. SBA as research • Many concerns and issues about SBA have been systematically gathered from key stakeholder groups during the process of SBA development, implementation and evaluation. • Data was gathered over a year via questionnaires, individual and focus interviews, classroom observation and stimulated recall, and public briefing sessions as well as from the trialling of the tasks and assessment processes, and the training workshops, in total involving more than 1800 teachers and over 600 schools.

11. SBA as research Adapting Brindley’s (1995) taxonomy, these concerns and issues can be divided into three types: • Sociocultural (political) • Theoretical (technical) • Practical

12. Socio-cultural issues Still a traditional testing culture in Hong Kong: • Assessment practices are still primarily oriented towards providing data to select students for education or employment (Biggs 1995). • External exam results are still the dominant way schools, students and teachers are evaluated and held accountable. • Exam preparation is the traditional role of assessment in the senior secondary classroom (cf. classroom-based assessment).

13. Socio-cultural issues • Traditional cultural practices exemplified in a study of writing assessment in senior secondary English (Davison & Tang, 2003): “Many teachers reported they cannot ‘assess’, only ‘mark’. They feel unable to make a difference in teaching and learning, to respond to individual needs, because of community expectations of convergence and commonality. Teachers feel their assessment process are expected to change, without fundamental purposes being explicitly challenged. Such role conflict results in increasing stress and a decline in perceived teacher expertise.”

14. Socio-cultural issues • However, the official adoption of the UK Assessment Reform Group’s (1998) distinction between assessment for learning, and assessment of learning has stimulated the beginnings of a major paradigm shift • Hong Kong schools are moving from a culture of testing to a learning and assessment culture (Hamp- Lyons 1999; in press).

15. Socio-cultural issues • Some teachers in the SBA study perceived cultural differences as a major stumbling block to assessment reform: “I feel it takes time because the culture. The education culture in Hong Kong is different from other western countries and the students may not used to that kind of assessment. They like to do exam paper, they think they have something to learn.”

16. Socio-cultural issues • Another teacher newly returned from overseas, commented: “Actually I was so surprised that ... How slow we Hong Kong people are in terms of education… because I remembered when I was in Canada, we never ... You would never be graded on just one exam. It’s quite like what we are trying to do actually, I believe that (assessment for learning) has been practised in those places for years and I was actually surprised nobody did anything (here)… I am totally for assessment for learning.”

17. Socio-cultural issues Research shows need for much more understanding, even at the system level, of … • the dual role of the teacher as facilitator and assessor • the concept of a student being assessed against criteria (rather than other students) • the need for teacher consultation and interaction as part of the standardization process.

18. Socio-cultural issues • The schools which were already doing extensive reading and whose students engaged in oral group work and individual presentations on a regular basis found it the easiest to integrate the assessment tasks into their existing practice: “I just briefly tell the students about the task because it is in mid May, so they were quite busy that moment. So I asked them just make use of what they have been doing, say they just, they can just took from ERS report and work on it, say prepare a better review so that they can just have their presentation based on the review and I told them …

19. Socio-cultural issues • …That’s what I did at the very beginning. Later on, I, I met them some days later, and I asked them to show me the book review they had written and I took a look at it and I found that there weren’t any major problems in it. So I just returned them the review and they started to prepare those tasks … and later on, just right before they did the presentation, I helped them with the vocabulary and the names because they didn’t know how to pronounce them. So I just helped them pronounce them correctly… (but) I gave them more guidance according to the SBA documents … because the five questions listed there (see Appendix II) suggested some sort of high order thinking skills … So I try to scaffold them to think in that way.”

20. Socio-cultural issues • Some teachers took longer to come to grips with the principles involved in SBA, and their implications for teaching and learning as well as for assessment practice: “For students of higher forms, the time (8 min) is quite limited. They can’t have enough time to introduce their books and ask each other questions.” … but no time limits actually prescribed in SBA.

21. Socio-cultural issues ‘Fairness’ was – and is - a major concern. • Fairness is fundamentally a sociocultural, rather than a technical, issue, “a justice that goes beyond acting in agreed upon ways and seeks to look at justice of the arrangements leading up to and resulting from those actions.” (Stobart, 2005, p. 1). • “Fair assessment cannot be considered in isolation from both the curriculum and the educational opportunities of the students … some similarities to the more traditional notion of ‘fairness’ embodied in the classical examinations for the Chinese civil service—the concept that conditions should be consciously created to make opportunities open to all” (Hamp-Lyons, 2005).

22. Socio-cultural issues

23. Socio-cultural issues • Preliminary findings suggest that these different approaches to fairness can be reconciled to some extent by providing teachers with opportunity to tailor classroom-based assessments to the needs of their students, according to commonly-agreed processes, outcomes and standards, with teacher assumptions about students and their oral language levels being made explicit through collaborative sharing and discussion with other teachers.

24. Theoretical issues At the theoretical level concerns with SBA revolve around the understanding and interpretation of traditional concepts such as reliability, validity and authenticity, eg. • The students will memorize everything. • Some teachers will tell students the task ahead of time • Teachers will favour their best students. • Panel chairs will make all teacher agree with them. • Schools will make up results. • Tutorial schools will coach students. • The HKEAA should take up all the ‘scripts’ to check the accuracy of the marks.

25. Theoretical issues “The traditional positivist position on language testing, with the tendency to map the standard psychometric criteria of reliability and validity onto the classroom assessment procedures, has been called into question, and the scope of validity has been significantly broadened (Chapelle 1999; Lynch 2001, 2003; McNamara 2001) and taken further by a number of researchers.” (Rea-Dickins, in press)

26. Theoretical issues Defining characteristics of school-based assessment (Stiggins & Conklin 1992; Black & Wiliam 1998; Brookhart 2003): • Teacher-mediated • Co-constructed and dialogic • Context-dependent • Multiple and varied • Dynamic and evolving

27. Theoretical issues “Such an approach can be seen as constructive and enabling because of its focus on assessing the process of learning, its attempt to elicit elaborated performance, and its emphasis on collaborative activity …” Rea-Dickins (in press)

28. Theoretical issues However, if there is no reinterpretation of traditional conceptualizations of reliability and validity, SBA may be reduced to • a series of summative mini-achievement tests external to the teaching and learning programme. • the assessment of rehearsed monologues or dialogues with little or no opportunity for authentic language use. • competition rather than collaboration

29. Theoretical issues • Cf. Clapham (2000, p. 21) Traditional test criteria do apply to alternative assessment: “A problem with methods of alternative assessment, however, lies with their validity and reliability: Tasks are often not tried out to see whether they produce the desired linguistic information; marking criteria are not investigated to see whether they ‘work’; and raters are often not trained to give consistent marks.”

30. Preliminary findings suggest … Potentially much greater validity with SBA than external oral paper - More natural and authentic tasks - Students have a genuine reason to communicate as have read different texts - Students in comfortable, familiar environment-- Class teacher already familiar with the range of student performance, can ask questions to ensure the text is student’s own ‘work’ little possibility of cheating Theoretical issues

31. Potentially much greater reliability with SBA than external oral paper - A series of assessments (instead of one) by a teacher who is familiar with the students - Exemplars of student ‘performances’ can provide a strong basis for discussing and internalizing the set standards. - Multiple opportunities for assessor reflection and standardization. - Disagreement between teachers the key to establishing trust (and trustworthiness). Theoretical issues

32. But initially lots of misconceptions … Superficial tasks A lot of over-rehearsal and memorization Use of buzzers and rigid timelines Over-reliance on scripts and formula An emphasis on performance and recording Overbearing or stressed teachers Unhappy students Theoretical issues

33. Practical issues • Not surprisingly, many practical concerns: - Access to appropriate assessment and extensive reading) resources - Access to activities and techniques, models/resources - Access to technical resources/expertise - Lack of recognition/support at the school-level - Concerns about student and parental expectations - (In)adequacy of training - Lack of time - Competing priorities

34. Practical issues • Teachers also expressed a need to understand much better the underlying assumptions of SBA, how to modify their teaching and learning, and how to set up effective assessment tasks: “Many teachers have an urgent need to view others’ practices and share experiences ... We can film the good lessons for teachers and analyze the lessons. We (need to) focus on teaching instead of assessment only.”

35. Practical issues  More training will be provided to all teachers teaching S4 English between January and April so that they can be supported during the whole of the assessment period in second semester. SBA handbook, introductory DVD and booklet are available, on-line and face to face support.

36. Practical issues Preliminary findings suggest … • Much extra time and effort required during the first year(s) of implementation as SBA is still new to both teachers and students, but once SBA a routine part of classroom activities, should be no significant increase in student/teacher workload? • SBA changes nature of what is done rather than how much is done; the key is to integrate assessment, teaching and learning?

37. Positive impact on teachers, learners and schools • Despite many concerns, the attitudes towards SBA by teachers, students and school communities are generally very positive: “Personally, I enjoyed this trialing experience. I learnt how to judge the students through this activity. Moreover, my students tried to do the presentation based on the guiding questions given to them. Students found this presentation quite interesting and motivating. They learnt how to speak confidently and bravely during this assessment activity. They found this presentation rewarding since they can learn not only from the book but also through their actual participating experience.”

38. Positive impact on teachers, learners and schools • Those teachers involved now much more aware of values and principles underlying SBA, and options now available to them: “After my students … had finished the presentation, I give them feedback and try to improve their performance and … I was really amazed by the response … I thought that one of my students actually is not very good in English, but after the presentation she tried very hard to do it again and again. Feedback really works and I found that my students have improved a lot.”

39. Positive impact on teachers, learners and schools • Teachers commented on increase in student confidence and positive effect of assessment activity on other language skills: M: I think my student Sandy, just the one, just videotaped. She has great improvement… She thinks that is useful, very useful and she told me that … it also helps her reading. I think the student can really improve a lot after this trialing. C: Well, my student has shown a great improvement in terms of confidence and English proficiency. They like talking to each other in English and they are not being afraid of being videotaped.

40. Positive impact on teachers, learners and schools A:Yeah, they start, they start reading English books as well. S: I think is just like what you said, is very good training for confidence and the students actually articulate what they read and … they found reading very useful, purposeful, that is something that you can share with someone, is not something just happening in your inner self. So I think the one that I trained out of the two, one girl and she does quite well even in her writing. So she did well in this writing exam. I’m not sure if it is the effect of that training experience (but) it seems that she gained some confidence during this period of time.

41. Positive impact on teachers, learners and schools • Students also commented favorably on the assessment activities: T: What do you think about the assessment task you did in the presentation task 5? S1: It was quite interesting that we need to think about what the character needs. We can buy a gift. S2: I just think it’s easy to handle it. T: Why? S2: It’s quite interesting to think for a gift to the character. T: When you are thinking of a gift is it difficult?

42. Positive impact on teachers, learners and schools S2: I don’t think so because I can think of many, many gifts to solve the problem. T: Did you enjoy working with your partners? S2: Yes. I did because my partners are all my best friends. We didn’t have any gaps so we did the project perfect. T: What about you? S3: I also very enjoy doing the task with my friends as they know me very much. When I don’t know what can I say they will help me to continue the conversation. S4: With the partner I won’t feel nervous.

43. Positive impact on teachers, learners and schools • Students can comment on own development and receive constructive feedback immediately after assessment, hence improving learning. • Students encouraged to work consistently. • Teachers and students become partners in the assessment process • Collaboration and sharing of expertise take place within and across schools. • Teachers build knowledge and skills in alternative assessment, readily transferable to other areas of English language curriculum, and beyond.

44. Conclusions • The SBA initiative, is a major educational reform, entailing significant changes in school culture and structures as well as in pedagogic expectations among students, teachers, administrators, parents and the wider community. • The SBA initiative requires the development of content- and context-appropriate assessment activities, instruments and procedures which are explicitly linked to high-quality teaching and learning.

45. Conclusions • The SBA initiative requires English language teachers who are not only confident and skilled at making highly-contextualised, consistent and trustworthy assessment decisions, but also effective at involving students in the assessment process. • The SBA initiative requires high levels of teacher collaboration, leadership and support within and across schools. SO … much research and (informed) practice is needed to help make this initiative a success.

46. Assessment for learning in Hong Kong http://cd.emb.gov.hk/basic_guide/BEGuideeng0821/chapter05.html “Based on the beliefs that every student is unique and possesses the ability to learn, and that we should develop their multiple intelligences and potentials … there should be a change in assessment practices and schools should put more emphasis on 'Assessment for Learning' as an integral part of the learning, teaching and assessment cycle …

47. … In other words, teachers should use assessments (e.g. as simple as effective verbal questioning, observation of student behaviour) and provide immediate feedback to enhance student learning in everyday classroom lessons. The focus is on why they do not learn well and how to help them to improve rather than just to use assessments to find out what knowledge students have learned”

48. Selected References • Adamson, B. & Davison, C. Innovation in English language teaching in Hong Kong primary schools: One step forwards, two steps sideways. Prospect, 18 (1), 27-41. • Andrews, S. 1994. The washback effect of examinations: Its impact upon curriculum innovation in English language teaching. Curriculum Forum, 4 Carless, D., (2005). Prospects for the implementation of assessment for learning. Assessment in Education, 12(1), 39-54. • Chapelle, C. (1999). Validity in language assessment. Annual Review of Applied Linguistics, 19, 254-272. • Cheng, L.Y. 1998. Does washback influence teaching? Implications for Hong Kong. Language and Education 11 (1), 38-54.

49. Selected References • Davison, C. & Tang, R. (2003, March). Assessing in the swamp: Formative assessment in Hong Kong secondary school English. Paper presented at the American Association for Applied Linguistics, Arlington, VA, USA. • Hamp-Lyons, L. (in press). The impact of testing practices on teaching: Ideologies and alternatives. Classroom-based assessment: Possibilities and pitfalls. In Cummins, J. & Davison, C. (Eds) The International Handbook of English language teaching, (Vol. 1), Norwell, MA: Springer. • Lynch, B. (2003). Language assessment and programme evaluation. Edinburgh: Edinburgh University Press. • McNamara, T. (2001). Language assessment as social practice: challenges for research. Language Testing, 18(4), 333-349. • Rea-Dickins, P. (in press). Classroom-based assessment: Possibilities and pitfalls. In Cummins, J. & Davison, C. (Eds) The International Handbook of English language teaching, (Vol. 1), Norwell, MA: Springer. • SBA Consultancy Team (2005). 2007 HKCE English examination: Introduction to the school –based assessment component. Hong Kong: HKEAA.