International Language in Education Conference, Hong Kong, Dec 2004

1. International Language in Education Conference, Hong Kong, Dec 2004 Changing cultures, enculturating change: School-based English language assessment in Hong Kong Chris Davison Faculty of Education, University of Hong Kong

2. Changing cultures: From a testing to an assessment culture “Assessment is the practice of collecting evidence of student learning in terms of knowledge, skills, values and attitudes through observation of student behaviour when carrying out tasks, tests, examinations, etc”. (Curriculum Development Council, 2000)

3. Changing cultures: From a testing to an assessment culture • Assessment practices in Hong Kong until very recently focused on providing data to select students for education or employment (Biggs, 1995). • External set and assessed examination results are still used for accountability, but recent adoption of value-added indexes to compare schools. • Traditionally main role of assessment in schools (vs. school-based assessment) = exam practice, Primary 1-Secondary 7.

4. Changing cultures: From a testing to an assessment culture • Wide perception that teacher feedback to students is often fairly limited, tends to be deficit-oriented, exacerbated by • Assumption that motivation = effort, not ability • Limited and vague published assessment criteria eg. Form 7 Use of English writing examination uses a scaled double-impression marking scheme divided into content, language and organization, no qualitative criteria (Curriculum Development Council, 1999).

5. Changing cultures: From a testing to an assessment culture • Currently major educational reforms underway, with school-based and standards-referenced assessment being introduced as way to improve learning and teaching (Education Commission, 2000; CDC, 2000; HKEAA 2003). • Official adoption of UK Assessment Reform Group’s distinction between assessment for learning vs assessment of learning

6. Changing cultures: From a testing to an assessment culture http://cd.emb.gov.hk/basicguide/BEGuideeng0821/chapter05.html “Based on the beliefs that every student is unique and possesses the ability to learn, and that we should develop their multiple intelligences and potentials … there should be a change in assessment practices and schools should put more emphasis on 'Assessment for Learning' as an integral part of the learning, teaching and assessment cycle …

7. Changing cultures: From a testing to an assessment culture … In other words, teachers should use assessments (e.g. as simple as effective verbal questioning, observation of student behaviour) and provide immediate feedback to enhance student learning in everyday classroom lessons. The focus is on why they do not learn well and how to help them to improve rather than just to use assessments to find out what knowledge students have learned”

8. Changing cultures: From a testing to an assessment culture • This major shift from a "culture of testing" to a "culture of assessment" (Gipps, 1994) is being supported by the development of standards-referenced assessment frameworks and "authentic" alternative school-based assessment activities/tasks embedded within the instructional program. • Such assessment schemes seen as complementing the recent adoption of a more outcomes-oriented English syllabus.

9. Changing cultures: From a testing to an assessment culture Range of school-based assessments: • Formal to informal, eg. peer testing, portfolios, reading logs, structured observation • Collaborative (peer/collegial) to individual (self/teacher) • Process to product eg. process writing, reading conferences • Graded to ungraded (but always criterion-referenced and explicit) • Formative to summative

10. Changing cultures: From a testing to an assessment culture In addition, from 2005-07, introduction of substantial school-based assessment as an integral and innovative component of public examinations: ie. 20% summative school-based assessment for Form 5 English examinations: group and individual oral assessments, based on an extensive reading programme, including print and non-print texts.

11. Changing cultures: From a testing to an assessment culture • Assessment reform seen as impacting on whole school community, including parents and tutors/tutorial schools. “Fundamental changes in school assessment practices need to be planned, discussed, shared, negotiated and agreed by all teachers in each school. A corresponding assessment policy and mechanisms to bring it about need to be worked out at whole-school and classroom levels … parents (need to be informed) of the rationale underpinning the change in assessment practices…”

13. Changing cultures: From a testing to an assessment culture However, still much uncertainty and disagreement around school-based assessment in “old(er)” assessment cultures, eg. Educational Measurement: Issues and Practice, 22 (4), Winter, 2003. So…what issues and concerns arise in implementing school–based assessment as official government policy in Hong Kong (in a traditional examination-dominated culture)?

14. Changing cultures: Teachers’ perspectives Example 1: Current concerns of teachers and principals from 30 primary and secondary schools about school’s assessment policy and practice (Berry, 2003) • Found top four most frequently used assessment activities were exercises, demonstrations, oral questioning, and revision. • Few used portfolios, journals, case studies, peer assessment and presentations. “The forms of assessment most of the teachers used tend to encourage passive learning … teachers did not bring what they believed about assessment to their classroom.”

15. Changing cultures: Teachers’ perspectives Example 2: Current concerns of senior secondary English teachers with formative assessment of writing (Davison & Tang, 2000, 2002) • Need better marking schemes to manage workload/guide feedback eg. “it would be better is we can sometimes focus mark some of the writing assignments”, “a more effective and student-friendly system of ‘symbol marking’”, “a balanced marking scheme and not having every single mistake corrected”, “focusing on a major area may be more effective”…

16. Changing cultures: Teachers’ perspectives • Need more time eg. “more time to enjoy the work produced”, “time to read and digest students work first and understand their line of thought and structures” • Need more professional dialogue/peer interaction eg. “a sharing session whereby we can learn from each other”, “share with colleagues”, “peers to help marking”.

17. Changing cultures: Teachers’ perspectives • Need more trust, less surveillance to allow innovation eg. “the school will take up the books and check on the effort that we put in marking students assignments”, “it will check whether a teacher has completed a fixed number of assignments and whether he or she has marked each one of them intensively”, “….whether the teacher has done his job properly in terms of quantity or quality”, “…checking and monitoring the progress of teacher’s work in teaching and learning”, “…checking the type of tasks given and the way they are marked and assessed”

18. Changing cultures: Teachers’ perspectives • Need for cultural, not just structural change “I think you need to know whether we believe this is a paper exercise and we are doing it simply to justify their writing in the first place or whether the feedback we give them has a lot of bearing on how they write the next time round. The more I do this, the more I believe this is a justification process, that we mark to show them we have been there and we encourage them [unintelligible] and to give them a grade. …So, my faith in the system is pretty low. And therefore, my faith in how I mark and the devotion I put into marking is very low, and therefore I am very frequently pretty superficial in the way I mark because I don’t believe it’s going to make the slightest bit of difference. I am sorry that it should be the case. I really wish it was going to improve the students’ writing, but it isn’t the case.”

19. Changing cultures: Teachers’ perspectives Example 3: Current concerns of *senior secondary English teachers with introduction of new school-based assessment in 2005-07 (Davison, Andrews, Hamp-Lyons, Hyland & Tang, in progress) (see Appendix, Tables 1, 2, 3) *Volunteers for action research project into improving the effectiveness of school-based assessment

20. Changing cultures: Teachers’ perspectives • Summative assessment still very traditional but signs of change in positive direction. • Teachers agree that the aims of assessment proposed by CDC are very important, with the mean between 4.19 to 5.32 out of 6. • However, they think that their school’s assessment programme is not effective enough in meeting each corresponding aim (each corresponding mean is lower), except ‘to record data about students’ progress for ranking/comparative purposes, including for promotion’. • The teacher respondents think that schools have put too much emphasis on this aim (4.81>4.46)

21. Changing cultures: Teachers’ perspectives Perceived problems with the school’s current arrangements for assessment: “The assessment activities are not well-planned” “Most of the time the assessment is not designed to test what is learnt/taught” “Not enough preparation; too much testing, teaching to the tests” “Students cannot get detailed individual comments about their own performance” “Only marks or grades are given to the students. Teachers don't have time to talk to students individually”

22. Changing cultures: Teachers’ perspectives “Currently, almost all assessment activities are traditional and summative. They can give students some feedback on their progress but not quite effective in helping them to develop” “Formative assessment tends to be neglected by both students and parents as they only believe studying for the exam is the ultimate goal of learning” “Formative assessment is not very effective in every class because the poorest students are not capable to work on their own even with heavy teacher guidance”

23. Changing cultures: Teachers’ perspectives … but also signs of progress “Assessment is quite effective as students are concerned about their results and they have shown improvement in their work” “The current arrangements are effective as the formative components facilitate learners' autonomy and meta-cognitive strategies. The retention of the summative parts serves a stabling factor for the selection purpose” “We give guidelines to assess students' work more fairly and help motivate students' learning interest”

24. Changing cultures: Teachers’ perspectives “Fairness is ensured through marking guidelines. Teachers have discussion on the assessment methods before they are carried out” “Effective in making students realize their strengths and weaknesses, e.g. the rubrics in the book report, the descriptions in the mark scheme for essay writing” “The school makes better use of formative assessments to help students learn to take more responsibility in their own learning and be more active learners”

25. Changing cultures: Teachers’ perspectives Perceived problems with implementing change in assessment at school level: “As a teacher with only two years of teaching experience, I think not much can be done at the school policy level, but I can try out some new methods in my own class provided that they don't interfere too much with the normal curriculum” “It's hard. Many of these policy and practices are old and traditional. Unless I get most of my colleagues to agree to the new change, it will be difficult” “I could raise my ideas but we still have to follow the direction of the panel”

26. Changing cultures: Teachers’ perspectives “It's quite difficult because of the ranking and experience. Experienced teacher can influence it more easily” “The management level of our school should update and equip themselves with policy for whole school assessment and to what extent SBA be incorporated into the curriculum. “The school is pre-dominantly composed of teachers who do not strongly believe in the effectiveness of assessment policy and practice. I'm not a heavy weight in the English panel” “There is not much an individual can do”

27. Changing cultures: Teachers’ perspectives … but also signs of progress “Assessment policy and practice are decided by the English panel chairperson and the Principal. But teachers can express the opinions and suggestions to the panel head and she may take these into consideration” “Individuals can still influence practice by their suitable adaptations on assessments” “To a certain degree, an individual may influence school's assessment policy by innovating his/her teaching so that other colleagues may want to follow if they find assessment can be done successfully in the everyday classroom”

28. Changing cultures: Teachers’ perspectives “I can voice my opinions in panel meetings, the ideas can be discussed and actions may be taken to change existing policy if the principal also agrees with the conclusion that my panel has come up with” “As a panel head, my job is to head members towards the appropriate requirement and practice set by the EMB” “The English Department as a whole has been moving rather fast towards the alternative assessments, whereas some other academic subjects have still been engaging in the traditional assessment modes”

29. Major theoretical, socio-political and practical issues for Hong Kong Theoretical issues … • Need to change the still dominant psychometric paradigm and assumptions of school and system-level processes (Brookhart 2003), eg. summative/formative dichotomy, traditional conceptions of fairness and “reliability”, role of standardization, input, moderation, context, etc • Need to understand teachers’ assessment decision-making processes, their “assessment readiness” and the nature of support required

30. Major theoretical, socio-political and practical issues for Hong Kong Defining characteristics of school-based assessment (vs assessment in the school/teacher assessment) (Stiggins & Conklin 1992; Black & Wiliam 1998; Brookhart 2003): • Teacher-mediated • Multiple and varied functions and formats (observation, inquiry, analysis, test, etc) • Co-constructed and dialogic • Context-dependent • Unstable and evolving

31. TEACHER KNOWLEDGE, BELIEFS AND ATTITUDESeg. Role of education; nature of assessment, teaching and learning; nature of (English) language and language learning, etc EXTERNAL PRESSURES eg. Syllabus and exam requirements, other educational reforms, parental/community expectations, etc SITUATIONAL FACTORS eg. Time, trust, collegial support, student numbers/characteristics, perceived impact of assessment, etc Teachers’ assessment orientation and decision-making processes ASSESSMENT PRACTICES (Davison, 2004, in progress, adapted from McMillan 2003)

32. Major theoretical, socio-political and practical issues for Hong Kong Sociopolitical issues … • Need to establish a more realistic and interactive strategy for change/adequate resources and time to support teachers eg. necessity of teacher moderation and verification not widely understood, little official recognition that deep cultural change takes time, over-reliance on Western “thought packages” (Marton, 2000) for quick solutions • Need to educate and build community trust eg. threats to trustworthiness from tutorial schools, threats to validity from traditional community expectations, etc

33. Major theoretical, socio-political and practical issues for Hong Kong “Although 85 percent of the Primary Six students were positive about the change they had experienced in their dictation lessons, and all commented that they found the new activities less stressful and more interesting and helpful to learning, they were worried about their scores. They were also concerned about their parents’ negative attitudes to any curricular change, demonstrating the power a fairly conservative educational community can have on the implementation of curricular reform” (Chiang 2002, cited in Adamson and Davison, 2003)

34. Major theoretical, socio-political and practical issues for Hong Kong Practical issues … • Need systematic and comprehensive inservice and preservice teacher training in school-based assessment; • Need appropriate assessment resources, activities and techniques as models/resources for textbook–bound teachers • Need structural and collegial support at the school and classroom level

35. Major theoretical, socio-political and practical issues for Hong Kong “At the heart of change for most teachers is the issue of whether it is practical … in the ethic of practicality amongst teachers is a powerful sense of what works and what doesn’t, of which changes will go and which will not – not in the abstract, or even as a general rule , but for this teacher in this context”. (Hargreaves, 1994: 12)

36. HKU school-based assessment project(s) Aims: • To stimulate, support, document and evaluate the development of effective formative and summative assessment practices in English language teaching, with a particular focus on the new SBA activities in HKCE 2005-07. • To identify and describe factors which may facilitate and/or hinder the connection of formative (and summative) assessment and feedback with learning and teaching

37. HKU school-based assessment project(s) • 50 + Form 4 teachers from a range of HK schools • Five university–based researchers (Davison, Andrews, Hamp-Lyons, Hyland & Tang) & five assistant researchers • Small-scale collaborative action research “Action research [rejects] the concept of a two-stage process in which research is carried out first by researchers and then in a separate second stage the knowledge generated from the research is applied by practitioners. Instead, the two processes of research and action are integrated” (Somekh, 1995)

38. HKU school-based assessment project(s) Undertake systematic Identify problem; analysis; identify patterns develop plan for assessment activity Structure observation Try out new assessment activity and evaluation; collect data • Collaborate • (and disseminate) 5. Reflect 2. Plan 4. Observe 3. Act

39. HKU school-based assessment project(s) • Video and audio (self) recordings, questionnaires, interviews, structured reflection • Small manageable focus on different aspects of school-based assessment eg. oral assessment, feedback, reading conferences • Compilation of findings and exemplar material into a report/CD-rom to be disseminated to all schools • QEF and RGC-funded, strongly supported by HKEAA For more information, go to http://web.hku.hk/~sbapro

40. Enculturating change “Political and administrative devices for bringing about educational change usually ignore, misunderstand or override teachers’ own desires for change. Such devices commonly rely on principles of compulsion, constraint and contrivance to get teachers to change … to make teachers more skilled, more knowledgeable and more accountable”. (Hargreaves, 1994: 11)

41. Enculturating change …but there are other ways to enculturate change which take into account the inherently social (and cultural) nature of assessment, by • Reconceptualizing assessment not as a series of measurable individual products, but as a process of dialogue between teachers, learners and the wider school community. • Exploring and discovering how assessment for learning “looks and feels very different” (Leung, 2002) to traditional assessments of learning.

42. Enculturating change • Providing teachers and schools with the opportunity to share evolving assessment beliefs and practices in order to develop a sense of ownership, common understanding of the assessment process, and a more critical but informed perspective on assessment practices. • Recognizing that the problem of teacher interpretation and conflict is an inherent strength, not a weakness, of school-based assessment.