The Changing Climate of Assessment: Addressing the Changing Demands of Assessment Policy

1. The Changing Climate of Assessment:Addressing the Changing Demands of Assessment Policy 21st February 2009Australian Literacy Educators’ Association Val Klenowski Queensland University of Technology val.klenowski@qut.edu.au

2. Changing Climate • Impact of Assessment for Learning • International Comparative Analysis Data • Standards-driven reform • National Curriculum and Assessment • Building communities of assessment learners

3. Literacy in the Context of Assessment • “There has been a shift from a traditional view of literacy as skills, knowledges and cognitions that reside within the individual to a conceptualization of literacy as visible social practices with language, text and discourse.” (Gee, 2003 cited by Hipwell, 2009) • A sociocultural view underpins this perspective adopted in this presentation

4. Literacy Demands in the Context of Assessment • Assessment is the purposeful, systematic and ongoing collection of information as evidence for use in making judgments about student learning. • Through multiple formal and informal assessment opportunities observation, consultation, focused analysis, peer and self-assessment can monitor the learner’s progress.

5. Assessment Purposes • Promote, assist and improve students’ learning • Inform teaching and learning • Provide data that can be communicated to a range of people about the progress and achievements of individual students or groups of students.

6. Formative and Summative Purposes • Assessment of learning equates to summative assessment. Process of summing up or checking what has been learned at the end of a particular stage of learning. • Assessment for learning equates to formative assessment. Assessment that helps students learn. (Weeden, Winter and Broadfoot, 2002)

7. Changing Emphases: Torrance 1997

8. Assessment of and for Learning • Assessment of Learning (AoL) adds procedures or tests to existing work and involves only marking and feedback of grades or marks to students • Assessment for Learning (AfL) is the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there.

9. Assessment for Learning • Is embedded in a view of teaching and learning of which it is an essential part • Involves sharing learning goals with students • Aims to help students know and recognise the standards for which they are aiming • Involves students in self-assessment

10. Assessment for Learning • Provides feedback that leads to students’ recognition of their next steps and how to take them • Is underpinned by confidence that every student can improve • Involves both teacher and students in reviewing and reflecting on assessment data The Assessment Reform Group http://k1.ioe.ac.uk/tlrp/arg/publications.html

11. The Assessment – Learning Relationship • Assessment is not a separate linear function • Teacher – student relationship has a central mediating role - as does the complex interrelationship between teacher and student beliefs about learning, the structure of the tasks and the social and cultural interactions and contexts of the classroom, school and policy environment . (Willis, 2007: 2)

12. The teacher – student relationship in the context of AfL • Sociocultural view of learning is participatory as opposed to an acquisition view of learning • “Students appropriated participatory practices into their traditional cultural narratives of learning, forming ‘entangled’ (Elwood, 2008) learner identities.” (Willis, 2009) • AfL practices seen as both social and cognitive practices that help students learn the norms of a community where the teacher leads the students to both ‘belong’ and ‘become’ skilled learners in a collaborative community. (Willis, 2009)

13. Research Evidence • Wiliam (2007: 36) research reviews of formative assessment reveal that formative assessment raises standards “can effectively double the speed of student learning” • Black and Wiliam’s study (1998) showed that practices were under-developed.

14. Effective Formative Assessment Wiliam (2007) • Activate students as the owners of their own learning (eg have students assess their own work using agreed-on criteria for success) • Encourage students to be instructional resources for one another (eg peer assessment and feedback)

15. Effective Formative Assessment Wiliam (2007) • Clarify and share learning intentions and criteria for success with students (eg share work samples completed by previous students) • Engineer effective classroom discussions, questions, and learning tasks (eg use effective questioning techniques) • Provide feedback that moves learners forward (eg link comments to criteria and standards)

16. Changing Climate of Assessment • Global shift towards standards-driven assessment (A-E reporting) • Accountability context where standards are used as a lever to improve reliability and consistency of teacher judgment • High-stakes assessment e.g. International Tests, National Curriculum and Assessment, NAPLAN Tests

17. Changing Demands of Assessment Policy: International • Politicians and policy makers are increasingly interested in international assessment surveys of educational attainment • Programme for International Student Assessment (PISA) • Trends in Mathematics and Science Study (TIMSS)

18. International Comparative Data • Since we began the program for the assessment of 15 year olds, countries are paying a great deal of attention. And with the release of the third set of data, last year, we can see that while Australia’s maths levels are holding at the same level, we’ve slipped relatively because other countries have moved ahead. So it’s not a stable competition out there. Every country is seeking to do better and to learn from those that are doing better than themselves already. (Australian Labor Party, 2008)

19. International Comparative Analyses • International comparative surveys have influenced policy development • Need for caution, however, in the use of the results from such comparative studies • Politicians keen to introduce particular policies based on these results may overlook identified limitations of such studies

20. International Comparative Analyses: Invalid Uses • Potential for unintended and harmful consequences • Inadequacy of some high-stakes assessments • Lack of sufficient reliability or validity for their intended purposes • Policy officers can be misled by ‘spurious’ increases in assessment results i.e. do not relate to improved learning

21. International Comparative Analyses: Invalid Uses • Students might be placed at increased risk of failure or disengagement from schooling • Teachers may be blamed for inequitable resources which remain beyond their control • Curriculum and teaching can become distorted if high grades per se, rather than learning become the overriding goal

22. International Comparative AnalysesPISA and TIMSS • International measures of educational attainment are used by policy developers • UK post 1996 TIMSS results - National Numeracy Project to address perceived weaknesses in the teaching of Maths, • the National Numeracy Strategy, • the National Literacy Strategy and • KS 3 strategy

23. International Example: England 1988 National Curriculum (NC) • Schools preoccupied with achievement in terms of student results • Evaluation and assessment took on an accountability function • NC identified content of programs, objectives and processes in terms of standards and targets • Publication of league tables device for making judgments about school performance

24. Accountability: Cautions • In the name of efficiency there is a return to technological and behaviouristic refinements of curriculum evaluation and a possible trivialization that threatens the intellectual activity for those involved in the discipline of curriculum evaluation. • Data analyzed for a particular purpose may be used for another unintended purpose.

25. Standards and Assessment • The term ‘standards’ is ubiquitous but there are no simple measuring instruments that can be used to determine an appropriate value for a student’s achievement or for that matter a school. • There is no natural unit of measurement as there is for physical qualities such as weight or height. • Assessment in education is inherently inexact and should be treated as such (Harlen, 1994)

26. Standards and Assessment • Standards in the context of standards-referenced assessment describe the expected features or qualities at different levels of performance. • Standards are used to monitor growth in student learning and provide information about the quality of student achievement.

27. Unintended consequences of high-stakes assessments • Policy makers set high standards of achievement to inspire greater effort on the part of students, teachers and principals • Govt in England set target of 85% of pupils to reach Level 4 in Maths and Eng by Year 6 (a goal that is yet to be met)

28. Standards and Assessment • Pressure is on teachers and schools as they are judged publicly • Reay and Wiliam (1999) researched a class preparing for the national tests (SATs) • Stobart (2008) has indicated how aware children are of their expected level

29. Assessment and Accountability HANNAH: I’m really scared about the SATs. Mrs O’Brien [teacher at the school] came in and talked to us about our spelling and I’m no good at spelling and David [the class teacher] is giving us times tables tests every morning and I’m hopeless at times tables so I’m frightened I’ll do the SATs and be a nothing. RESEARCHER: I don’t understand Hannah. You can’t be a nothing. HANNAH: Yes, you can ‘cos you have to get a level like 4 or level 5 and if you’re no good at spellings and times tables you won’t get those levels and so you’re a nothing. RESEARCHER: I’m sure that’s not right. HANNAH: Yes it is ‘cos that’s what Mrs O’Brien was saying. (Reay and Wiliam, 1999 cited by Stobart, 2008:2)

30. Assessment and Accountability • May 2008 Headlines in England • ‘Tests damaging to school system’ • ‘Teachers criticise over-testing’ • ‘The original purposes of examinations, to assess students’ progress has become confused with school accountability and the performance management of teachers’(BBC, 2008:2)

31. Changing Climate of Assessment: England • Testing regime in England has been researched and analysed with the identification that on average each student in that education system will take over 100 formal tests or examinations (Mortimore, 2008) • October 2008 government in England abolished national tests for 14 year olds t

32. Changing Climate of Assessment: England • “… the government’s decision to listen to advice about the impact on schools and the need for change was welcomed” (Chair of the Office of the Qualifications and Examinations (Ofqual) Regulator) • New assessments would be subject to validation by Ofqual to ensure that they command confidence, and standards are secure.

33. Changing Climate of Assessment – Singapore Ministry of Education • At lower primary (Primary 1 and 2), … focus should be … on building pupils’ confidence and desire to learn. In these early years, too much emphasis on formal semestral examinations may not be the optimal way to achieve these outcomes … better … to use assessment to support and improve learning. • … Committee recommends that MOE consider using “bite-sized” modes of assessment, such as topical tests, to provide regular feedback on pupils’ learning to parents. • MOE should also train teachers to use a range of appropriate assessment techniques and tools to provide parents and pupils with better feedback on their progress. http://www.moe.gov.sg/media/press/2009/01/strong-fundamentals-for-future.php

34. Intelligent Accountability • Teachers, students and education authorities involved in accountability processes and develop a strong sense of professional responsibility and initiative (Sahlberg, 2006) • Finnish education system recognised internationally as a successful system that has built on the values of quality equity and social cohesion through public funding. • Finland a slow implementer ofmarket-oriented approaches to reform

35. Intelligent Accountability • In Finland all “basic school teachers must hold a Masters degree to become permanently employed” (Sahlberg, 2006: 153) and teaching is considered a research-based profession that has been central to teacher education developments • Increased numbers of school principals and teachers who have completed doctoral degrees (ibid)

36. Intelligent Accountability • Finland was top of the OECD countries in the 2003 PISA tests results in reading, mathematics and scientific literacy. • Strong performance was uniformly distributed across Finnish schools and very little between-school variance. • High in quality and high in equity.

37. Teacher Assessment • Teacher designed assessment tasks need to involve: • a variety of contexts, • range of modes with the assessment, • range of response formats and styles • range of indicators particularly to address issues of equity.

38. Teacher Assessment • TA is an important alternative to centrally devised tests because the teacher can address student’s needs that emerge from a particular context, sociocultural or historical background • Students work is assessed at the local level and forms part of the state system of assessment of student performance • Multiple judges are needed as in Queensland’s Senior Secondary System

39. Teacher Assessment • As the importance of teacher assessment has increased teachers have developed their assessment literacy • Fundamental issues in assessment design: • ‘fit for purpose’ • mode of assessment to impact positively on teaching and learning (Gipps, 1994)

40. Queensland Curriculum, Assessment and Reporting Framework • Essential Learnings • Standards • Bank of assessment tools • Promote teachers’ professional learning • Queensland Comparable Assessment Tasks (QCATs) in Years 4, 6 and 9 to measure essential learning in English, Maths and Science • Common reporting framework http://www.qsa.qld.edu.au/assessment/qcar.html

41. Queensland Curriculum Assessment and Reporting (QCAR) Framework Aims to: • improve student learning • increase comparability of assessment and reporting across schools • align curriculum, teaching and reporting • address teacher concerns re amount to be taught Yr 1-10 curriculum • use standards for the essential learnings to judge the quality of student performance at a particular level of schooling • assist school level planning by providing more comprehensive and comparable data on school performance • provide reportsto parents that make use of a common reporting framework to describe student achievement using a five point scale (A-E)

42. ARC Linkage Project: “Investigating standards-driven reform in assessment in the middle years of schooling” • Project Aims • Examine how teachers use stated standards to judge quality of student work; • Develop, trial and evaluate models of moderation to support consistency of teacher judgment and to inform standards-based reporting to students, parents/ carers and systems • Assess the utility of models of moderation for their responsiveness to different pedagogical and geographical contexts and student diversity and • Produce a best practice framework for teacher judgment in statewide moderation in years 1 - 10 including exemplars of how standards are met

43. Teacher Judgment • Centrality and complexity of teacher judgment practice needs to be understood • Teacher judgment involves: • Supplied textual artifacts (standards, criteria, samples of student work) • Tacit knowledge of different types • Social processes of dialogue and negotiation

44. Teacher Judgment • Standards play a part in the judgment process but are insufficient to account for how teachers ascribe value and award a grade to student work in moderation • The issue that emerges is the extent to which the identified factors (textual artifacts, tacit knowledge, social processes etc.) impact on teachers’ judgment and the implications regarding reliability and validity

45. Teacher Assessment • Teacher-based assessment is viewed as having high validity but questionable reliability • To address reliability teacher assessors need to: • Develop a common understanding of the standards • Recognise performances that similarly demonstrate those standards (Maxwell, 2002)

46. Findings • Textual referents (annotated work samples, guide to making judgments) meticulously specified what teachers were to attend to: “locate the evidence in the student work for each assessable element. Match the evidence for each assessable element to a task specific descriptor in the Guide to making judgments. Refer to the Annotated student work samples (if available) to support your understanding of the expected student response for each task specific descriptor” (Information sheet on ‘reviewing process’, QSA, 2007)

47. Working towards consistency • Conditions that facilitate greater dependability of teacher judgment “include the extent to which teachers share interpretation of criteria and standards” (Harlen, 2005: 213) • Moderation provides the context for teachers to share interpretations of assessment tasks and the requisite standards and to develop a common language for describing and assessing students’ work.

48. Two Approaches to Qualitative Judgments (Sadler, 2008) • Analytic - the assessor makes separate, qualitative judgments on each of the preset criteria - after the criterion by criterion judgments are made, they are combined, by rule or formula - the aggregate is either used as is or converted to a grade - final mark is thus built up from a series of smaller-scale decisions.

49. Two Approaches to Qualitative Judgments • Holistic • the assessor progressively builds up a complex response to a student work • this involves attending to particular aspects that draw attention to themselves, and allowing an appreciation of the quality of the work as a whole to emerge • the assessor then makes a qualitative judgment as to its overall quality and maps that judgment to the appropriate point on the grading scale • in addition to assigning the grade the assessor may provide a rationale for it (in summary form for the whole work or as running comments on various features of the work) • rationale and feedback statements necessarily invoke one or more criteria i.e. the global judgment is made first; references to criteria follow from reflection on appraisal

50. Two Approaches to Qualitative Judgments • Sadler (in press) argues that: …an explicit model produces outputs (grades) that appear to have been substantially ‘validated’ through careful attention to all the steps. However, the model itself is characterized by indeterminacy, that is inherently weak. … its implementation creates a veil of rigour that makes it difficult for learners to question either the process or the outcome.