1 / 38

Assessment as a washback tool: is it beneficial or harmful?

Assessment as a washback tool: is it beneficial or harmful?. Nick Saville Director, Research and Validation University of Cambridge ESOL Examinations October 2008. Outline. Introduction Washback and impact: some definitions Impact and the law of unintended consequences

dusty
Download Presentation

Assessment as a washback tool: is it beneficial or harmful?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Assessment as a washback tool: is it beneficial or harmful? Nick Saville Director, Research and Validation University of Cambridge ESOL Examinations October 2008

  2. Outline • Introduction • Washback and impact: some definitions • Impact and the law of unintended consequences • Researching impact: the Cambridge ESOL approach • Conclusion

  3. 1. Introduction • Examinations • access to opportunity • fairness • Exert a powerful influence on educational processes • possibly negative? • concern for social consequences, ethicality, accountability

  4. Questions to be addressed: • How can educators and assessment providers foster beneficial effects from their tests and examinations? • How can harmful consequences be avoided or their impact mitigated?

  5. “Impact by design” • Integral part of a framework for developing and validating examination systems • A concept akin to social impact assessment

  6. 2. Washback and impact: definitions • Washback (or backwash) has been broadly defined in the assessment literature as the effect of testing on teaching and learning • One aspect of the broader phenomenon known as impact – as we shall see later

  7. Washback • Alderson and Wall, 1993: • 15 washback hypotheses • Based on whoor whatmight be affected: • Teaching • Learning • Content • Rate of learning • Sequence of teaching/learning • Degree/depth of curriculum coverage • Attitudes of teachers/learners • Etc.

  8. Washback • A continuum - stretching from harmful at one end, through neutral to beneficial at the other end Negative Neutral Positive - +

  9. Washback • Negative? • Restriction of content – narrowing of curriculum • Too much time practising for the test • Positive? • Transparent objectives and outcomes • Increased motivation of learners • Increased accountability of teachers (?) • BUT – cause and effect explanations are rarely adequate …..

  10. Washback Models • In the language testing literature: • Bailey (1996) • Watanabe (2004) • Cheng (2004, 2005) • Green (2007)

  11. Bailey’s Model (1996) - based on Hughes,1993 • 3 Ps: • Participants • students • teachers • Processes • Products • learning • teaching • materials • curricula

  12. Watanabe’s Model (2004) • Five dimensions: • Specificity • Intensity • Length • Intentionality • Value • Factors influencing theprocess of washbackare related to: • the test itself • status • stakeholders

  13. Green’s Model See: Studies in Language Testing, 25, 2007: IELTS Washback in Context Washback will be most intense – have the most powerful effects on teaching and learning behaviours – where participants see the test as challenging and the results as important (high stakes) SEE BLUE ARROW

  14. Impact Impact concerns the effects and consequences a test can have beyond the classroom and immediate learning context: • On individual career or life chances • In educational systems and in society

  15. Impact • Impact deals with wider influences and includes the “macro contexts” - tests and examinations in society • Washback is an aspect of impact related to the “micro contexts” of the classroom and the school

  16. Impact • Investigating impact is integral to validation • an essential component in establishing the usefulness of an assessment system - fitness for specific purposes and contexts of use • Consistent with Messick views of validity (1989, 1996) • Consequential aspects of validity • fairness and ethics

  17. Impact • Educational systems – complex phenomena • Stakeholder constituencies • many different stakeholders in educational processes • complex network of relationships

  18. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

  19. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

  20. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

  21. Impact • Multiple voices • views, beliefs and attitudes • Important for examination providers • to collect feedback from the stakeholder • to take stakeholder perspectives into account • Dynamic relationship between micro and macro contexts

  22. 3. The “law” of unintended consequences • “Any purposeful action will produce some unintended consequences” or side-effects • “Goodhart’s Law” (or “Campbell’s Law” in the USA) • a variant of the “law” of unintended consequences

  23. “Goodhart’s Law” • “All performance indicators lose their meaning when adopted as policy targets” • Examples: • England - school achievement targets - school league tables • USA – No Child Left Behind (NCLB) • The clearer you are about what you want, the more likely you are to get it – but the less likely it is to mean what you wanted it to! (Dylan Wiliam, 2008)

  24. Perverse incentives? • Assessment policy can create a tension between • educational objectives at the micro level(teaching and learning in schools) and • a requirement for accountability at the macro level

  25. What role can examination providers play? How can examination providers ensure that : • examination systems are “fit for purpose”? • research is carried out to “find out what is going on” in contexts of use? • mitigating action is are carried out if/when negative effects and consequences occur?

  26. 4. Researching impact • Cambridge ESOL • an international examinations board • Maxims of Test Impact • An attempt to integrate an action-oriented approach to investigating impact into working practices • Milanovic and Saville, 1996

  27. Maxims for achieving/monitoring impact • Maxim 1 PLAN • Use a rational and explicit approach to test development • Maxim 2 SUPPORT • Support stakeholders in the testing process • Maxim 3 COMMUNICATE • Provide comprehensive, useful and transparent information • Maxim 4 MONITOR and EVALUATE • Collect all relevant data and analyse as required. Milanovic and Saville, 1996 - Considering the impact of the Cambridge examinations

  28. Maxim 1 Plan Use a rational and explicit approach to test development • a process model – cyclical and iterative • creates the necessary conditions for appropriate tests to be developed and for on-going validation to take place • begins with the purpose - including anticipating • how the test should (or might) be used • how relevant and useful it is likely to be - social consequences and value implications • potential (unplanned) side-effects

  29. Maxim 2 Support Support stakeholders in the testing process • Involvement of stakeholders - during test design and development • consultation on specifications/syllabus design • participation in pilot tests • Professional support programmes • for institutions and individual teachers/students etc. • Training of suitable personnel to work on all aspects of the examination cycle • item writers, examiners, etc.

  30. Stakeholders in the testing constituency Learners Teachers Test writers/examiners Receiving institutions School owners Future employers Government agencies Professional bodies Test centre administrators Materials writers Publishers etc Learners Parents/carers Teachers Receiving institutions Employers School owners Examiners Government agencies Professional bodies Academic researchers Test writers/ Examiners etc Test constructs Test format Test conditions Test assessment criteria Test scores Testing System Inputs to test design Contexts of test use - consequences

  31. Maxim 3 Communicate Provide comprehensive, useful and transparent information • Explain issues related to assessment adapted for the different stakeholders • the nature of the language constructs being assessed • the meaning of language test results • etc. • A major challenge for all test providers!

  32. Maxim 4 Monitor and Evaluate Collect all relevant data and analyse as required • For example, feedback from the candidates and their teachers • from local contexts where the test is used (i.e. at the micro level) • A long-term endeavour as it involves: • the development of suitable instruments for the collection of adequate data • appropriate research methodologies e.g. mixed method designs, case studies etc. • Evaluate the test’s usefulness routinely • determine the need for changes and periodic revisions

  33. Outcomes of impact studies • The World-wide survey of the impact of IELTS • The Italian Progetto Lingue 2000 (PL2000 project) • See Hawkey, (2006) • Studies in Language Testing, 24 The theory and practice of impact studies: Messages from studies of the IELTS test and Progetto Lingue 2000

  34. Progetto Lingue 2000 • The policy was • “.... to introduce innovation into the teaching and learning of other languages by putting greater emphasis on the development of communicative competence in all grades of the school system” • It included: • the adoption of the Council of Europe’s Common European Frameworkof Reference (CEFR) as the basis for learning objectives and standards • certification of proficiency – the testing • (by a certificating body recognised internationally)

  35. Students Parents Teachers Teacher-trainers Curriculum developers Testers Publishers Receiving institutions Employers Students Parents Teachers Teacher-trainers Curriculum developers Testers Publishers Receiving institutions Employers Italy - PL2000 Impact Project Materials Learning goals, curriculum, syllabus Methodology Teacher Support Testing

  36. 5. Conclusion • Impact research now reflects the growing importance of evidence-based approaches to educational planning and evaluation • By adopting an “impact by design” approach - through careful test development and validation strategies, and by using the findings from impact research to guide future actions - more effective assessment policies and practices can be developed to meet the needs of contemporary education

  37. Thank you

More Related