Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment ?

1. Going for Gold - Can Checklists be a Gold-testing Tool in Language Testing and Assessment? Sauli Takala University of Jyväskylä 12th CERCLES Conference London Sept. 6-8, 2012

2. Overview • Some current issues in definition of the construct in language education • Task-based language learning/assessment : neglect of some relevant sources, critique of some sources • Some models of TB- assessment • Some arguments for and against a checklist approach • Assessment using checklists • Concluding remarks

3. Approaches to definition of construct of language performance : • ability-focused (eg,. Bachman 2002, 2006) • task-focused (eg., Norris 2005) • interaction-focused. • See next slide for an overview of approach to construct definition by Bachman (2007; What is the construct? The Dialectic of Abilities and Contexts in Defining Constructs in Language Assessment. In Fox et al. Language Testing Reconsidered. Ottawa: U of Ottawa Press, 41-71)

4. Approaches to defining the construct in language testing/assessment, Bachman, 2007

5. Are there fundamental differences or only a matter of difference in emphasis? Are we, in practice, usually interested in finding out to what extent there is ability in using language to do some tasks? Is there an inevitable interaction between degree of ability and the nature of the tasks? Do tasks represent the goals/required competences of interest?

6. Some useful early sources - but to some extent neglected • Linguistic philosophy (esp. later Wittgenstein, Philosophical Investigations, 1953) • Sociology and social psychology (Kurt Lewin, Bales (1950), Festinger (1951), Watzlawick (1967) – communication and interaction in small groups • Linguistic pragmatism – especially early critique of Austin (1955/1962), and Searle (1969) by German pragmalinguists (Wunderlich, 1972; Ehlich & Rehbein, 1972 – praxeograms...)

7. ROLE/SOCIAL ROLE: also neglected? • In social psychology, a role: a set of connected behaviours, rights and obligations as conceptualised by actors in a social situation. It is an expected or free or continuously changing behaviour providing a given individual social status or social position. • Functionalist and interactionist understandings of society and all social behaviour, including communicative behaviour (cf. next slide) • Division of labour - interaction among heterogeneous specialised positions/roles. • Roles are occupied by individuals/actors. • Social roles/behaviours are guided by social norms (expectations, degree of conventionality). • Legitimate and constructive roles – anticipation of rewards (vs. punishments) • Changed social conditions role change.

9. External situation Channels. media Time, place, ”noise” Code: usage Persons: goals, relation-ships Topic/aspect Dynamics of situated inter-action Processing: input-output Internal situaton

10. Fischer et al., Guidelines for task-based university language testing. Graz: ECML, 2011, p. 21.

11. Puren, C. ( 2008). Formes pratiques de cominaision entre perspectives actionnelle et approche communicative: analyse comparative des trois manuels. Langue Modernes. Cited in Fischer et al., 2011, p. 22. (note the inclusion of roles!)

12. Testing/assessment – issues and problems • Testing/assessment is notoriuosly demanding/ complex • Testing/assessment should fulfil many criteria – (a) there are many things that should be got right • (b) there may be some things that one may not know well enough (inevitable degree of ignorance/ even ineptitude?) • (c) there may be serious consequences. • -> Can testing/assessment be made less demanding, less complex, and exclude avoidable problems/ errors?

13. Assessment – So Many Things to Get Right • Observing aspects of good practice <-> learning to first avoid less good practices • For instance, in learning to rate, avoiding • What about using Checklists? • Halo (horn) effect • Central tendency effect • Restriction of range effect • Severity or leniency

14. Checklists: Everyday use – shopping lists • Checklists/Inventories – have a long history in eg. personality inventories (self-assessment; preferences). • Checklists – frequent use esp. in health care/ nursing, aviation/pilots, building industry…. • Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books. • Western Michigan Checklist Project (www.xxx) • Scriven, M. (2000/2007). The Logic and Methodology of Checklists. Western Michigan University.

15. Reluctance to use checklists - objections It somehow feels beneath us to use checklists, an em-barrassment. It runs counter to the deeply held beliefs about the how truly great among us – those we aspire to be – handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism need updating. (Gawande, p. 173) Checklists – boring -> Professionalism – expectations: selflessness, skill, trustworthiness, discipline. We are not built for discipline (novelty, excitement). We have to work at discipline. (p. 182-3). What about education?

16. Checklists are valuable evaluation devices when carefully developed, validated, and applied. • A sound evaluation checklist • clarifies the criteria that at least should be considered when evaluating something in a particular area • aids the evaluator not to forget important criteria • enhances the assessment's objectivity, credibility, and reproducibility. • Moreover, such a checklist is useful in planning, monitoring and guiding operations, and assessing their outcomes. In the evaluation vernacular, checklists are useful for both formative and summative evaluations. • (Scriven, M. The Logic and Methdology of Checklists. Western Michigan University. 2000, 2007)

17. Checklists are not created equal – more bad than good checklists? • Bad checklists: vague, imprecise, too long, hard to use, impractical • Good checklists: precise, efficient, to the point, easy to use, provide reminders (not exhaustive), practical. Usually 5- 9 points only.

18. Types of Checklists • List (mnemonic) – eg. shopping list; laundry list – grouping important • Sequential checklist - order matters: strongly sequential kind, where the sequencing (of some or all checkpoints) must be followed in order to get valid results. • A weakly sequential checklist: order is of some importance, but for psychological or efficiency reasons rather than from logical or physical necessity

19. Iterative checklist is sequential, in whole or part, but requires—or may require—multiple passes in order to reach a stable reading on each checkpoint. Diagnostic checklist that is used—for example— by taxonomists, mechanics, and toxicologists. It typically supports a classificatory kind of conclusion—one that may be descriptive or evaluative. Probably the most important kind of checklist for evaluation purposes is the criteria of merit checklist ( COMlist). This is what judges use when rating entries in a skating or barbeque or farm produce competition (and, of course, when rating language performance).

20. Key Requirements for Comlists /Criteria of Merit) • Most of the following are self-explanatory and refer to the criteria or checkpoints that make up a comlist: • The checkpoints should refer to criteria and not mere indicators. • The list should be complete (no significant omissions).The items should be contiguous, i.e., nonoverlapping (essential if the list is used for scoring). • The criteria should be commensurable. • The criteria should be clear (comprehensible, applicable). • The list should be concise (to assist its mnemonic function); i.e., it should contain no superfluous criteria. • The criteria should be confirmable (e.g., measurable or reliably inferrable) • (Scriven, 200, 2007)

21. Positive developments • Lack of knowledge /ignorance diminishing: reasonable guidelines of good practice. • Transparency is making some headway: useful research eg., on linking exams to the CEFR is being carried out and increasingly reported. • Training on language testing/assessment is increasingly available (EALTA, ALTE). • Thus, we have increasingly better chances to get things right, • Scriven, M. (2000/2007). The Logic and Methodology of Checklists. Western Michigan University. • Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books.