If Language is a Complex Adaptive System, What is Language Assessment?

1. If Language is a Complex Adaptive System, What is Language Assessment? Presented at “Language as a Complex Adaptive System”, an invited conference celebrating the 60th Anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, 2008. The first author was supported by a grant from the Spencer Foundation.

2. Key Ideas • Assessment as evidentiary argument, not simply as measurement. • Arguments constructed around … • View of the nature of proficiency. • Situations and ways people acquire it and use it. • Relevant work taking place in language testing from an interactionist perspective. • Reconceiving measurement models

3. The Assessment Argument(Messick, 1994) • What complex of knowledge, skills, or other attributes should be assessed? • What behaviors or performances should reveal those constructs? • What tasks or situations should elicit those behaviors? We’ll look at a more technical representation in a little while.

4. Snow & Lohman, 1989 Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. [q] … An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

5. LaCAS and Assessment Arguments Interactionalist perspective in language testing: • Communicative competence • Contextual features of tasks • Language tests for specific purposes

6. An Interactionalist Perspective(Young, 2000, 2008) … language used in specific discursivepractices rather than … language ability independent of context. Focus on the co-construction of discursive practices by all participants ... A set ofgeneralinteractional resourcesthat participants draw upon in specific ways in order to co-construct a discursive practice.

7. An Interactionalist Perspective(Young, 2000, 2008) Relationship between participants’ employment of interactional resources and the context in which they are employed. Varying with the practice and the participants…

8. Challenges for Assessment (Chalhoub-Deville, 2003) • Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and … • reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

9. Sociocognitive Foundations • Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: • Connectionist metaphor, associative memory • Situated cognition & information processing • Construction-Integration (CI) theory of comprehension (Kintsch and others) • Individual  Sociocultural perspectives • A cognitive theory of cultural meaning (Strauss & Quinn, 1997)

10. A Cognitive Theory of Cultural Meaning “Interactional Resources” • Extrapersonal: • Cultural models: What ‘being sick’ means, restaurant script, Newton’s laws, complaints • Linguistic: Grammar, conventions, constructions • Intrapersonal: • Patterns from experience at many levels • Schemas / frames / understandings / assumptions • Interplay  Situated understandings • Access to, and ways of interacting with, shared structures in order to accomplish goals • The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them. (R. Ellis, 1985)

11. not observable not observable Inside A A Inside B B observable

12. and internal and external aspects of context … Context A la Kintsch: Propositional content of text / speech… Inside A A Inside B B

13. Context • The C in CI theory, Construction: • Activation of both relevantandirrelevant • bits of cultural models, experiences, e.g., • Restaurant script, Human motivation • Guided in part by linguistic models, e.g. • Conventions, constructions, rhetorical frames • Content of utterance • History with interlocutor • Conversation thus far Inside A A Inside B B E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings. • If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction). • A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

14. Context Inside A A Inside B B • The I in CI theory, Integration: • Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns • Situation model is basis of understanding

15. Context Inside A A Inside B B Situation model is also the basis of planning and action.

16. Context Context Context Previous situation models are input to subsequent situation models. Context Inside A A Inside B B

17. Context Inside A A Inside B B Context Context Ideally, activation of relevant and compatible cultural & linguistic models… Context

18. Context Inside A A Inside B B Context Context to lead to (sufficiently) shared understanding; i.e., co-constructed meaning. Context Kramsch’s "shared internal context "

19. External / pubic aspects of context, e.g., • Setting • Physical attributes Comments about context… Can distinguish external and internal aspects of context (e.g., Douglas, 2000) • Re assessment, • Target language use (TLU) features • Task features Inside A A Inside B B Context Context Context Context

20. Aspects of cultural/linguistic/interaction context as interpreted by an external observer. Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models. • As such, in assessment, plays role in • Evaluation, hence • Observable variables Inside A A Inside B B Context Context Context Context

21. What can we say about individuals? Use of resources in appropriate contexts in appropriate ways; i.e., Attunement to cultural/linguistic patterns: • Recognize markers of externally-viewed patterns? • Construct internal meanings in their light? • Act in ways appropriate to targeted cultural/linguistic models? • What is the range and circumstances of activation? (variation of performance across contexts)

22. The Assessment Argument(Messick, 1994) • What complex of knowledge, skills, or other attributes should be assessed? • What behaviors or performances should reveal those constructs? • What tasks or situations should elicit those behaviors?

23. Toulmin’s Argument Structure Claim unless Alternative explanation since Warrant so Backing Data

24. Backing concerning assessment situation unless Alternative explanations on account of Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Claim about student so Data concerning student performance Student acting in assessment situation

25. Backing concerning assessment situation unless Alternative explanations on account of Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation • In interactive task, • performance flows in time, • performance changes situation, • may or may not be series of task and observable variables Claim about student so Data concerning student performance Student acting in assessment situation

26. Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference. Backing concerning assessment situation unless Alternative explanations on account of Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Claim about student so Data concerning student performance Student acting in assessment situation

27. Backing concerning assessment situation unless Alternative explanations on account of Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?) Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Claim about student so Data concerning student performance Student acting in assessment situation

28. “Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models? Backing concerning assessment situation unless Alternative explanations on account of Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Claim about student Fundamental to situated meaning of student variables in measurement models; Both critical and implicit. so Data concerning student performance Student acting in assessment situation

29. Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees. (i.e., “method effects”) Backing concerning assessment situation unless Alternative explanations on account of Warrant concerning assessment • Important for … • Alternative explanations • Variable performance since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Claim about student so Data concerning student performance Student acting in assessment situation

30. Backing concerning assessment situation Claim about student unless Alternative explanations on account of Warrant concerning assessment since so Data concerning student performance Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Student acting in assessment situation Design Argument

31. unless Alternative explanations Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Use Argument Claim about student in use situation (Bachman) unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation Backing concerning assessment situation Claim about student on account of so Data concerning student performance Design Argument Student acting in assessment situation

32. unless Alternative explanations Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Use Argument Claim about student in use situation unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation Backing concerning assessment situation Claim about student on account of so Data concerning student performance Design Argument Student acting in assessment situation

33. unless Alternative explanations Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Use Argument Claim about student in use situation unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation Backing concerning assessment situation Claim about student on account of so Data concerning student performance Design Argument Student acting in assessment situation

34. unless Alternative explanations Warrant concerning assessment since Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Use Argument Claim about student in use situation unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation Backing concerning assessment situation Claim about student on account of so Data concerning student performance Design Argument Student acting in assessment situation

35. This is the essence of warrant for claim in use argument. Backing concerning assessment situation Claim about student unless Alternative explanations on account of Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context. Warrant concerning assessment since so Data concerning student performance Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Student acting in assessment situation Use Argument Claim about student in use situation unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation • What features do tasks and TLUs share? • Implicit in trait arguments • Explicit in interactionalist arguments Design Argument

36. Questions of validity / generalizability: • TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources. • Target models not activated in LTM in TLU context. Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998) Backing concerning assessment situation Claim about student unless Alternative explanations on account of Warrant concerning assessment since so Data concerning student performance Data concerning task situation Warrant concerning task design Warrant concerning evaluation since since Other information concerning student vis a vis assessment situation Student acting in assessment situation Use Argument Claim about student in use situation unless Alternative explanations Warrant concerning use situation since on account of Other information concerning student vis a vis use situation Backing concerning use situation Data concerning use situation What features do tasks and TLUs not have in common? Design Argument

37. Implications for measurement models • Basic form: Probability of aspects of performance Xij given parameters for person i and situation j (all could be vector-valued) • Way too simple • No explicit connection with CI comprehension model, interaction processes, etc. • Apparent separation of person and situation characteristics These are indeed properties of the conventional meaning of the measurement model and parameters.

38. An Interactionalist Perspective: Instantiation in a Context • Xs result from particular persons calling upon resources in particular contexts (or not, or how) • Mechanicallyqs simply accumulate info across situations • Our choosing situations and what to observe drives the situated meaning ofqs. • Situated meaning of qs are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

39. How to model inconsistent performance? • Traditional: Model as “noise” / unreliability • Promising direction: Model individual’s degree or pattern in variation in terms of context features • If “motivated”: Model in terms of qs • Divide & Conquer: Multiple unidimensional tests (OET) • Exploratory multidimensional models • Controlled : Structured multidimensional models • Critical importance of what else you know • multidimensional • Structured

40. Conclusion How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt (1986)

41. Conclusion I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population. The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context. Douglas (1998)

42. Conclusion Interactionalist view of test theory… • for arguments in interactionalist view of language • to assemble, analyze, & interpret assessments in light of context and purpose. Methods and exemplars needed, but pressing need is interpretive frame … • To connect view of language proficiency with the machinery of test theory, • Toward modeling students’ (inter)actions in purposeful variations in task contexts.

43. Thank you!