Toward a Test Theory for the Interactionalist Era

1. Toward a Test Theory for the Interactionalist Era Robert J. Mislevy University of Maryland Samuel J. Messick Memorial Lecture Presented at the Language Testing Research Colloquium, Barcelona, Spain, June 9, 2007.

2. Chi, Feltovich, & Glaser, 1981

3. Key Ideas • Generative patterns from the domain organized experts’ perceptions, understanding, and actions • Experts assembled pieces from patterns to fit particulars of context and purpose • F=MA: Conventional v. situated meaning • Critical role of narrative layer • Integrates principles & equations with context • Locus for understanding, planning, & action

4. Why Am I Telling You This? • Situative / sociocultural / interactionist perspective is frontier in all domains • Analogues between model-based reasoning and using language • Test theory is model-based reasoning • Interactionalist perspective on test theory • Bottleneck the narrative layer

5. Outline • Messick on assessment arguments • Interactionalist perspective • Re language, comprehension, cultural meaning • A narrative space / metaphor for assessment in this light • Attention to senses and roles of context • Implications for building and using measurement models

6. The Assessment Argument • What complex of knowledge, skills, or other attributes should be assessed? • What behaviors or performances should reveal those constructs? • What tasks or situations should elicit those behaviors? Messick (1994) “The interplay of evidence and consequences in the validation of performance assessments”

7. Perspectives on the L2 Construct What is foregrounded? • Trait • Focus on underlying abilities of individuals that are called upon in a wide variety of situations. • Behaviorist • Focus on context, from external point of view—success of action in specified classes of situations. • Interactionalist …

8. An Interactionalist Perspective Concern with language used in specific discursivepractices rather than on language ability independent of context. Focus on the co-construction of discursive practices by all participants ... A set ofgeneralinteractional resourcesthat participants draw upon in specific ways in order to co-construct a discursive practice. (Young, 2000, p. 5)

9. An Interactionalist Perspective successful interaction presupposes not only a shared knowledge of the world, … but also the construction of a shared internal context … that is built through the collaborative efforts of the interactional partners. Kramsch ( 1986, p. 367)

10. Challenges for Assessment • Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and … • reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries. Chalhoub-Deville (2003, p. 373)

11. A Narrative Frame • Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: • Connectionist metaphor, Associative memory • Situated cognition & information processing • Construction-Integration (CI) theory of comprehension (Kintsch and others) • Individual  Sociocultural perspectives • A cognitive theory of cultural meaning (Strauss & Quinn, 1997)

12. A Cognitive Theory of Cultural Meaning “Interactional Resources” • External: knowledge “out there”, patterns that exist in use by groups, in tools & processes • Cultural models: What ‘being sick’ means, restaurant script, Newton’s laws, complaints • Linguistic: Grammar, conventions, frames • Interactional: enable the co-construction of new shared meanings • Internal: patterns in individuals attuned to shared external patterns

13. Inheritance from Schema Theory • Knowledge as organized patterns, at many levels… • Assembled to understand & to create particular situations in the world • Developed, strengthened, modified by use • Associations of all kinds, including applicability, affordances, procedures, strategies, affect • “The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them” (Ellis, 1985)

14. and all aspects of context… Context A la Kintsch: “Conventional” meaning, or propositional content of text / speech… Inside A A Inside B B

15. Context • The C in CI theory, Construction: • Activation of both relevantandirrelevant … • Linguistic models, e.g. • Conventions, Rhetorical frames • Cultural models, e.g., • Equilibrium, Human motivation Inside A A Inside B B • If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction). • Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

16. Context Inside A A Inside B B • The I in CI theory, Integration: • Resulting synthesis of reinforced activated cultural / linguistic / situational patterns: • the Situation model • Akin to ‘stories’ in Larkin’s physics study • Situation model is the understanding

17. Context Inside A A Inside B B Situation model is also the basis of planning and action.

18. Context Context Context Context Inside A A Inside B B

19. Inside A A Inside B B Context Context Context Ideally, participants activate cultural & linguistic models that are compatible in relevant ways… Context

20. To lead to co-constructed meaning / sufficiently (?) shared understanding Inside A A Inside B B Context Context Context Kramsch’s "shared internal context" Context

21. Preceeding overheads highlight the importance of a common narrative space for thinking about assessment: A = Linguist B = Psychometrician Inside A A Inside B B Context Context Context “shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand. Context

22. Some external aspects of context are public & objective, e.g., • Setting • Physical attributes • Directives Senses and roles of “context” Can distinguish external and internal aspects of context (e.g., Douglas, 2000) Target language use (TLU) features Assessment task features (Bachman & Palmer) Inside A A Inside B B Context Context Context Context

23. Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case. The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model. Inside A A Inside B B Context • Note the need to activate many other l/c models in order to construct a situation model, plan, and carry out action. • Many places to slip, but others to compensate. • “Alternative explanations” in assessment argument. Context Context Context

24. Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen. As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them. Inside A A Inside B B Context • This sense of context plays a key role in • Evaluation of performance, hence • Observable variables that go into a measurement model. Context Context Context

25. What can we say about individuals? Use of interactional resources in appropriate contexts in appropriate ways; i.e., Attunement to targeted cultural/linguistic patterns: • Recognize markers of externally-viewed patterns? • Construct internal meanings in their light? • Act in ways appropriate to targeted c/l models in the assessment contexts? • What are the ranges and circumstances of activation? (variation of performance across contexts)

26. Implications for measurement models • Basic form: Probability of aspects of performance Xij given parameters for person i and situation j (all could be vector-valued) • Way too simple • No explicit connection with CI comprehension model, interaction processes, etc. • Apparent separation of person and situation characteristics These are indeed properties of the conventional meaning of the measurement model and parameters.

27. An Interactionalist Perspective: Instantiation in a Context • Xs result from particular persons calling upon resources in particular contexts (or not, or how) • Mechanicallyqs simply accumulate info across situations • Our chosing situations and what to observe drives their situated meaning. • Situated meaning of qs are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

28. An Interactionalist Perspective: Instantiation in a Context • Inference to criterion contexts (TLU) depends on analysis of what l/c models are called upon in assessment use argument… • What is similar, what is different, re the resources task & criterion situations call for? • To what degree does activation and success in task context correspond to activation and success in criterion context? (e.g., Bachman, Chalhoub-Deville, Douglas, Chapelle)

29. An Interactionalist Perspective: Instantiation in a Context • What demands do we minimize via task design, so needn’t model? • What resources do we already know examinees can draw upon, so tasks can require them but we needn’t model? • “Hidden” but essential to meaning • Occupational English Test (McNamara) • Analogous to ‘focus on forms’ learning

30. An Interactionalist Perspective: Instantiation in a Context How to model inconsistent performance? • If “unmotivated,” it’s noise; via probability model • Promising direction: Model individual’s degree or pattern in variation in terms of context features • If “motivated”: Model in terms of qs • Divide & Conquer: Multiple unidimensional tests (OET) • Exploratory multidimensional: Discover patterns in data. • Controlled: Structured multidimensional models (e.g., Embretson, Adams & Wilson, von Davier)

31. Structured Multidimensional Models • Still way too simple, but … • purposeful modeling of motivated variation in complex tasks when persons differ in targeted ways • exploit what is known about examinees but not modeled • Controlled mixes of demand features • E.g., in OET-like situations, wrt medical knowledge, complexity of stimulus language, complexity of language to be produced. • “Throwing the data over the wall” won’t work

32. Structured Multidimensional Models Probabilities modeled in terms of task features: • Which dimensions are relevant for which observables from which tasks? (Robinson’s “difficulty” features) • Task parameters modeled in terms of Robinson’s “complexity” features. • Hence a priori structure of patterns to interpret • Can organize qs in terms of traits or context features • Coordinated task design and measurement model • Create tasks within task models

33. Conclusion How much can testing gain from modern cognitive psychology? So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be made are minimal. Buzz Hunt (1986)

34. Conclusion I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population. The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context. Douglas (1998)

35. Conclusion Interactionalist view of test theory… • for assembling, analyzing, and interpreting assessments, • for arguments in interactionalist view of language • Methods and exemplars needed, but more pressing need is narrative frame … • To connect view of language proficiency with the machinery of test theory, • Toward modeling purposeful variations in a coherent design space.