Group Interaction in SBA: How Authentic is it?

1. Group Interaction in SBA: How Authentic is it? Jasmine Luk Faculty of Education The University of Hong Kong

2. Why is authenticity a concern in language testing? • SBA - a performance-oriented test  to elicit the demonstration of observable behaviours that reveal features of the “target language use domain” (TLU) (Bachman & Palmer, 1996, p.44) • TLU for SBA  how language is used essentially for purposes of communication in real-life situations  shifting emphases in L2 oral proficiency assessment fromaccuracy in linguistic structurestocommunicative competence that embraces the ability to initiate, respond and negotiate meaning in situated performances (e.g. Canagarajah, 2006; Lazaraton, 2002; Purpura, 2008)

3. How do we know if a test is authentic? • A growing interest in analyzing the discourse of students’ performance in different formats of oral proficiency assessments for the purpose of language test validation (Fulcher, 1987; Shohamy, 1991)

4. Discourse features of group interaction • He and Dai (2006) investigated the group interaction corpus (170000 words) of the College English Test-Spoken English Test (CET-SET) in China • Findings • The test design failed to elicit adequate interactional language functions that reflect negotiation of meaning, modifying, challenging, and persuading (all below 5% whereas (dis)agreeing carried 49.5%) • candidates more concerned with delivering their own lines; tended to frame the task as an assessment rather than a communicative event

5. Gan, Davison and Hamp-Lyons (in press) report contrary findings from a single case of peer group interaction for SBA: • The range of speech functions that characterize the topic negotiation process in the interaction share features of ordinary conversations; candidates able to relate to others

6. The Present Study • Objective: to analyze the discourse features of group interaction for oral proficiency assessment • Context: School-based assessment (SBA) in Hong Kong • Test-takers: 43 female Hong Kong secondary students in 11 groups studying in an English-medium girls’ school • Task: In groups of four and based on the different print fictional texts read, students talked about either (1) which character they would choose from the text to live with, and where; or (2) which character they would choose from the text to open a shop with, and what kind of shop. In both cases, students have to justify their proposals. Each group had 6 minutes for the interaction.

7. Types of Data collected • the group interaction data of the whole class; • the feedback sheets for the whole class with scores assigned by the teacher; • data from interviews (conducted after the SBA tasks completed) with the English teacher (Susan, a pseudonym) and six students (Heidi, Eva, Daisy, Tracy, Ally and Nellie, all pseudonyms) identified by the teacher representing high and low English proficiency levels; and • data from a questionnaire administered to the whole class.

8. General Background of the Student Participants • Being an EMI school, students among the top 35% of students in terms of academic performance in HK • The school is situated in Kowloon surrounded by several public housing estates and a few private estates • T’s comments on the students’ general English performance: “These students are extremely cooperative, well-mannered and hard-working, but they were not eager at all to speak English even though they know they have to speak it. They were shy and had the impression that being too eager to speak English is an act of showing off.” (translated from Cantonese)

9. Students’ self-evaluation of language use -Data from the questionnaire Level of agreement on a five-point Likert scale (with ‘1’ meaning strongly disagree and ‘5’ strongly agree) to two statements: • “I often interact with my friends in English” (2.29) • “I often interact with my friends in Cantonese.” (4.74)

10. Methods of Analysis Qualitative analysis grounded in the data: • Applied conversation analysis (e.g. the sequence of the turns, the size of the turns, the turn-allocation mechanism; supplemented with data from the interviews and the questionnaire) • Frame analysis (principles of organization in speech events) (see Goffman, 1974) • Types of talk in small group activities (e.g. Mercer’s (1995) classification into disputational, cumulative, and exploratory modes) • Functional speech acts (e.g. informing, (dis)agreeing, reasoning, hypothesizing)

11. Findings • Recurrent frames, types of talk, and speech acts (Tables 1 & 2; Excerpt 1) • Ritualized openings and closings • Orderly turn-taking practices • Heavier-weighting and front-loading CON frames • Frequent surface converging responses • Avoidance of negotiation(Excerpts 2 & 3) • Self-initiating a response to avoid dead air (Excerpts 4 & 5) • Role-playing critical respondents (Excerpts 6 & 7)

12. Ritualized openings and closings • “Let’s start our discussion.” • “Yes” • “To conclude,…” • “That’s the end of our sharing” • “Sorry…”

13. Orderly turn-taking practices • “What’s your opinion?” • “What do you think, xxx?” • “sorry sorry, you first.”

14. Heavier-weighting and front-loading CON frames • words uttered by students in the CON frames (4980); • words uttered by students in the REsP frames (2500); • CON frames tended to appear one after the other right from the beginning of the interaction, leaving relatively less time for responses.

15. Frequent surface converging responses • surface agreements such as “I agree with you”, and positive comments such as “Oh, that’s a good idea”frequently found • cumulative talk in which students show attempts to “build on each other’s contributions, add information of their own and in a mutually supportive, uncritical way construct together a body of shared knowledge and understanding” (Mercer, 2000, p.97) was rare  see Excerpt 1, turns 3 & 4 • Diverging responses were infrequent, and direct disagreements usually prefaced with “I am sorry.”

16. Avoidance of negotiation • According to Pica (1987), negotiation of meaning, namely confirmation checks, clarification requests, and comprehension checks, usually occur with signals of comprehension difficulty. • See Excerpts 2 & 3

17. Findings from the teacher interview - Comments on students’ overall performance (translated from Cantonese, italized words originally in English) “They seldom elaborate upon peers’ points. Very often, they only wait for opportunities to deliver what they have prepared rather than patiently and seriously listen to other people talking before giving responses. So they performed and take turns in a rather mechanical way. It’s unnatural. When other people were talking, they were only concerned with reading their own notes. Sometimes, you know they didn’t understand what their peers talked about because sometimes I didn’t understand too, but they still habitually say yes, I agree without any justification.”

18. Findings from the students interviews – comments on own and peers’ performance (translated from Cantonese, italicized words originally in English • “I found it difficult to give responses during group interaction. It seems that I didn’t have enough ideas. I lacked ideas, and it seems every time, I could only think of I agree as a response.”- Daisy • “I heard that some classmates after knowing who they were going to do the group interaction with would get together to discuss who should start, who should follow up, and who should give conclusion, etc. This really makes it look like acting.” -Nellie • “During the interaction, we seemed to have delivered the script we have prepared mechanically with very little exchange.”-Tracy

19. “The way we spoke was totally unnatural when doing SBA. For example, people would say ‘I agree with you’ in a dull and wooden way. It was so obvious that people just used this sentence to help themselves start their talk. It was used to fill up the time gap.”- Ally • “Not just SBA, but any other public exams that involve having to discuss with others look like acting. Before I learnt discussion skills in Form 3, I wouldn’t be so polite as to say “sorry” or “I am sorry that” when I ban other’s views. Now we have to say “sorry” for exams, but in real life, we won’t act like this. I feel rather awkward to be over-polite to other participants in the SBA talk as they are all my friends.”- Zoe • “I am satisfied with the interaction on that day. We were really discussing and not just trying to present own ideas.” - Heidi

20. Findings from the score sheets • Level 5 or 6 (the top two levels) for “communication strategies” awarded to 33 out of 43 students; only two students scored below ‘4’; • Even Daisy who expressed difficulties in responding to peers, scored level ‘4’ for communication strategies; • Except for Heidi, the other five students interviewed felt that their scores were higher than what they had expected; • Whether these high scores would create a false impression on the part of the students about what constitutes effective interaction is an issue that needs to be explored.

21. Findings from the Questionnaire Students responded to a list of 19 statements developed based on initial analysis of the interaction data on 5-pt scales from ‘absolutely agree’ (pt. 5) to ‘absolutely disagree’ (pt. 1). Five highest scoring statements: • I would feel more confident if I could write out my speech and recite it. (4.07) • Presenting my ideas as early as possible would make me feel more relaxed. (3.95) • I should present my ideas first before commenting on peers’ opinions or asking questions. (3.88) • I would like to start the interaction in order to take the first turn of talk. (3.40) • When I cannot think of any good responses or questions for my peers, I would say “I agree with you” or “That ‘s a good idea.” (3.32)

22. Implications • Findings generally resonate with those from He and Dai (2006) – candidates were more concerned with interacting to present a best impression of themselves than with genuine interaction. • Two key features of the talk: • institutionalized and ritualized; and • contrived and collusive in nature.

23. Institutionalized talk • According to Sacks, Schegloff and Jefferson (1974), very little of what we say, the actions we perform, or the order in which we do things in ordinary conversations is determined in advance.

24. Collusive talk • The students worked together to co-construct an interaction based on their understanding of what makes an ideal performance.

25. Dramaturgical Nature of the Talk • A notion proposed by Goffman (1959) – “The presentation of self in everyday life” • interaction is viewed as a “performance,” or a kind of “self-staging” and “self-presentation” before some audience, or significant others. • goal-directed and shaped by the environment and desires of the interactants or “actors” to provide others with “impressions” that are consonant with expectations of the audience as perceived by the actors. • pressure to display idealized conduct, or to make prominent those characteristics that are socially sanctioned

26. Impact on oral communicative competence • As suggested by Heritage (1998), ordinary conversation encompasses a vast array of rules and practices for the pursuit of every imaginable kind of social goal whereas institutional interaction generally involves “a reduction in the range of interactional practices deployed by the participants” and “restrictions in the contexts they can be deployed in” (p.3).

27. Recommendations • Raise students’ awareness of the importance of being an active listeneras an essential quality for being an effective interlocutor; • Engage students in critical analyses of different examples of group interaction data (from the SBA data bank or documented studies), or quasi-authentic interaction data from movies or TV dramas – like a critical ethnographer • Engage students in developing a portfolio (see Riggenbach, 1998) containing situational function-based and task-oriented instances of language use with the support of information technology.

28. Limitations • Single case, single gender • Only reflect what happened with one task type (opinions sharing; divergent in nature?) • Unable to explore the situational thinking of the students

29. References Almasi, J.F., O’Flahavan, J.F., & Arya, P. (2001). A comparative analysis of student and teacher development in more and less proficient discussions of literature. Reading Research Quarterly, 56(2), 96-120. Bachman, L.F., & Palmer, A.S. (1996). Language testing in practice. Oxford: Oxford University Press. Canagarajah, S. (2006). Changing communicative needs, revised assessment objectives: Testing English as an international language. Language Assessment Quarterly, 3(3), 229-242. Fulcher, G. (1987). Tests of oral performance: The need for data-based criteria. ELT Journal, 41, 287-291. Gan, A., Davison, C. & Hamp-Lyons, L. (in press). Topic negotiation in peer group oral assessment situations: A conversation analytic approach. Applied Linguistics. Goffman, E. (1959). The presentation of self in everyday life. Harmondsworth : Penguin. Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters. He, L. & Dai, Y. (2006). A corpus-based investigation into the validity of the CET-SET group discussion. Language Testing, 23(3), 370-401.

30. Heritage, J. (1998). Conversation analysis and institutional talk: Analyzing distinctive turn-taking systems. In S. Cmejrkova, J. Hoffmannova, O. Mullerova & J. Svetla (Eds.), Proceedings of the 6th International Congress of IADA (International Association for Dialog Analysis) (pp.3-17), Turbingen: Niemeyer. Lazaraton, A. (2002). A qualitative approach to the validation of oral language tests. Cambridge: Cambridge University Press. Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters. Pica, T. (1987). “Second-language acquisition, social interaction, and the classroom.”Applied Linguistics, 8, 3-21. Purpura, J.E. (2008). Assessing communicative language ability: Models and their components. In E. Shohamy & N.H. Hornberger (Eds.), Encyclopedia of language and education, 2nd edition, Vol. 7: Language testing and assessment (pp. 53-68). Springer Science+Business Media LLC. Riggenbach, H. (1998). Evaluating learner interactional skills: Conversation at the micro level. In R. Young, & A. W. He (Eds.), Talking and testing: Discourse approaches to the assessment of oral proficiency (pp.53-67). Amsterdam/Philadelphia: John Benjamins Publishing Company. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696-735. Shohamy, E. (1991). Discourse analysis in language testing. Annual Review of Applied Linguistics, 11, 115-131.

31. More proficient: Able to sustain a topic after introduction Able to return to an old topic to maintain topic coherence Able to make intertextual links to the source text and to other related texts to support discussions. Less proficient: Spent more time on task metatalk Unsuccessful in sustaining topics, moving from one topic to another Fewer attempts at intertextual links How more proficient and less proficient students discuss literature in small groups (Almasi, O’Flahavan, and Arya, 2001)

32. Episode 1 – topic shifts (The Case of the Crowing Rooster) Aaron: I think that story is about, um…is about the chicken that the person… ?: [It is a] rooster. Aaron: Yeah, rooster that, um…got stolen, and they, and the guy, I don’t know what his name was, but he came to the house, and he was just looking at the rooster and then he finally went home and then came back. Henna: As I read the story I felt pretty good because it is a mystery, and I like Encyclopedia Brown. Timmy: You already said that. Henna: I know. I was just… Tracy: I don’t agree with Aaron because no one stole the rooster, stole that rooster because he just had [inaudible] the person who wanted him to crow did not, but he did not make him crow just by the machine. A machine cannot make a bird crow. Tracy: [nods head positively] U-hmm. Brian: No, it was just made of cardboard.

33. Episode 2a – linking topics (Soup’s New Shoes) Aaron: On page 279 again [it said] “the left shoe played one turn and the right played another.” Brian: S0? Aaron: It’s two different melodies on, on one shoe. Brian: [reading from text] “orchestra with the violins playing and the drums playing on the other side.” Derek: He’s probably just kicking his shoes up against the wooden floor. Aaron: Or it’s like so shiny, it’s like so new that when he steps it squeaks. Brian: ‘Cause they’re, like, new shoes, when they’re broke in they don’t squeak, and they’re nice and comfortable, but when you first get’em, they’re all tight and they squeak. Kinda bend ‘em a little so you can break ‘em in.

34. Episode 2b – linking topics (Soup’s New Shoes) – Aaron linking the topic to an earlier topic Aaron: But since they’re new shoes, how come they, like, break so easily? Brian: Because they were big shoes, bigger than his feet. They were bigger than his feet. Derek: He probably tore ‘em on the fence because of a nail stickin’ out or something. Brian: It said somewhere that they were big, big for his feet.