Improvements in the cognitive and context validity dimensions of academic speaking tests

1. Improvements in the cognitive and context validity dimensions of academic speaking tests Nükte Durhan METU, Northern Cyprus Campus, School of Foreign Languages (Ankara, 30 May 2012)

2. OVERVIEW • General look at socio-cognitive framework • Cognitive validity (theory-based) and routine skills • Needs analysis results of the Academic Speaking Skills (211) Course • Description of the speaking tasks in the course and evaluation from cognitive validity aspects • Context validity: considering its relevant aspects and the speaking tasks • Areas that need work: scoring validity

3. Performance Testing Performance tests are characterised by the relationship of the test task to the world beyond the test. Ideally tasks are… • direct and • authentic • situationally authentic • interactionally authentic But a test can never fully represent reality.

4. Oral Communication: direct spoken language activities • We want candidates to… • perform relevant language tasks • adapt their speech to circumstances • make decisions under time pressure • implement them fluently • make any necessary adjustments as unexpected problems arise

5. A socio-cognitive framework

6. Test Taker Context Cognitive Response Scoring Validity Score/Grade Criterion- Related Validity Consequential Validity A socio-cognitive framework

7. Sources of validity evidence • Test-taker characteristics • Context focused • Cognitive focused (theory-based) • Scoring focused • Criterion focused (external) • Consequentially focused

8. What does ‘socio-cognitive’ mean? • The abilities to be tested are demonstrated by the mental processing of the candidate (the cognitive dimension) • The use of language in performing tasks is viewed as a social rather than a purely linguistic phenomenon (the socio- dimension) • Test construct resides in interactions of underlying cognitive ability and context of use, i.e. socio-cognitive model

9. Cognitive validity • Executive processes • Conceptualizer • Pre verbal message • Linguistic formulator • Phonetic plan • Articulator • Overt speech • Audition • Speech comprehension (Weir,2005) • Executive resources • Language knowledge • Grammatical • Textual • Functional • Social • Content knowledge • Internal • External

10. How speakers organize in routines what they want to communicate • Routine Skills (Bygate, 1987) • Informational Routines • expository routines: involve factual information • evaluative routines: involve drawing of conclusions, expression of reasoning; explanation, predictions, justifications, preferences and decisions

11. How speakers organize in routines what they want to communicate • Routine Skills (Bygate, 1987) • Interactional Routines : • telephone conversations • interviews • meetings • discussions • decision making tasks

12. How speakers organize in routines what they want to communicate • Routine Skills (Bygate, 1987) • Interactional Routines :They are assisted by… • interaction management skills involving content-focused agenda management and interaction-focused turn-taking • Improvisational Skills • brought into play when an interaction falters

13. Old Syllabus • Informative Speech (individual) (10%) • Persuasive Speech (group) (20%) • Final Presentation (individual) (30 %) • No assessment of discourse and strategic competences

14. Results of theNeeds Analysis

15. Speaking Skills Participants want more of the following: • More focus on spontaneous speech • A variety of speaking activities • More presentations with smaller percentages • Different types of presentations • Focus on fluency • Interactive activities • Creativity and individuality • Speech critiques

16. Speaking activities suggested • Debate • Role plays • Discussions • Impromptu speech • Pronunciation activities • Critical thinking activities • Asking & answering questions

17. Description of thespeaking tasks in the course

18. Team Presentation (10%) • Reporting the team’s observation of a store’s merchandising strategies and store design • Organizing and sharing points in the group • Presenting findings • Giving evidence from the observations • Evaluating the effectiveness of the strategies

19. Informative Presentation (10%) • To inform and enlighten the audience about a given topic on the general theme of technology

20. Current Events Presentation (5%) • Choosing a news item related to art, analyzing it and presenting the analysis to class • Includes: • justification of the selection of the item • reaction to points and style • evaluation of the ideas

21. Final Presentation (30%) • Giving a balanced view of the selected topic (related to the five themes in the book) • Includes: • extensive literature research • reaction to the presented material/discussion or evaluation of the topic

22. Going one step further (15%)activities • Debates • Discussions • Panel discussion • Role plays • Conflict resolution tasks Three different activities should be carried out in class throughout the semester.

23. Context Validity

24. Context validity • Setting: task • Purpose • Response format • Known criteria • Weighting • Order of items • Time constraints • Setting: administration • Physical conditions • Uniformity of administration • Security (Weir,2005) • Demands: task • Linguistic • Channel • Discourse mode • Text length • Nature of information • Topic familiarity • Lexical range • Structural range • Functional range

25. Context Validity In Action 1. Rubric • Is the rubric accurate and accessible? 2. Purpose • Is the purpose of the test made unequivocally clear for the candidate?

26. Context Validity In Action 3. Weighting • Are any weightings for different test components adequately justified? 4. Known criteria • Are the criteria to be used in the marking of the test explicit for the candidates and the markers?

27. Context Validity In Action 5. Time constraints: particularly important for speech as normal speech takes place under time pressure • Is the timing for each task (e.g. preparation and completion) appropriate?

28. Context Validity In Action 6. Discourse mode: Reciprocity conditions are concerned with the dimension of interpersonal interaction, the relation between speaker and listener. • Are there reciprocal exchanges where speakers have to adjust vocabulary and message and take each other’s contributions into account?

29. Context Validity In Action 7. Type of information • Is the type of information appropriate for the target situation requirements of the students being tested? 8. Content knowledge • Is the content sufficiently familiar to candidates so that they have sufficient existing schemata to enable them to deploy appropriate skills and strategies? (Urquhart & Weir, 1998)

30. Context Validity In Action 8. Content knowledge “every attempt should be made to allow background knowledge to facilitate performance rather than allowing its absence to inhibit performance” (Alderson, 2000)

31. Context Validity In Action 9. Linguistic knowledge • Lexical range • Are the lexical items in the test both in input text and required as output appropriate for the level of the candidates? • Syntax Are the grammatical items in the test both in input text and required as output appropriate for the level of the candidates?

32. Context Validity In Action 9. Linguistic knowledge • Function: is a term used to describe the illocutionary force of what is said. • Are the functions in the test both in input text and required as output appropriate for the level of the candidates? The variety of tasks with different functional purposes and increasing number of tasks improve the validity.

33. Areas that need work: scoring validity • Rating • Criteria / rating scale • Rating procedures • Rater selection • Rater training • Standardisation • Moderation • Rating conditions • Statistical analysis

34. Thank you dnukte@metu.edu.tr