Multilingual Platform for CALT (Computer Adaptive Language Testing) in six languages

1. Hellenic Ministry of Culture, Education and Religious Affairs National & Kapodistrian University of Athens, Aristotle University of Thessaloniki & Computer Technology Institute of the University of Patras The National Foreign Language Exams for the State Certificate of Foreign Language Proficiency in Greece: The KPG examination suite http://rcel.enl.uoa.gr/kpg Multilingual Platform for CALT (Computer Adaptive Language Testing) in six languages Professor Bessie Dendrinos, President of the KPG Examination Board

2. National Strategic Reference Framework (NSRF) – 2007-2013

3. e-KPG includes • The multilingual CALT platform • Platform for the e-training of oral examiners and script raters (http://galaxy.cti.gr/e-kpg-devel23/) • The KPG e-school (http://rcel.enl.uoa.gr/kpgeschool/)

4. Multilingual CALT platform

5. Multilingual standards • Both the computer-based and the paper-based KPG exams contain common: • proficiency scaling (aligned to the 6 level scale of the Council of Europe) • illustrative descriptors (aligned to the CEFR) • accompanying criterial features • (constructivist) view of language-in-use

6. Multilingual framework • The exam suite is based on a generic (portable across languages) methodological framework suitable for describing language performance across proficiency levels.

7. Multilingual operationalization • And… across languages, the exams: • test common competences and strategies • are regulated by common specifications • use a common test-task typology

8. Basic tools of the CALT platform • Authoring tool • Piloting tool • Test player

9. Authoring tool roles Diagram showing the flow of roles, and work required to insert test tasks and give final approval

10. Testing understanding • The KPG CALT is developed to test both comprehension and production: • The first part tests: • reading comprehension • listening comprehension • language awareness

11. The task-based CALT • The CALT is not item but task based. • The smallest unit of the comprehension part is a task, not an item. • Three test tasks comprise a ‘testlet’. • The candidate’s performance in each testlet determines what testlet level s/he will be served next.

12. Composition of testlets • Each testlet is composed of three tasks: • One reading comprehension task • One language awareness task • One listening comprehension task • There is a common (across languages) formula as to the mix of the types of tasks that each testlet contains.

13. CALT task typology • The KPG CALT contains 10 task types giving us in all 12 types of testlets. • Each task is composed of 5 items as follows: • items testing comprehension of a source (written or audio) text • items testing awareness of correct and appropriate use of language

14. CALT testlet typology • 3 tasks comprise a testlet • 3 testlets comprise a testing unit • 3-5 testing units comprise a complete • a complete test for each of the lower levels (Α1-Β1) contains 3 testlets (45 items in total) • a complete test for each of the higher levels (B2-C2 level) contains 4 testlets (60 items).

15. Starting the test and moving through it • The candidate chooses the level of the testlet s/he wants to start with and then if • s/he is successful on 50% of the items, the next testlet is of the same level • his/her success is below 50%, the next testlet is of lower level • If his/her success is above 50%, the next testlet is of higher level

16. Digital e-test typology

17. Reading comprehension (French B2)Multiple choice (MC)-tick

18. Reading comprehension (German C1)Multiple choice (MC)-tick

19. Reading comprehension (English C2)Multiple choice (MC)-tick

20. Language awareness (Italian C1)Multiple match-insert missing item

21. Language awareness (Spanish A2) Matching-drag-’n-drop (Q&A)

22. Language awareness (English B1)Fill in - write in a word

23. Language awarness (German A1) Match-drag’n-drop

24. Language awareness (English B1)writing in a word

25. Listening comprehension (A1) Multiple match (drag-’n-drop letter/number)

26. Listening comprehension (Italian B1) Multiple match (drag-’n-drop letter/number)

27. Testing production • Production performance is also computer based • The tests on reading and listening comprehension determine the level of the speaking and writing tests in which candidates will be asked to participate

28. Testing leveled performance Testing speaking & oral mediation Testing writing & written mediation

29. Oral production • Similar test tasks across languages are aimed to assess context specific: • oral performance, stimulated by visual or multimodal texts in the target language • oral mediation performance, stimulated by texts in the home language

30. Speaking task (German B1) Diese Fotos stammen aus Ihrem elektronischen Familienalbum. Welche Personen sind auf den Fotos abgebildet? Wann und warum wurde jedes einzelne Foto gemacht?

31. Speaking task (Italian B2) A tuo parere, prima della ripresa di queste due fotografie, che cosa può essere accaduto? E cosa hanno in comune i due avvenimenti?

32. Speaking task (Spanish B2) Dinos cómo crees que se sienten estas personas y por qué.

33. Mediation in English (C2) Tell your friend Edward about two games he knows almost nothing about: backgammon and jenga. Click here to speak

34. Writing production • Similar test tasks across languages are aimed to assess context specific: • writing performance, stimulated by cues, visual or multimodal texts in the target language • written mediation performance, stimulated by texts in the home language

35. Writing task (French B2)

36. Writing task (Spanish A2) Escribe un e-mail (60-70 palabras) a tu amiga Valentina para hablarle deun bonito lugar de tu ciudad donde estuviste ayer. Cuéntale:1. Qué tipo de lugar era2. Dónde está exactamente3. Qué hiciste allí

37. Writing task (English A1)

38. Written mediation in Italian (C1)

39. Test task piloting

40. The Piloting Tool_1 • It’s a management system linked to the Authoring Tool. • Its purpose is to manage trial runs (or the piloting) of each test task, in real-time conditions, with a view to examining the task’s validity.

41. How does the PT operate? • Groups of tasks are packed in groups and piloted by sample population of candidates • Depending on how well the tasks have ‘behaved’, they may be considered: • good enough to be inserted in the Test Bank • partially problematic and in need of amendments • seriously problematic (warranting serious modification or ditching)

42. The piloting tool_2 • The PT has been designed to measure basic indices with an inbuilt statistics system • The functions of the PT are available to the members of the language-specific author team who have been assigned the role of ‘task evaluator’ and/or the role of ‘Administrator’.

43. Managing the test packs They aremanaged as follows: • The user ‘characterizes’ a total number of tasks on the basis of the task. typology and creates test packages • Token are created for the sample population who will use them. • Indices of statistical interest are worked out for results and ultimately findings.

44. www.rcel.enl.uoa.gr