Introduction to the Specification Phase

1. Introduction to the Specification Phase Relating language examinations to the Common European Framework of Reference for Languages José Noijons APEOICVA/ECML Valencia, 6-7 February 2009

2. Aims of today’s session on Specification • Raising awareness of good practice in linking to the CEFR • Defining minimum standards in linking • Providing practical support in linking

3. Raising Awareness • of the importance of good content analysis of language examinations; • of the CEFR, especially its descriptor scales; • of the rationale for relating language examinations to an international framework like the CEFR; • of ways in which the CEFR can be exploited in planning and describing language examinations

4. Defining minimum standards in terms of • the quality of content specification in language examinations; • the process of linking examinations to the CEFR.

5. Providing practical support to • complete the proposed content analysis and linking process; • provide evidence of internal consistency and construct validity; • report a claim that makes the results of the examination in question more transparent to both the users of examination results and to test takers themselves.

6. Four steps in the specification phase • assuring adequate familiarisation with the CEF (cf. yesterday); • analysing the content of the examination or test in question in relation to the relevant categories of the CEFR; should an area tested not be covered by the CEFR, the user is asked to describe it; • profiling the examination or test in relation to the relevant descriptor scales of the CEFR on the basis of this content analysis; • making a first claim on the basis of this content analysis that an examination or test in question is related to a particular level of the CEFR.

7. Specification phase: three types of activity • familiarisation activities (cf. yesterday) • filling in a number of checklists with details about the content of the language examination; • using relevant CEFR descriptors to relate the language examination to the levels and categories of the CEFR.

8. (1) Specification process gives the opportunity to • increase the awareness of the importance of a good content analysis of examinations; • become familiar with and use the CEFR in planning and describing language examinations; • describe and analyse in a detailed way the content of an examination or test; • provide evidence of the quality of the examination or test;

9. (2) Specification process gives the opportunity to • provide evidence of the relation between examinations/tests and the CEFR; • provide guidance for item writers; • increase the transparency for teachers, testers, examination users and test takers about the content and quality of the examination or test and its relationship to the CEFR.

10. Validity & Reliability • A test should test what it is supposed to test • Use test grids • Find experts for developing the test • A test should consistently measure what it is supposed to measure. • Organize experts’ work in a team • Assign screeners • Use data analysis

11. Objectivity • Need for unambiguous answers to questions • Use written tests • Organize discussion among experts • Use marking schemes

12. Acceptability • The test should be seen to reflect desired and relevant behaviour • Involve all stakeholders • Take educational reform into account • Develop methods for standardisation • Organize research on psychometrics

13. Efficiency • There should be a positive correlation between information to be gained from the test and test length • Use written tests • Use short questions • Limit necessary stimulus material

14. General Descriptions: Forms A1-A8 • Test Purpose • Development Process • Marking Procedures • Standards • Reporting • Analysis and Review Procedures • Rationale for decisions • Initial Estimations of CEFR levels

15. Communicative Language Activities:Forms A9 - A18 Form Communicative Language Activity A9 Listening Comprehension A10 Reading Comprehension A11 Spoken Interaction A12 Written Interaction A13 Spoken Production A14 Written Production A15 Integrated Skill Combinations A16 Integrated Skills A17 Spoken Mediation A18 Written Mediation

16. Communicative Language Competences: Forms A19 – A22 A19 Reception A20 Interaction A21 Production A22 Mediation

17. Standardisation Phase • Both benchmarking and standard setting are procedures which require group decisions, which in turn have to be carefully prepared by appropriate training.

18. Benchmarking in direct tests • In holistically rated tests, the judgment on the level (the six CEFR levels) is direct, and therefore it is important to assist raters in giving valid judgments. The main tool used for this special type of standard setting is called benchmarking. • Benchmarking involves providing one (or more) typical sample(s) to illustrate performance at a given level both for standardisation training and to serve as a point of reference in making future decisions about performances of candidates.

19. Standard Setting in indirect tests • For tests with a numerical score, performance standards have to be set. A performance standard is the boundary (cut-off score) between two levels on the scale reported by a test. A cut-off score of 30 says that a score of 30 or more on the tests grants a level of a particular level (e.g. B1) or higher, while a lower score points to a level lower than the level of the cut-off score (here: B1). • The process to arrive at a cut-off score is commonly referred to as standard setting. In the case of receptive skills (reading and listening) or underlying competences (grammar, vocabulary), cut-off scores need to be decided upon.