The International Legal English Certificate Issues in Developing a Test of English for Specific Purposes

1. The International Legal English Certificate Issues in Developing a Test of English for Specific Purposes David Thighe, Cambridge ESOL EALTA Conference Krakow, May 2006

2. Outline • What is ILEC? • Features of ESP tests • Continuum of Specificity • Evidence from the Development Practice • Evidence from the Trialling stages • Future Directions

3. What is ILEC? • A test for law students & newly qualified lawyers • A test that assesses language skills in a legal context • Set at CEFR levels B2 & C1 • Recognised by leading associations of Lawyers

4. A definition of specific purpose assessment …. One in which test content and methods are derived from an analysis of a specific purpose target language use situation (TLU), so that test tasks ….are authentically representative of tasks in the target situation, allowing for an interaction between the test taker’s language ability and specific purpose content knowledge, on one hand, and the test tasks on the other. Douglas (2000: 19)

5. Revisiting Douglas’ definition of LSP tests • All tests can be seen as lying on a specificity continuum, between the highly specific and the general purpose. • This continuum is multi-componentialand includes the twin aspects of authenticity – situational and interactional. O’Sullivan (2006)

6. Generalisability Increase in test of content knowledge PET, FCE, CAE etc BULATS BEC ILEC? Specificity – a continuum A test of English for air-traffic controllers

7. CLAIM JUSTIFICATION OF CLAIM SUBSTANTIATING CLAIM Where does ILEC go on this continuum? • Test purpose • Needs Analysis/Viability Study • Development practice • Working together with subject experts at the planning, design and development stages • Evidence from Candidate Performance • the trialling stage on authenticity and specificity

8. Test Purpose To assess candidate’s ability in English in an international legal environment To allow candidates to demonstrate to employers their ability to communicate in English with clients and colleagues To allow candidates to demonstrate to educational institutions their ability to follow a course of legal study in English

9. Sources Items produced and vetted Test construction Tests for vetting Development practiceCambridge ESOL and External Content Experts External content experts Cambridge ESOL Item Writers and Chair

10. Development PracticeTest Content • Reading, Listening, Writing and Speaking components • Tasks based on law-related texts and topics • Set in the context of International commercial law

11. Evidence from ILEC TrialSpecific Issues • Authenticity of components • Specificity of ILEC components • Reliability issues • Practical constraints e.g. time, compulsory/optional parts, length of paper etc

12. Sample and Instruments • 320 candidates from 12 centres worldwide • Candidates completed • ILEC Reading, Writing and Listening papers • General Placement Test • Information Sheets • Questionnaires • Instructors questionnaires

13. Table 6 Q7 Candidate Questionnaire: On the Reading paper the topics are authentic (I or my colleagues may be expected to cover these topics in our legal profession). Table 7. Q8 Candidate Questionnaire: On the Reading paper the texts and language are authentic ( I or my colleagues may be expected to read similar texts in our legal profession). Trial Findings: Authenticity • Similar results for other papers • Similar results for instructors

14. Trial Findings: SpecificityCandidate Performance Table of Comparison of scores of candidates with or without professional experience in GPT, Reading, Listening and Writing

15. Trial Findings: Reliability Cronbach’s Alpha = 0.81 to 0.89

16. Candidate Questionnaire Approx 45% responded that they strongly agree or agree that there is enough time, 33% strongly disagree or disagree, 22% indicated no opinion Shadowing of candidates taking Reading and Writing components All candidates shadowed completed components, 30% writing candidates used full time allowed. 70% candidates completed tests in order of sequence in paper Trial Findings: Time Constraints

17. ILEC Writing – Compulsory or Optional Part 2 • The ILEC Writing Test was offered to a group of trial candidates as two versions. • Version 1: Both parts compulsory (56 cands) • Version 2: Part 2 gives a choice of 3 options (48 cands) Table: comparison of mean scores in both versions of writing trial paper

18. ILEC: Future Directions • Evidence from Live Data on specificity and the role of content knowledge in the test • Performance of those with and without work experience • Using expert judgement to provide further evidence of Task Specificity

19. In Summary • Outline of ILEC • Defining features of ESP tests • Purpose from Viability Study • Test Development Process with External Expert • Empirical Evidence from ILEC Trial • Future Research http://www.cambridgeesol.org/exams/ilec.htm thighe.d@ucles.org.uk

20. References • Douglas, D. 2000. Assessing Languages for Specific Purposes. Cambridge: Cambridge University Press. • O’Sullivan, B. 2006. Issues in Testing Business English: the revision of the Cambridge Business English Certificates SILT Volume 17. Cambridge: Cambridge University Press. • Weir, C. 2005. Language Testing and Validation: an Evidence-based Approach. Hampshire: Macmillan