1 / 54

Bánóczy Erika Benke Eszter Budapesti Gazdasági Főiskola Nyelvvizsga és Továbbképző Központ

Into Europe - European Standards in Language Assessment Conference 9-10 February 2006. The BGF Linking Experience. Bánóczy Erika Benke Eszter Budapesti Gazdasági Főiskola Nyelvvizsga és Továbbképző Központ.

falala
Download Presentation

Bánóczy Erika Benke Eszter Budapesti Gazdasági Főiskola Nyelvvizsga és Továbbképző Központ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Into Europe - European Standards in Language Assessment Conference 9-10 February 2006 The BGF Linking Experience Bánóczy Erika Benke Eszter Budapesti Gazdasági Főiskola Nyelvvizsga és Továbbképző Központ

  2. Harmonizing national examinations with the Common European Framework of Reference (CEF) • Background to the project • The origins of the existing level system • The Strasbourg project • Case Studies (WIP) • Objectively scored tasks • Subjectively scored tasks • Examiners’ familiarity with the CEF • LSP and the CEF

  3. Background Language examination levels according to the Accreditation manual (1999), p.36.

  4. Background Intermediate language proficiency corresponds to the intermediate level as suggested by the Council of Europe intermediate level (Vantage Level) and extends into the lower part of advanced level (Operational Proficiency). Language examination levels according to the Accreditation manual (1999), p.33.

  5. Language examination levels accredited in 2000

  6. Background Modified language examination levels according to the new Accreditation manual (2004), p.25.

  7. Background Language examination levels according to the Accreditation manual (2004) p.8. „The state-accredited examination system intends to harmonize the theory and practice of national and international (European) language examinations. This effort is manifest … in the transferability of the Hungarian three-level system and the more detailed and up-to-date level system of the Council of Europe. (Work towards harmonization is still in progress.)

  8. Levels to be harmonized ? Advanced Intermediate Elementary ? ?

  9. Aim of the BGF project • Piloting the Manual • Empirical validation of original levels • Training of experts • Modification of existing levels • Benchmarking existing performance samples • Creating new tasks based on new levels

  10. Seminal documents and materials • Relating Language Examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment. Manual (2003) • Reference Supplement (2004) • Nyelvvizsgák illeszkedése a Közös Európai Referenciakerethez(2005) • Case Studies • Calibrated performance samples

  11. Procedures to relate examinations to the CEF • Familiarisation with the CEF • Specification • Standardisation • Empirical validation • Reporting the results

  12. Work completed so far • 3 stages • English project • participants (anchor persons) • procedures • German project • large-scale examiner training • time and materials used

  13. Breakdown of working hours in the first stage (1)

  14. Breakdown of working hours in the first stage (2)

  15. Technical equipment used

  16. Objectively scored tasks • Familiarisation with CEF and DIALANG scales • Study of calibrated sample tasks (if and where available) • Standard setting – modified Angoff method • Collation and tabulation of results • Comparison of qualitative and quantitative data

  17. Standard-setting grid for a reading comprehension task

  18. Data collection 1

  19. Data collection 2

  20. Data analysis • Method applied • Comparison of qualitative and quantitative data

  21. Problematic item Alpha if item deleted .787 Alpha = .771

  22. External validation • Criterion test • BEC Vantage level • Content analysis to establish comparability – (ALTE content analysis checklist) • Correlation • comparison of statistical data

  23. Results – descriptive statistics

  24. Results - correlation • r=0.59 (p< 0.01) after correction for attenuation r=0.9 • considerable similarity between empirical and intuitively set levels of difficulty

  25. Subjectively scored tasks • Oral proficiency tests • watching videotaped calibrated performance samples • benchmarking own samples • comparison of CEF based and original scores • Writing • benchmarking own samples • comparison of CEF based and original scores

  26. Writing – selection of criteria

  27. Results so far • benchmarked intermediate performance samples in six languages • accumulated experience and emerging expertise – fairly thorough familiarity with the CEF level system

  28. Survey on familiarity with the CEF • 82 respondents • 35 English • 21 German • 10 French • 10 Spanish • 4 Italian • 2 Russian • 18 yrs average teaching experience • questionnaire • 15 statements • 5 point Likert-scale

  29. Familiarity with the CEF - questionnaire • I have heard of the CEF before. • I know the Hungarian version of the CEF. • I know the language version of the CEF I teach. • I have my own copy of the CEF. • I know the level system used in the CEF. • I am familiar with the descriptors used in the CEF.

  30. Use of the CEF - questionnaire • I use the CEF for defining proficiency levels. • I use the CEF in curriculum design. • I use the CEF letter level system when choosing course books. • I use the diagnostic self-assessment scales.

  31. The benchmarking workshop and the future - questionnaire • I find the sorting tasks useful in getting familiar with the CEF levels. • The CEF descriptors are easy to apply. • It is possible to realistically assess students’ language proficiency with the help of the CEF scales. • I find it feasible to harmonize our existing system of levels with the CEF levels. • The CEF levels are becoming more and more common in the professional discourse/communication of those involved in education (teachers, students, parents, employers etc.).

  32. Top 3 • I find the sorting tasks useful in getting familiar with the CEF levels. ( =4.1) • I know the level system used in the CEF. ( =3.9) • I find it feasible to harmonize our existing system of levels with the CEF levels. ( =3.8)

  33. Bottom 3 • I use the CEF in curriculum design.( =2.3) • I have my own copy of the CEF.( =2.3) • I use the diagnostic self-assessment scales. ( =2.2)

  34. ESP and the CEF

  35. ESP and the CEF Some intriguing issues: • L(SP) competence • definitions of LSP • Swales, 1985 • Strevens, 1988 • EAP-EOP (Robinson, 1991) • general-specific continuum (Dudley-Evans & St John, 1998)

  36. ESP and the CEF Some intriguing questions: Which ESP/LSP definition is our system based on? How are ESP/LSP skills different from EGP skills?

  37. ESP and the CEF General purpose language testing vs LSP testing: • authenticity of task • interaction between language knowledge and specific purpose content knowledge (Douglas, 2000)

  38. ESP and the CEF • specific aspect of the specifications (‘specific specifications’) • specific aspect of the operationalisation of the construct • specific aspect of the tasks

  39. ESP and the CEF special* job* field* vocation* terminology*

  40. ESP and the CEF • Table 1. Common Reference Levels: global scale ( p.24) B2 Can understand the main ideas of complex text on both concrete andabstract topics, including technical discussions in his/her field ofspecialisation.

  41. ESP and the CEF • Table 2. Common Reference Levels: self-assessment grid (pp. 26-27) A2 Spoken Production I can use a series of phrases and sentences to describe in simple terms … my present or most recent job. B1 Spoken Interaction I can enter unprepared into conversation on topics that are familiar … (e.g. work). C1Reading I can understand specialized articles … even when they do not relate to my field. C2 Reading I can read with ease virtually all form of written language …such as … specialised articles.

  42. ESP and the CEF Sustained monologue (p.59) A2 Can describe his/her family, living conditions, educational background, present or most recent job. Creative writing (p.62) B1 Can write about everyday aspects of his/her environment, e.g. … a job… inlinked sentences.

  43. ESP and the CEF • Overall listening comprehension (p.66) B2 Can understand standard spoken language, live or broadcast, on both familiar and unfamiliar topicsnormally encountered in personal, social, academic or vocational life. C1 Can understand enough to follow extended speech on abstract and complex topics beyond his/her own field …

  44. ESP and the CEF Listening as a member of a live audience (p. 67) C2 Can follow specialised lectures and presentations employing a high degree of colloquialism, regional usage or unfamiliar terminology. Overall reading comprehension (p.69) A2Can understand short, simple texts on familiar matters of a concrete type which consist of highfrequency everyday or job-related language. B1Can read straightforward factual texts on subjects related to his/her field and interest with asatisfactory level of comprehension. C1 Can understand in detail lengthy, complex texts, whether or not they relate to his/her own area ofspeciality, provided he/she can reread difficult sections.

  45. ESP and the CEF Reading for information and argument (p.70) B2Can obtain information, ideas and opinions from highly specialised sources within his/her field.Can understand specialised articles outside his/her field, provided he/she can use a dictionaryoccasionally to confirm his/her interpretation of terminology. Reading instructions (p.71) B2Can understand lengthy, complex instructions in his field, including details on conditions and warnings, provided he/she can reread difficult sections. C1 Can understand in detail lengthy, complex instructions … whether or notthe instructions relate to his/her own area of speciality…

  46. ESP and the CEF Overall listening comprehension (p.66) B1Can understand straightforward factual information about common everyday or job related topics… B2Can understand standard spoken language, live or broadcast, on both familiar and unfamiliar topics normally encountered in personal, social, academic or vocational life. C1 Can understand enough to follow extended speech on abstract and complex topics beyond his/her ownfield

  47. ESP and the CEF Understanding a native speaker interlocutor (p.75) C1 Can understand in detail speech on abstract and complex topics of a specialist nature beyond his/herown field … C2 Can understand any native speaker interlocutor, even on abstract and complex topics of a specialistnature beyond his/her own field …

  48. ESP and the CEF A2 B1 job, work, B2 C1 own field, speciality, specialised, C1 C2 beyond own field

  49. Useful/beneficial experience • Internal validity of tests is vital for external validation • continuous internal validation • Ongoing local and global validation

More Related