1 / 63

Educating the Stakeholders

Educating the Stakeholders. ECOLT 2008 November 7, 2008 ILR Testing Committee. Overview of Presentation. Intuitive Language Testing Testing Misconceptions Educating Stakeholders Continuing Challenges. Why educate stakeholders?.

pello
Download Presentation

Educating the Stakeholders

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Educating the Stakeholders ECOLT 2008 November 7, 2008 ILR Testing Committee

  2. Overview of Presentation • Intuitive Language Testing • Testing Misconceptions • Educating Stakeholders • Continuing Challenges

  3. Why educate stakeholders? • Consequences of language tests reach far beyond the testing sessions. • Stakeholders • Examinees • Language teachers • Supervisors of linguists • USG agencies • Policy makers • Citizens

  4. What is limited proficiency? ILR Level 2 Proficiency in what? Speaking What type of Arabic? Modern Standard Arabic “Five years after Arab terrorists attacked the United States, only 33 FBI agents have even a limited proficiency in Arabic …” Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

  5. Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01 Were all the agents tested? Most agents do not have a language requirement. What is meant by “any familiarity?” Speaking Modern Standard Arabic What does a score of zero mean? “Counting agents who know only a handful of Arabic words -- including those who scored zero on the standard proficiency test -- just 1 percent of the FBI’s 12,000 agents have any familiarity with the language, the statistics show.”

  6. Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01 What is ‘advanced professional proficiency?’ ILR Level 4 The tests were administered by the ILR? The ILR does not administer tests. “Youssef … is one of only six FBI agents who scored a 4 for ‘advanced professional proficiency’ in Arabic on standardized speaking tests administered by the Interagency Language Roundtable for federal agencies.”

  7. Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01 Who is interpreting the language testing statistics? An attorney for a former linguist, suing the FBI. What language test was given? Speaking, not Translation Who do agents rely on? Language Analysts and Intelligence Analysts Tested in Reading, Listening, Translation, Speaking “Youssef’s attorney, Stephen M. Kohn, said the statistics indicate that most FBI agents have no way to gauge the accuracy of translated materials and must rely on linguists or other third parties for their information.”

  8. Intuitive Test Theory Braun and Mislevy • General testing P-prims • A test measures what it says at the top of the page. • A test is a test is a test. • A score is a score is a score. • Any two tests that measure the same thing can be made interchangeable with a little “equating” magic.

  9. P-Prims, continued • You score a test by adding up scores for items. • 70% is passing – on any and every assessment. • You can tell if an item is good by looking at it. • Multiple-choice questions only measure recall. • Multiple-choice tests equal standardized tests equal high-stakes tests.

  10. ILTA Code of Ethics Principle 7 • Language testers in their societal roles shall strive to improve the quality of language testing, assessment and teaching services, promote the just allocation of those services and contribute to the education of society regarding language learning and language proficiency.

  11. ILTA Code of Ethics Principle 8 Language testers shall be mindful of their obligations to the society within which they work, while recognising that those obligations may on occasion conflict with their responsibilities to their test takers and to other stakeholders.

  12. Overview of Presentation Intuitive Language Testing Testing Misconceptions Educating Stakeholders Continuing Challenges

  13. Quote 1 Hassan (Foreign Service Specialist): “I worked for an NGO in Mosul and helped several US-based companies set up their computer systems. I constantly talked to vendors, made arrangements to access the appropriate routers, negotiated bandwidth for their Internet hubs, the works! In my spare time, I talked to my Iraqi buddies about all sorts of things. They were great at tipping me off whenever good stuff came at the local grocery store. I thought I was a 5 in speaking Iraqi Arabic. You scored me a 2. Need I tell you what I think about your test?”

  14. Testing Misconception A 3 is a 3 is a 3. • Proficiency testing does not equal achievement or performance testing. • Scores in one skill do not transfer across other skills. • A 3 given by a non-USG testing organization may not equal a 3 given by a USG agency.

  15. Educating Stakeholders: Types of Tests • Achievement = Memorized responses using the content of a specific textbook or curriculum. • Performance =Rehearsed ability to communicate in specific, familiar settings. • Proficiency = Unrehearsed general ability to accomplish communication tasks across a wide range of topics and settings. The following self assessment used by permission, Dr. Ray Clifford, ACTFL 2007

  16. A Self Assessment Quiz • You will be asked about 6 different testing purposes. • For each of those test purposes, which type of test would you choose? a. Achievement b. Performance c. Proficiency

  17. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course?

  18. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses?

  19. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course?

  20. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks?

  21. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be the NATO spokesperson on a news show with a “hostile” moderator?

  22. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be the NATO spokesperson on a news show with a “hostile” moderator? • To document a language ability rating for a person’s military or civilian personnel record?

  23. Achievement, Performance, or ProficiencyWhich test type did you choose? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be the NATO spokesperson on a “hostile” news show? • To document a language ability rating for a person’s military or civilian personnel record?

  24. Educating Stakeholders:FSI Examinees Information on the Skill Level Descriptions available online at www.govtilr.org Self appraisals are suggested/required for some language positions Formal presentations on language testing given to new entrants

  25. FSI Stakeholder Education Before Speaking Self-Assessment: 85% of applicants with self-assessed proficiency tested at 42% passing rate. After Speaking Self-Assessment: 61% of applicants with self-assessed proficiency tested at 58% passing rate.

  26. Quote 2 Nasra (Swahili language applicant): “Oh yeah, that test won’t be a problem at all. It’s my mother tongue, so of course I’m a Level 5. I speak Swahili all the time. It’s not a big deal. You can test me right now, if you want. I’m ready.”

  27. Testing Misconception All native speakers are Level 5. • Native speaker as a social variable. • Native speaker as the ideal.

  28. Typical Scales Satisfactory Good Excellent Poor Fair C B A F D 0 50 100

  29. FBI OPI Data • Testing Native Speakers • 80% + of FBI Linguists are native or heritage speakers of the language. • Native Speaker Self Appraisal Scores • Native Language • Level 5: 70% • Level 4/4+: 10% • Level 3/3+: 20% (largely heritage speakers) • English • Level 5: 51%

  30. Quote 3 James (Manager of a new national security initiative): “We need at least 10 linguists dedicated to this assignment in Arabic alone. Since this matter is of highest importance, we need them all to be Level 5. It’s important that we have the best people on the assignment.”

  31. Aleem: heritage speaker of Saudi Arabic went to Arabic classes at a private language school Language school-Listening: 4 Language school-Reading: 4 Language school-Speaking-MSA: 3+ Richard: native speaker of English learned Arabic for 4 years while in college, (1 year in Qatar) DLPT-Listening: 3 DLPT-Reading: 3 OPI-Speaking-MSA: 2 Who would you hire…As a Foreign Service Officer?

  32. Adrienne: worked at an Iraqi refugee program for 1 year DLPT-Listening: 2+ DLPT-Reading: 2 OPI-Speaking-MSA: 2+ Consecutive Interpretation Test-Iraqi/English: 2 Catherine: worked as a translator for the United Nations for 2 years DLPT-Listening: 2+ DLPT-Reading: 4 OPI-Speaking-MSA: 3 Translation Test: 3 Who would you hire…As a Military Interpreter?

  33. Jennifer: native speaker of English DLPT-Listening: 2 DLPT-Reading: 3 OPI-Speaking-MSA: 1+ Translation Test: 2 English Comp: 3+ Who would you hire…As a Foreign Language Monitor? Mirna: • native speaker of Arabic • DLPT-Listening: 2+ • DLPT-Reading: 2 • OPI-Speaking-MSA: 2+ • Translation Test: 2 • English Comp: 2

  34. Mike: heritage speaker of Jordanian Arabic OPI-Speaking-MSA: 3 OPI-Speaking-Jordanian: 3 OPI-Speaking-English: 3+ Imad: native speaker of Jordanian Arabic OPI-Speaking-MSA: 5 OPI-Speaking-Jordanian: 4 OPI-Speaking-English: 1 Who would you hire… As a MSA Oral Proficiency Interview Tester?

  35. Julie: native speaker of English DLPT-Listening: 2 DLPT-Reading: 3 OPI-Speaking-MSA: 1+ Translation Test: 2+ Hiam: native speaker of Moroccan Arabic DLPT-Listening: 3 DLPT-Reading: 3 OPI-Speaking-MSA: 4 English Composition: 3 Who would you hire… As a Document Translator?

  36. Testing Misconception Higher scores make better linguists. • Some job requirements do not necessitate high scores, or need high scores in only some skills. • Tests are usually not performance. • Other non-language skills may be required.

  37. Audio Sample • Native Speaker of English • College Graduate • Was in Peace Corps for 14 months in Uzbekistan

  38. Educating Management Stressing proficiency and performance Understanding job requirements and working with HR to establish criteria

  39. Quote 4 Debra, a newly hired Foreign Service Officer signed up for a test in Swedish, hoping to be assigned to Stockholm. The testing team reports that Debra understands Swedish quite well, but she speaks Norwegian.

  40. Testing Misconception I know what languages I can understand, speak, and write. • Mutually intelligible does not equal bilingual. • Languages are sometimes misidentified by examinees. • One skill ability does not equal all skills ability.

  41. Educating Stakeholders Ensure recruiters understand language requirements Define language for testing purposes Resources such as the NVTC site, Languages of the World web site

  42. Quote 5 Hanke (Polish OPI Tester Applicant, has an ILR Speaking 5 on record, 10 years old): “Why would I want to lower my score on record? Who and why decided that a Polish linguist working for the Bureau cannot score higher than a 4? And, if this applies only to Polish, why are the requirements lower for the Polish language?”

  43. Testing Misconception Test batteries and qualifying scores should be the same across languages for the same position. All tests from a particular battery may not be available in a language. Some tests may not have the same maximum possible score across languages.

  44. Educating Stakeholders Document testing standards Back up decisions with research, where available Ensure policies are documented and available to language professionals

  45. Quote 6 Silke (German linguist who has to retest because of expired scores): “I don’t see why I have to retest. I’m a 5. Even assuming that my command of the German language somewhat diminished over the years (which it has not), it could not have gone lower than a 4.”

  46. Testing Misconception Once a 4, always a 4. Test scores are considered valid for varying timeframes, depending on the agency and the level scored Regular retesting with new forms should be encouraged. Score may reflect the highest score given from that exam, not the ‘true score’ of the examinee.

  47. Educating Stakeholders Research project underway at the Center for the Advanced Study of Languages to investigate testing periodicity.

  48. Quote 7 Olga (Russian teacher): “I’ve had a lot of experience in test development. I’ve been giving tests in my classroom for 20 years, and I give oral exams at the end of the class, so I could be an OPI tester. I don’t think that I even need to attend the training.”

  49. Testing Misconception The best teachers make the best testers. Teachers sometimes have difficulty swapping hats from someone who fosters language learning to someone who is taking an impartial measurement. True whether or not the examinee is the teacher’s student.

  50. Educating Stakeholders:Tester Training Extensive testing training: Up to 1 year (Tester or Examiner training) Discuss FSI’s Testing Manual Rate taped tests Observe and discuss live tests Conduct (unofficial) progress tests Complete online program & mastery exercise Conduct certification tests

More Related