Surviving (and Thriving) in a “Perfect Storm” Environment

1. Surviving (and Thriving) in a “Perfect Storm” Environment Ray T. Clifford 22 February 2007

2. The Perfect Storm • Too little time. • Limited funding. • Not enough qualified teachers. • Rising expectations.

3. Some Recent Headlines • Court Creates Demand for Translators • Home Invasions Show Need for Translators • On the Table: A Cure for Medicare Language Gap • Built to be Bilingual: Researchers Pinpoint How the Brain Switches Between Languages

4. Some Recent Headlines • Untying U.S. Tongues • FBI Agents Still Lacking Arabic Skills • U.S. has a Strategic Need for Multilingual Citizens • Learning Second Language Changes the Brain: Bilingual People have More Grey Matter in Key Region

5. Some Recent Headlines • Iraq Panel: Put Forward Officials who Speak to Arabs in Arabic • First Ears, Then Hearts and Minds: Facing Shortage of Arabic Interpreters, Pentagon Seeks a Technological Solution

6. High Demand, Low Supply • More than 80 federal agencies employ individuals with skills in more than 100 languages. • Degrees granted in 2002: • 2, 396 French • 13 Arabic • Only one certified Arabic teacher in U.S. public schools.

7. And Yet… • Our nation suffers from delusions of linguistic adequacy!

8. Overview • What do Americans “know” about languages? • Why aren’t foreign languages required subjects in U.S. schools? • How can the U.S. respond to changing national language needs?

9. Limited Experience Leads to Naiveté • The general public believes: • Every English word has an exact match in the second language. • Second language learning is simply a re-lexification process. • High levels of language competence can be attained in two years of school classes…or perhaps in a few weeks with a computer program.

10. Limited Experience Leads to Naiveté • Even teachers often believe: • Students’ failure to attain meaningful levels of proficiency is primarily the result of poor teaching. • There is a single, best teaching method.

11. Needs + Naiveté = Nonsense • The Reality of FL Learning • “Time on Task,” not teaching method, is the best predictor of success in second language learning. • Of course, how that time is spent does make a difference. • Nothing clarifies the debate about teaching methods like a definition of goals.

12. The Major ACTFL Proficiency Levels

13. ILR Proficiency Level Summary [With language “text type” highlighted in red] LEVEL FUNCTION/TASKS CONTEXT/TOPICS ACCURACY All expected of an educated NS [Books] Accepted as an educated NS All subjects 5 Tailor language, counsel, motivate, persuade, negotiate [Chapters] Wide range of professional needs Extensive, precise, and appropriate 4 Errors never interfere with communication & rarely disturb Support opinions, hypothesize, explain, deal with unfamiliar topics [Multiple pages] Practical, abstract, special interests 3 Concrete, real-world, factual Intelligible even if not used to dealing with non-NS Narrate, describe, give directions [Multiple paragraphs] 2 Q & A, create with the language [Multiple sentences] Intelligible with effort or practice 1 Everydaysurvival 0 Memorized [Words and Phrases] Random Unintelligible

14. The ACTFL and ILR Scales Represent Significant, Substantive Levels of Accomplishment • Language is the most complex of human behaviors. • Language learning is an educational process which slowly, but inexorably reveals the limitations of any monolingual view of the world.

15. Language Teachers are Professional Educators. • The study of second language acquisition is a professional discipline. • Graduate degrees are offered in the field. • The discipline has professional journals, organizations, and conferences. • The field conducts an ever-expanding body of research. • Being a “native speaker” is not enough.

16. Bloom’s Taxonomy • Evaluation and persuasion through refined use of professional, literary, and rhetorical skills. • Synthesis of concepts to produce and comprehend abstract ideas and hypothetical situations. • Analysis and definition of factual relationships in paragraph length communications. • Application of skills to create and understand new communications. • Comprehension and use of words and phrases. • Memorization of facts.

17. Language and Cognition • Levels 4 & 5, evaluation and persuasion through refined use of professional rhetorical skills. • Level 3, synthesis of ideas to produce and comprehend abstract comments and hypothetical situations expressed in essays, chapters, etc. • Level 2, analysis and definition of relationships expressed in multiple interrelated paragraphs. • Level 1, application of skills to create and understand sentence length communications. • Level 0+, comprehension and use of words and phrases. • Level 0, memorization of facts.

18. Language and Perceived Intelligence It is good [for students] to become acquainted with languages, for they may have to go abroad, and should be able to talk to people, and not look like fools. I care not how much intelligence you have, if you cannot exhibit it, you look like an ignoramus. John Taylor, 1852

19. A Continuing Problem • 1975: The International Association for the Evaluation of Educational Achievement published the results of a study titled, The Teaching of French as a Foreign Language in Eight Countries. • In the U.S., the researchers could not find enough 12th grade students with four years of language study to complete the study as designed. • The U.S. students ranked last in competence. • The study found that “…the primary factor in the attainment of proficiency in …any foreign language… is the amount of instructional time provided.”

20. A Continuing Problem • 1979: The President’s commission on Foreign Language and International Studies reported, “Americans’ incompetence in Foreign Languages is nothing short of scandalous, and it is becoming worse.”

21. A Continuing Problem • 1983: The Commission on Excellence in Education • Heard testimony that in the U.S., FL instruction had yet to attain mediocrity. • Published the report A Nation at Risk, which recommended longer course sequences for foreign language programs.

22. A Continuing Problem • 2004: The Modern Language Association survey of foreign language enrollments in higher education reported: • Total foreign language enrollments have increased slightly. • However, as a percentage of total enrollments, foreign language enrollments are decreasing. • In 1960 they were 16.1% of total enrollments. • In 2002 they were only 8.3% of total enrollments.

23. And Now… • The original 2007 federal budget figures from the Office of Management and Budget showed the following planned reductions: Defense – 8.7% Education – 24.4% • The forecast: A competitive fiscal environment.

24. How can language programsbecome competitive? • Assess the current status of our programs. • Tell the real story. • Support our story with test results.

25. What do tests have to do with Teaching and Learning? • It helps to begin with the end in mind. • You can’t teach skills you can’t measure. (Even if you could, how could you prove you’ve done it?) • Education is undergoing a paradigm shift from instructional objectives to learner outcomes.

26. Changes in Accreditation • There is an unprecedented move to replace process reviews with outcome reviews. • Accreditation and Student Learning Outcomes: A proposed Point of Departure • Knowledge outcomes • Skills outcomes • Affective outcomes • Abilities (the integration of KSA outcomes) Peter T. Ewell , Council for Higher Education Accreditation, September 2001

27. What will be the effect of these accreditation requirements? • More testing will take place. • Some beneficial • Some detrimental • These tests will influence learning. • The “Will that be on the test?” phenomenon. • The temptation to “teach the test” instead of teaching the skills necessary to pass the test. • Every testing decision creates a washback effect.

28. “Washback” Effects • Testing has a negative impact when: • Educational goals are reduced to those that are most easily measured. • Testing procedures do not reflect course goals, for instance… • Giving multiple choice tests in speaking classes. • Using grammar tests as a measure of general proficiency. • The test results don’t provide useful information.

29. The National Debate onSchool Testing • One formula for evaluating school performance School score = (((((X23*100)*Y23) + ((X24*100)*Y24) + ((X25*100)*Y25) + ((X26*100)*Y26) + ((X27*100)*Y27) + ((X28*100)*Y28) + ((X29*100)*Y29) + ((X30*100)*Y30) / ((X23 + X24 + X25 + X26 + X27 + X28 +X29 + X30)*100)) + ((((Z23*100)*Y23) + ((Z24*100)*Y24) + ((Z25*100)*Y25) + ((Z26*100)*Y26) + ((Z27*100)*Y27) + ((Z28*100)*Y28) + ((Z29*100)*Y29) + ((Z30*100)*Y30)) /((Z23 + Z24 + Z25 + Z26 + Z27 + Z28 + Z29 + Z30)) / ((Z23 + Z 24 + Z25 + Z26 + Z27 + Z28 + Z29 + Z30)*100))) / 2 The Wall Street Journal, May 21, 2001, page A24

30. The National Debate onSchool Testing • What would be the washback effect of this evaluation formula? • Perhaps Confusion? • Perhaps Frustration? • Perhaps “teaching the test” in a desperate attempt to improve results?

31. Washback Effects of Tests • Testing has a positive impact when: • Tests reinforce course objectives. (For students, nothing defines course objectives like a test.) • Tests can act as change agents for improving teaching and learning.

32. Language Testing and Motivation • Appropriate tests can motivate learners to improve their skills. • Appropriate tests can motivate teachers to refine their teaching to match their students’ needs.

33. “Some considered projections” from THE NATIONAL INTEREST AND FOREIGN LANGUAGES Eight- and ten-year sequences of foreign language study will become common in the public schools. The better colleges and universities will require demonstrated proficiency (not high school “units”) in a foreign language for entrance, and demonstrated proficiency in a second foreign language (often non-Western) for graduation. WILLIAM RILEY PARKER for THE U.S. NATIONAL COMMISSION FOR UNESCO, DEPARTMENT OF STATE, 1961.

34. The National Debate onSchool Testing • What might have been the washback effect if students’ language abilities had been used to evaluate program effectiveness in 1961? • Perhaps clearer statements of objectives? • Perhaps more realistic learner expectations? • Perhaps more accurate course descriptions? • Perhaps qualitative feedback that would have justified longer course sequences?

35. After four decades, why are Parker’s projections not yet a reality? • Lasting improvements are only sustainable if there is a recognition of need for them. • Norm-referenced grading gave little evidence of real-world ability gaps. • Accreditation was focused on process rather than on outcomes, and provided no incentive to change. • Administrators didn’t read the research results showing that no improvements in teaching methods or curricula can compensate for the current lack of “time-on-task” in our educational system.

36. International Research on Learning French • John B. Carroll, The Teaching of French in Eight Countries (International Studies in Evaluation V) John Wiley & Sons, New York, 1975. • “… the primary factor in the attainment of proficiency in French (and presumably, any foreign language) is the amount of instructional time provided.” [Page 276]

37. International Research on Learning English • G. Bonnet, et al. The Assessment of Pupils’ skills in English in Eight European Countries: 2002. European Network of Policy makers for the Evaluation of Education Systems, 2004. • Students from those nations where there is more contact with (and more time spent using) English have higher levels of competence in English. • In a language-rich environment, time spent using the language is more important than the teaching methods used in the classroom.

38. Assessing FL Learner Outcomes • FL profession has made much progress in the area of assessment and standards. • Our ability to test students’ language proficiency puts us ahead of many other disciplines. • But proficiency tests are not the only way to measure learner outcomes. • Which test is “best” will depend on: • The purpose of the test. • Each program’s instructional goals.

39. Matching Tests with Your Instructional Purposes • Some typical FL testing purposes. • Assigning course grades in a beginning class. • Placement of students into a sequence of courses. • Certifying a general level of ability for teachers. • Screening of job applicants. • Some general instructional purposes. • Achievement – knowledge of contributing language elements found in a specific curriculum. • Performance – ability to communicate in specific settings. • Proficiency – demonstration of unrehearsed ability.

40. If Tests Are to bePositive Motivators • We have to select the right type of test for each testing purpose.

41. The 3 Major Types of Tests • Achievement = Memorized responses using the content of a specific textbook or curriculum. • Performance =Rehearsed ability to communicate in specific, familiar settings. • Proficiency = Unrehearsed general ability to accomplish communication tasks across a wide range of topics and settings.

42. Some CommonTesting Purposes • Assigning grades in a class. • Placing students into a sequence of courses. • Selecting an applicant for a job with limited, static language requirements. • Screening employees for future jobs with broad, general language requirements.

43. Aligning testing programs with our testing purposes requires knowing what makes proficiency tests different fromperformance testsand achievement tests.

44. The ILR/ACTFL Proficiency Scales are Different, Because They: • Represent a geometric progression. • Use threshold rating criteria. • Do not use compensatory rating procedures. • Are a multidimensional measure of ability. • Apply Guttmann scaling criteria. • Are based on a wide range of “real world” communication needs. • Assess unrehearsed language abilities.

45. 1. The ILR/ACTFL scales represent a geometric progression. • Advancing from one level to the next becomes increasingly difficult as one moves up the scale. • For instance, moving from 0 to 1 requires less time and effort than moving from 2 to 3.

46. What are the Major Levels?How are They Defined?

47. 2. The ILR/ACTFL scales use “threshold” rating criteria. • Candidates must meet all of the stated criteria for a given level. • If they meet most, but not all of the criteria, some award a “plus” designation to the next lower level. • For example, someone who meets the criteria for Level 2 almost all of the time, could be rated 1+.

48. 3. The ILR/ACTFL Ratings Are Not Based On Total, Average, or Other “Compensatory” Scores. • Ratings may be limited by one or more of the task, context, and accuracy requirements. • For example, a large vocabulary does not compensate for inaccurate usage.

49. Implications of Points 2 and 3: Assessment Criteria-Speaking

50. The ILR/ACTFL scales representa complex set of abilities. • The ILR/ACTFL scale is a multi-dimensional scale. • The scale is not uni-dimensional like a rope that has been cut into segments. • It is multi-dimensional, because each level of the scale represents a different constellation of supporting and enabling skills. • The ILR/ACTFL scale (and other complex scales) cannot be represented by a one-dimensional statistical model.