Language Learning in Crisis: Evaluating the Importance and Impact of Language Education

1. Learning to value evaluation In language teaching John M. Norris University of Hawai´i at Mānoa

2. Dealing with change In language education

3. Language learning in crisis “We cannot afford to seek out foreign language skills after a terrorist attack occurs. The failures of communication and understanding have already done their damage. We must provide an ongoing commitment to language education and encourage knowledge of foreign languages and cultures.” Daniel Akaka, U.S. Senator from Hawaii

4. Language learning in crisis The present state of language education and proficiency in Australia is seriously inadequate for our current and emerging needs, and far behind comparable levels in our peers and competitors. The size of this gap and the work and time required to close it has led some to describe the situation as a crisis.

5. Language learning in crisis At present, though, the English-speaking abilities of a large percentage of the population are inadequate, and this imposes restrictions on exchanges with foreigners and creates occasions when the ideas and opinions of Japanese people are not appropriately evaluated.

6. Language learning in crisis “We are lousy at foreign languages and shouldn’t be”

7. Language learning in crisis “Viva la English” Tribune Media: 10-26-2007 “I think that the fewer languages we have, the better off civilization will be.” Andy Rooney

8. Language learning in crisis More than half of the world's 7,000 languages are expected to die out by the end of the century BIOCULTURAL DIVERSITY? Most of what we know about species and ecosystems is not written down anywhere, it's only in people's heads David Harrison (2007) – Living Tongues Institute

9. Language learning in crisis “English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms.” Margaret Spellings (2005) – U.S. Secretary of Education Number of children studying Chinese in U.S. schools: 24,000 Number of children studying English in Chinese schools: 200,000,000

10. Language learning in crisis Number of ‘newcomer’ children who need to study Japanese in schools in 1991 = 4,463 Number of ‘newcomer’ children who need to study Japanese in schools in 2002 = 19,764 (4X) Foreign students in Japanese universities in 1985 = 19,741 Foreign students in Japanese universities in 2006 = 100,804 (5X) Brazilians residing in Japan in 1986 = 2,135 Brazilians residing in Japan in 2005 = 302,080 (141X)

11. Language learning in crisis Are we seeking survival skills, intercultural competence or quality of mind? Who are we teaching and what do they need to learn? Why bother with language learning at all; what does it offer? Language Teaching & Learning What’s the relationship between L1, L2, social, & academic development? What’s the most effective way to teach language(s)? How do we develop teachers to meet current demands?

12. Language learning in crisis In the U.S. … High-quality teachers NCATE – TESOL/ACTFL Teacher Development Program Standards MLA White Paper Foreign Languages and Higher Education: New Structures for a Changed World

13. Language learning in crisis MEXT “The Ministry of Education, Culture, Sports, Science and Technology (MEXT) shall settle on an action plan to improve English education within the fiscal year 2002 with the aim of fostering "Japanese with English abilities". As of the fiscal year 2003, the employment of excellent assistant language teachers of foreign nationality as regular teachers is to be promoted.” In Japan …

14. Why worry about evaluation? “What we assess is what we value” --Lauren Resnick “How we choose to assess will determine what gets valued” --Norris

15. Why worry about evaluation? “As part of its re-accreditation, the university has required all undergraduate programs to create and implement outcomes-oriented assessment plans.” (survey respondent) • Top 3 sources of pressure for evaluation in U.S. college FL programs: • University administration • The dean • Accreditation process “Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured.” (survey respondent)

16. Why worry about evaluation? All universities, junior colleges and colleges of technology conduct self-checks and self-evaluations concerning education and research All universities, junior colleges and colleges of technology are regularly evaluated and accredited by organizations certified by the Minister of Education, Culture, Sports, Science and Technology

17. Why worry about evaluation? “This interest was fuelled primarily by a reduction in public spending to cope with recession, internal and external pressures for performance improvement, and constant demand to review, renovate, and reform the way the public sector does business in order to achieve social accountability.” Nagao, Kuji-Shikatani, & Love (2005) Founded in 2000

18. Why worry about evaluation? ニュース 公立中、英語で授業４％だけ　英語の授業の大半を英語で行っている公立中学校は約４％にとどまり、「授業の大半を英語で行う」とする文部科学省の目標に遠く及ばないことが、同省の調査で分かった。 　また、ビジネス界で英語力の指標とされる「ＴＯＥＩＣ」で７３０点以上の英語教員は、中学で１割、高校で２割にも満たず、英語教育のお寒い実態が浮き彫りになった。 　調査は今年２月、全国の公立中学校約１万２００校と、公立高校約３８００校を対象に実施した。授業での英語の活用実態を調べたのは初めて。 “Ministry survey finds schools' English teaching falling short”

19. Why worry about evaluation? NO College Leaving Exam CHILD COLLEGE STUDENT LEFT BEHIND “Does the Spellings Commission think about language education at all?” Michael Holquist (ADFL 2007 Summer Seminar West)

20. Why worry about evaluation? Quality Rating Categories ☺How easy? ☺How fair? ☺How good? HOW HOT??? RateMy Professors

21. Why worry about evaluation? • Drake University language programs… • Low enrollments, student dissatisfaction, poor external reviews • Faculty refuse instructional development support • Faculty refuse to create strategic plan for improvement • No acknowledgement of need to change • No engagement with evaluation findings NO MORE LANGUAGE PROGRAMS, NO MORE FACULTY!

22. Why worry about evaluation? • Despite such problems, U.S. college FL survey respondents desired increaseduse of evaluation for: • Understanding & improving program outcomes • Understanding & improving program functions • Improving FL education on the whole • Understanding & Improving the worth of the program • Raising awareness about FL programs

23. Why worry about evaluation? And some expressed a professional ethic to engage in evaluation: “We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and why what we do is valuable.” (Anonymous survey respondent)

24. Evaluation as an agent of change Within this milieu, what is the role to be played by evaluation? Value of Language Education • Key • Challenge: • Define • Delimit • Detract • Dismiss • Key • Opportunity: • Enable • Enhance • Engender • Empower

25. Traditions, trends, and the status quo In language program evaluation

26. Traditions of language program evaluation Epistemology “The experimenting society” Methods Measure L2 outcomes, compare programs Program focus Products of program completion Summative: judge program worth Knowledge: identify best methods Evaluation purpose Evaluation use Perpetuate, fund, or shut down Evaluation practice Short-term, external, Jet-In Jet-Out Expert JIJOE

27. Traditions of language program evaluation …but why did the program fail (or succeed)? …but that test really doesn’t measure what we teach! …but how can you conduct a rigorous experiment on actual classrooms, teachers, and learners? …but what can you tell me about what language teaching method really works the best for my learners in my school? …but you cannot compare our program with their program—they are so different! …but what if we don’t have the option of closing down the program—how do we make improvements and what do we need to improve? …but you don’t really understand our program—you were just here for a week!

28. Traditions of language program evaluation • Proliferation of language program evaluators (and texts)… • Alderson & Beretta (1992) Evaluating second language education • Rea-Dickins & Germaine (1992) Evaluation • Brown (1994) The elements of language curriculum • Weir & Roberts (1994) Evaluation in ELT • Lynch (1996) Language program evaluation • Changing emphases in evaluation practice… • Focus on formative (improvement) purposes • Attention to process of language teaching/learning • Use of multiple methods (qualitative + quantitative +…) • Pragmatic problem-solving approach • Integration into curriculum and context

29. Traditions of language program evaluation Rea-Dickins (1994): “If evaluation in English Language Teaching is to be effective, we will see a stronger integration of evaluation within practice, as part of an individual’s professionalism, and an increase in collaborative activity where teachers (and other relevant participants) are actively engaged in the monitoring process.” Importance of language teacher participation

30. Countervailing trends in evaluation Swender (2002), on FL teacher professional development: “After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21st century”. Technocratic measurement problem

31. Countervailing trends in evaluation WebCAPE Foreign Language Placement Exam Standardized assessment problem: One size fits most purposes & settings

32. Countervailing trends in evaluation IELTS TSE – SPEAK – SLEP – TOEIC Standardized assessment problem: One size fits most purposes & settings

33. Countervailing trends in evaluation The ‘measurement mindset’ in teacher practice: “Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’.” Bernie Mohan example, TBLT 2007 Conference

34. Countervailing trends in evaluation What is the appropriate proficiency level to adopt as a student learning outcome for the 2-year language requirement? INTERMEDIATE - LOW ? ? ?

35. Countervailing trends in evaluation Accountability movement: using standardized tests to hold teachers and students to performance expectations “We’re going to stand strong on accountability” --Margaret Spellings, U.S. Secretary of Education “If you want to hold schools accountable and make sure they are learning, you have to test” --Robert Black, spokesperson for the Gov. of Texas Test Misuse Problem

36. Countervailing trends in evaluation "There has been an explosion of mandates for more and more standardized tests with very little evidence to support their use" --Walter Haney of Boston College's Center for the Study of Testing, Evaluation and Educational Policy.

37. Countervailing trends in evaluation “Bilingual education ends up being monolingual education in the language of the high-stakes test, until the test is over." --Deborah Palmer, University of Texas

38. Countervailing trends in evaluation “…parents are dealing with children vomiting on the morning of the tests …" --Gloria Pipkin, Florida Coalition for Assessment Reform

39. Countervailing trends in evaluation Student Learning Outcomes Assessment Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking  Provides a clear statement of educational program value; answers the question “How do you know?” with evidence of educational effectiveness

40. Countervailing trends in evaluation Large public institution Accreditation pressures to assess learning We have to assess our “liberal studies” core, ASAP! How about an electronic portfolio? Huge expenditure, $$$, time, effort Thousands of student portfolios created BUT…

41. Countervailing trends in evaluation Electronic Portfolio NEVER GOT USED …faculty didn’t understand it …students thought: • *!%#$!@* 2. waste of time …administrators wanted to do something with it but weren’t sure exactly what

42. Countervailing trends in evaluation Barrington (2003), on outcomes assessment in the liberal arts: “To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”. Perception problem

43. Traditions, trends, and the status quo Language program evaluation Back where we started Status quo • Focus on doing… • Based on standardized testing • Reactive v. proactive praxis • Driven by external impetuses • Technocratic measurement emphasis • Little scholarly investment • Not useful—not used! • Potential negative washback, reductionism, waste • Not relevant to curriculum & instruction, program values • Not ours—done to us, not for or with us • Not perceived as worth the effort by faculty • Minimal professional development

44. Re-envisioning evaluation in language education

45. Resolving terminological confusion Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity. Norris (2006) MLJ Perspectives

46. Resolving terminological confusion Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it. Norris (2006) MLJ Perspectives

47. Resolving terminological confusion Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives

48. The nature of useful evaluations Pragmatic: Context relevant use & focus Action oriented: Actions are taken based on evaluation findings Participatory: Active involvement of key stakeholders Democratic: Negotiated decision making Manageable & feasible: Adapted to available time and resources Educational & Transformative: Users learn by participating Responsive: Evaluation responds to primary intended users’ purposes Clear & understandable: Transparent processes and outcomes

49. Evaluative assessment Light (2001) on outcomes assessment: “…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively.” Richard Light (2001, p. 224)

50. Acknowledging multiple legitimate purposes Resisting change Empowering teachers Justifying $ requests Demonstrating value Holding accountable Revising curriculum Raising awareness PURPOSES Accrediting schools Developing programs Diagnosing need Improving teaching Articulating courses Revising materials