Everything 1 you ever wanted to know about Japanese University English Entrance Examinations

1. Everything1 you ever wanted to know about Japanese University English Entrance Examinations Melodie Cook, PhD University of Niigata Prefecture 1Okay, maybe not everything

2. A warning to me from a friend I'm bothered that you are attempting to write about [this topic] intelligently. That's Ok if you are Japanese and want change. Coming from a gaijin with a genuine career trajectory such as yourself, it might not look good on your resume. Experience on an entrance test committee is a definite plus on the resume. Having an article discussing the merits of these tests from an educational perspective might mark you out as a potential troublemaker like JD Brown. I'm sure you could land a 10m yen a year job, Mel, but I think you should be as pragmatic as possible. A really smart teacher in  Japan would think ‘this test is bullshit, but remember it's for the money, for the show, that's all and the smartest kids will float to the top somehow’.

3. Terms Expatriate/Local

4. Expatriate/Local

5. Background • Mel’s story • Summary of three main themes arising from a Japan-wide study of the beliefs of expatriate tertiary-level ELT faculty members about the creation of the English portion of entrance examinations. • Online survey (snowball sampling)n=102 • Open-ended follow-up interviews n=24 demographic information role in examination construction information about test design general beliefs and opinions about examinations

6. Outline • Expatriate Faculty Beliefs • Assumptions • Change

7. Question 1: Discussion Why do entrance examinations exist?

8. From the Literature Pedagogical Purposes Other Purposes To serve social, economic and political purposes (LoCastro, 1990) To demonstrate a university’s status and selectivity (Blumenthal, 1992; LoCastro, 1990; Shimahara, 1978) To indirectly measure students’ diligence (Frost, 1991; LoCastro, 1990) and intelligence (Kariya & Dore, 2006; LoCastro, 1990) • To test proficiency • To evaluate achievement • To diagnose strengths and weaknesses • To place candidates in appropriate classes (Hughes, 2003)

9. Shimahara (1978, p. 263) “an arbitrary device for social placement, rather than a pedagogical instrument"

10. Minority Report: Pedagogical Purposes Served • Paul, Sarah, and Cheryl’s institutions knew where candidates were coming from and based test items on topics covered in those textbooks/on the vocabulary at least (achievement)

11. Majority Report: Pedagogical Purposes Not Served • …not about English ability exactly… so the questions are not really designed to see how proficient a person is at English. (Debbie) • the test is constructed by teachers who … obviously weren’t very familiar with what had been going on for the previous five or ten years in the high school curriculum. (Jack) • Not as placement. (Joe)

12. Test Purposes • To test candidates’ test-taking skill A lot of them (students) will waste a lot of time just studying test-taking skills… you really don’t want to study something, there’s a way to get around through guessing; you can spend a lot of time just figuring out how to do well on the test. You know, think of it as a dance. You don’t really have to have a good sense of rhythm if you know which technical movements to do. (Phil)

13. Test Purposes • To indicate intellectual ability and/or academic potential I would say it’s for… especially a national university… one of the purposes is to indicate if the students are academically capable. Secondly, it should in some way show that the student is cognitively prepared to deal with… being an academic, being a student at a university, so we want to see that cognitive skills and basic intelligence come through. (Joe)

14. Test Purposes • To generate revenue The test fee is an important contribution (Diogenes) A payoff for the school (Phil) The kids pay about 10,000 yen or something… they pay a lot to do the test, and they’re not being tested. (Austin)

15. Test Purposes • To stratify students in society The tests have only one purpose which is to get students who are at a certain level overall in maths, English, and so on… so it’s just to distinguish between them so that the A-ranking university gets the top 5% and then the next one down… There’s no other function. (Mike)

16. Test Purposes • To show the face of the university to the public Our primary consideration was not, “Is this the right level of difficulty for students coming to take the test?” because then we would have to set a test that was so ridiculously easy that we would do nothing for the reputation of the school. Our consideration was “How does this look in the akahon?” and then we’ll find a way to admit people. (Paul)

17. Conclusions • Expatriate faculty understand there are many purposes for entrance examinations • Still, they need to acknowledge non-pedagogical reasons for testing in Japan

18. Question 2 Who is making these tests and what are they thinking?

19. Assumption 1: That an entrance examination is, or should be, a language test

20. Assumption 2: That students want to be in the institution in which they are studying Diogenes’ story (as read by Mel)

21. Assumption 3: That students want to be in the faculty/department in which they are currently studying Diogenes’ story about a junior high school teacher (as read by you) She was bright, but not very bright, and she had always wanted to be a junior high school teacher. And it turns out that she passed exams to “A” Uni and “B” Uni. So “A” Uni is the national university and “B” Uni is the well-regarded private university… And so when she took the exam for “B” Uni she chose the English department and for “A” Uni, she chose the Persian department. And she passed both exams, so which one do you think she took?

22. Origins of assumptions • 1:Own educational experiences I mean, where in Canada we don’t have any SAT we don’t have any entrance exam; all we have are the marks from our high school and that’s it. And so there is none of this studying for an exam and a mark that you get on the exam determines the rest of your life sort of thing. (Bob)

23. Origins of assumptions * 2: Influence of graduate school So for me, if we’re going to bother making this big document and make sure that it’s perfect in every way we can possibly make it, well then why not try to use it again? Why not try to test for the things we want our students to be able to do in the first place? (Bob)

24. Origins of assumptions * 3: Experience with large-scale testing I would like a speaking component, but I realize that’s not feasible. I come from a background of IELTS preparation where IELTS examinations do have tests in the four skills… whenever I look at a university entrance examination package, I always compare it to the IELTS package which is very well tried and tested over the years. (Peter)

25. Conclusions • Expatriate faculty on examination creation committees may be influenced by their own educational backgrounds and other testing contexts they have worked in • More communication between expatriate and local faculty might be helpful to avoid misunderstandings

26. Question 3: Discussion How can an entrance examination be ch-ch-ch-changed?

27. How: Hindrances • Hierarchy Seniority, and personal power trump expertise. Many faculty members are 55-65 years old and set in their ways. They prefer the way things have always been done. That being said, some (older) members are open to innovation, and if they are persuasive enough (and have no real enemies on the committee), occasionally things do change. (Mauve)

28. How: Hindrances 2. Face I think there were social pressures not to stand out… I think the same thing is affecting the introduction of statistics, looking at examination statistics, because some people are very much into this and they’ve done a lot of preparation and they’ve done statistics at school and they’ve done research papers involved with statistics, and other people don’t know the first thing about it. And so there’s a great difference in level of sophistication of what this is all about. And as it happens, the people who are higher up in the administration are the ones who don’t know much about this kind of thing. It’s a face-saving device, I think. (Diogenes)

29. How: Hindrances 3. Face and Fear … it (the test) has to go through other people who may be non-academics or non-practicing teachers who have their own situation and their own position to protect, and one way they do this is relying on what they consider to be a normal practice… and these people have a fair amount of clout… and nobody wants to lose face… it’s a big hassle to try to persuade these other people to change their views and if they’re changing, they’re moving into something potentially dangerous because it could go wrong… (Jack)

30. How: Hindrances 4. Continuity issues When I tried to introduce… critical thinking questions, they didn’t really have a problem with it at all… and then they translated them into Japanese and they thought, ‘Oh, that’s quite good’. … then when I was off the committee, those kinds weren’t put on anymore. (Mike)

31. How: Hindrances 5. No perceived need Native speakers… because this is kind of their job to be on the committee, they have a sense that, ‘I should do this well, as I’ve been trained.’ Whereas the Japanese teachers don’t see that as being necessary or it isn’t necessary. It’s objectively not necessary. It’s a fact, not an opinion. (Mike)

32. Understanding Japanese Management Practices • Hierarchy “Japanese teams and organizations are organized vertically. This means that older members have more power, earn more, and are considered more knowledgeable, even if this is not really the case” (Haghirian, 2010, p. 14).

33. Understanding Japanese Management Practices • Risk avoidance “High involvement in the group and a shared responsibility for each member further leads to a tendency to avoid risk and to ensure that all team members are safe” (Haghirian, 2010, p. 80).

34. How: Helps • Consensus Everyone in our department adds something, adds some small part or [has] gradually been changing how the test has been done over several years. (Phil)

35. Understanding Japanese Management Practices “(Group consensus)… is the reason why processes, once established, will hardly be individually improved or changed, even if a process proves inefficient or even damaging to the firm. Without group consensus, nothing can be changed…” (Haghirian, 2010, p. 100).

36. How: Helps 2. Offers Carlos’ offer to change question types was accepted by colleagues after he made a reasoned argument for it Carlos was asked, “What do you want to do?”

37. How: Helps 3. Threats to Face/Public Opinion We used to use… they have these booklets of things from other colleges that you’re allowed to use, and we’re basically taking them and entirely editing them, and then put them in the test… but then… somewhere it was published, the number of… the universities that use other people’s materials, and ours came out… in the top 10, or on top, and the president freaked out about this and said, “You can’t do this anymore.” … so now we’re making all of our own materials. (Cheryl)

38. Before Asking for Change:Questions to ask • Is your entrance examination a language test? • Why do you need to change your test? • Who can help you change your test? • How can you convince others that change is needed? • How long are you willing to wait for change?

39. If it Ain’t Broke… I don’t disagree with J.D. Brown or Tim Murphey in the sense that if there’s a better test, that’s surely a good thing, but there isn’t an advantage to a better test. What would be the advantage? The only thing you could potentially gain would be if your current system is not getting the kind of students that you want to have and you think you can make a test that will get them. But you won’t be able to do that, because people won’t try to take your exam if they think your university… doesn’t have the status or whatever it is, so making a better test has no meaning that I can see outside of just feeling good that we’ve got a test that measures something. (Mike)

40. You Don’t Know Jack Melodie: So, you’re on the committee, what, as a kind of proof reader or a checker? Jack: Yes, ostensibly that’s their view, but my view is I’m on the committee, and I’m gonna say what I like. And I can get away with it.

41. Claimed vs. Assigned Identity Johnston (2003) “A fundamental tension in ELT professionalism: that although many teachers strive to be professionals, their aspirations are not reflected in the way their work is perceived by those around them. In other words, there is a disjuncture between what can be called claimed identity and assigned identity (p. 108).

42. You Don’t Know, Jack Jack was told, “This is what we always do” Jack replied: “I’m on the committee now, and I’m telling you that this makes no sense.”

43. Understanding Japanese Management Practices “Being successful in a Japanese team means being more passive and listening to older peers. In the context of a Japanese firm, being active in finding a role and taking on an independent task is considered very inappropriate in general and unheard of for beginners. This behaviour often leads to intercultural conflict” (Haghirian, 2010, p. 101).

44. Who Can Help You Make Your Case?

45. Quality Circles • “In her book The Accidental Office Lady, Laura Kriska describes her time as an American working for a well-known Japanese carmaker. At the time, women working in an office had to wear uniforms, while men did not. After complaining about this, Kriska was told to form a quality circle and present suggestions for improvement. A group formed and collected data on costs, motivation, and other factors, and after a few months, the results, which suggested that the uniform rule should be abandoned, were presented to the top management. The presentation was convincing, and the company changed its policy” (Haghirian, 2010, pp. 9-10).

46. Will Others Share the Burden? It’s not like they asked me to make the test better; it’s just that they said, ‘Okay, you’re on the committee. What would you like to do?’ And I put up my hand said, ‘I’ll do the reading section if you don’t mind’… I came in the next week and the writing section wasn’t really done yet, so I said, ‘Well, I’ll start on this as well, if you don’t mind’… on the other hand, if I hadn’t done that, I’m sure it would have been same old same old. (Carlos)

47. How Can You Persuade? • Like Carlos, offer clear pedagogical reasons for proposed changes • Appeal to face There’s something to be said, just sheer pride and professionalism to create a really good test, and that’s the best way to pitch it. If you just say, ‘Look, we should do it because universities in Japan have a bad reputation and are not in the top 100 in the world, so here’s something we can do to make it more professional and better.’ (Mike)

48. How Long Will You Wait? Incremental Change We can persuade in some way gently, can’t we? I don’t think it happens quickly. (Peter)

49. Summary of Recommendations • Find out what the true purpose of the test is • Know what your role is (assigned, NOT claimed) • Make recommendations in Japanese, where possible • Be positive and tactful about criticism – show explicitly what YOU would like to do to contribute to positive change • Work with others who are sympathetic/knowledgeable and will support you (quality circles) • Do “nemawashi” to “prevent blind-siding or surprises, preserve relationships, and save face” (Haghirian, 2010, p. 83) • Show how your recommendations will have a positive impact on the reputation of your institution • If possible, show examples of other universities’ examinations which are doing what you want to do • Be patient – change is incremental

50. Today’s Takeaway • Entrance examinations may serve more purposes than you think • There may be differences between your and others’ opinions about what is important • Examinations may be constructed in ways you don’t expect, for reasons that may not be made explicit • Change is possible, if you make the right case in an appropriate way • In the final analysis, the test might be “for the money” and “for the show”, but it need not be “bullshit”.