1 / 23

Implementing Exit Exams in a University-level Intensive ESL Program

Implementing Exit Exams in a University-level Intensive ESL Program. Breana Bayraktar Marietta Bradinova Masha Vassilieva Virginia International University. Plan. Situate this project in the context of an intensive, university-level ESL program

maia
Download Presentation

Implementing Exit Exams in a University-level Intensive ESL Program

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Implementing Exit Exams in a University-level Intensive ESL Program BreanaBayraktar Marietta Bradinova MashaVassilieva Virginia International University

  2. Plan Presentation at the 2011 TESOL National Convention Situate this project in the context of an intensive, university-level ESL program Describe the creation and implementation of an alternative essay rating guide Share tentative conclusions and discuss implications for other university ESL programs.

  3. Background on the program Presentation at the 2011 TESOL National Convention • University-level ESL program serving students from all over the world • 4 levels of instruction: • Elementary • Intermediate • Advanced • High-Advanced (also known as College Prep) • Completion of the college prep level = advancement to academic programs, “General Education” courses, at the undergraduate and graduate level.

  4. Overview of Placement/Advancement Presentation at the 2011 TESOL National Convention • Placement of all new students • Computer-based test (ACCUPLACER) • Reading Skills evaluates comprehension of short passages • Language Use measures grammar and usage • Sentence Meaning assesses the understanding of word meanings in one- or two-sentence contexts • Essay (45 mins) • Interview • Advancement of all continuing students • Pass 80% of their classes at the previous level with an average of 75% or higher. • Or, re-take the placement test to see if they are eligible to advance to the next level.

  5. Concerns regarding placement/advancement procedures Presentation at the 2011 TESOL National Convention • Students can “pass” and graduate from the program by passing a majority of their classes at each level • program advancement protocol allows student to advance if they pass 80% their courses, or if they can perform well enough on the exit test to prove that they have “tested into” the next level. • Anecdotal evidence from instructors • wide range of proficiency levels in each class • students demonstrate an inadequate mastery of grammar and sentence structure, particularly at the higher levels

  6. Our Puzzlement Presentation at the 2011 TESOL National Convention • Concern that the essay scoring rubric didn’t give enough weight to grammatical accuracy and the descriptors were too vague. • Feeling that movement at the CP level—both into the level and advancing out of the level—was possible for students lacking important grammar skills. • Did the course grades (for continuing students) or the placement essay rubric not give sufficient weight to grammatical accuracy? • Our goal: to create and test an essay-rating mechanism specifically for students entering and exiting the college prep level that would: • Give grammar significant consideration • Produce consistent ratings among raters

  7. Our thinking behind revising the rubric Presentation at the 2011 TESOL National Convention Concern that the current rubric (side 1 of the handout) wasn’t precise enough for the upper levels, particularly the placement in college prep vs. academic courses Desire for rubric to reflect what courses at the ESL level actually focus on, so that students are placed appropriately Need for greater emphasis on mastery of language functions (as opposed to rhetorical functions)

  8. New Essay Rating Diagram Presentation at the 2011 TESOL National Convention • Academic (non-ESL) level OS1 G1 • College Prep level OS2 • College Prep level OS3 G2 • Advanced level OS4 • Advanced level OS3 G3 OS4 • Intermediate level

  9. Grammar Descriptors Presentation at the 2011 TESOL National Convention (G1) Grammar errors are either nonexistent or barely noticeable / minor. They do not distract from the overall essay; nor do they impede comprehension or slow the reader down. Students moving on to the Academic level need to be able to write clearly and comprehensibly, and be prepared to succeed in a “freshman composition” class, where the strategies of academic writing will be taught, but where there is not a focus on grammar.

  10. The first descriptor (G1) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE: Presentation at the 2011 TESOL National Convention G1 • (OS1) There is a clear main idea with a controlling idea and a conclusion or a concluding remark. The body is organized logically (not necessarily into paragraphs) and there is evidence of sequencing & connectors. For each major idea there is support (examples, details, explanations). • Placement in Academic (non-ESL) level • (OS2) There is a main idea but no clear controlling idea. There is a lack of logical organization. There is insufficient or irrelevant support in the body. There are evident limitations of vocabulary in the form of non-specific vocabulary ("things" or "stuff") and/or not-appropriate vocabulary. Either the lack of organization or vocabulary limitations are distracting to the point of slowing the reader down. • Placement in College Prep level

  11. Grammar Descriptors Presentation at the 2011 TESOL National Convention (G2) Grammar errors are noticeable, but significant portions (chunks of 3-4 sentences) show mastery of basic grammatical forms. There may be consistent errors in the choice of prepositions and/or articles. There is some evidence of the ability to use complex language accurately, although not necessarily consistently, such as advanced verb tense forms, subordinate clauses, gerunds, modifiers, etc. Students at the College Prep level need to be able to consistently use basic grammatical forms accurately, and be ready to receive support in developing more advanced structures within the context of learning to write academic essays.

  12. The second descriptor (G2) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE: Presentation at the 2011 TESOL National Convention G2 (OS3) The writing is of a sufficient amount (i.e., a longer paragraph) to allow the reader to identify one or more main points. There is some attempt at organization and support or development. • Placement in College Prep level (OS4) The writing is a short paragraph or collection of sentences. There is no evidence of logical organization. There is irrelevant information and/or the writing may not address a clear topic. • Placement in Advanced level

  13. Grammar Descriptors Presentation at the 2011 TESOL National Convention (G3) Most sentences have grammatical errors of different types. There are frequent errors even with basic forms and with word order. Students at the Advanced level need to be ready to receive support in developing accuracy in the use of grammatical structures within the context of learning to write longer and more complex paragraphs.

  14. The third descriptor (G3) for GRAMMAR has the following options for ORGANIZATION/STRUCTURE: Presentation at the 2011 TESOL National Convention G3 (OS3) The writing is of a sufficient amount (i.e., a longer paragraph) to allow the reader to identify one or more main points. There is some attempt at organization and support or development. • Placement in Advanced level (OS4) The writing is a short paragraph or collection of sentences. There is no evidence of logical organization. There is irrelevant information and/or the writing may not address a clear topic. • Placement in Intermediate level

  15. Data examined Presentation at the 2011 TESOL National Convention • 4 semesters of college prep students (n=75) • Entrance and exit tests • Computer-based test • Essay

  16. Improvement in exit test scores Presentation at the 2011 TESOL National Convention

  17. Using the new rubric... Presentation at the 2011 TESOL National Convention • Rescored entrance essays: • 7 students scored lower than college prep on their entrance essays (score = or < 3.5) • 10 scored at college prep level (score > 3.5) • 5 scored at academic level (score > 4.5) • Rescored exit essays: • 2 scored lower than college prep • 13 scored at college prep level • 7 scored at academic level

  18. Can we score the new essay reliably? Presentation at the 2011 TESOL National Convention 20 essays (of 44 total) had unanimous scoring across all scores (between 3-5 scorers) All the rest had 3 of 4 scorers in agreement, with 3 exceptions For the 3 essays with differences of more than 1 level (i.e., one scorer assigned a 5, another a 4, and another a 3), in each case the essay was particularly difficult to read (bad photocopy).

  19. Some tentative conclusions... Presentation at the 2011 TESOL National Convention • Average gains of approx. ½ level on all measures (ACCUPLACER, Old Essay & New Essay) tentatively indicate that program expectations that students are ready to move up to Academic level work after 1 semester of College Prep are problematic. • New scoring system • Fast and easy to use, and also looks promising in terms of reliability between program faculty. • Where there was disagreement in scoring using the new essay scoring system, it was largely at the grammar level—indicates that further consideration of the grammar level descriptors is needed.

  20. Recommendations we are considering Presentation at the 2011 TESOL National Convention Short-term… • Adding an optional fifth level which will be a combination of General Education courses and ESL courses (GED: Academic Writing, GED: Oral Communication Skills, ESL: Reading, ESL: Academic Listening & Note Taking). Long-term… • Repeat this study with a more “traditional” essay rubric to see what results we get. • Repeat this study with students at the other three levels to see how their exit scores compare with their initial placement results. • Are we expecting them to make unrealistic progress over the course of 15 weeks? • Should we restructure the program, and splitting the current 4-level (4-semester) curriculum into five or six semesters?

  21. Implications Presentation at the 2011 TESOL National Convention • Importance of multiple measures (computer test scores, essay scores, interview, previous course grades) • Importance of faculty/instructor input in the placement/advancement process • “highly recommend that multiple measures be employed in any EAP placement process and that regardless of the tools employed, faculty be involved in interpreting the results” (James & Templeman, 2009). • Importance of gathering assessment data to track student progress and for program improvement

  22. Selected References Presentation at the 2011 TESOL National Convention James, C. & Templeman, E. (2009, Spring). A Case for Faculty Involvement in EAP Placement Testing. TESLCanada Journal/Revue TESL Du Canada 26(2), pp.82-99. Brunk-Chavez, B. & Fredericksen, E. (Fall/Winter 2008). Predicting Success: Increasing Retention and Pass Rates in College Composition. Writing Program Administration, 32(1), pp.76-96. Andrade, H.G. (Dec. 2005). Understanding Rubrics. Learnweb. http://learnweb.harvard.edu/alps/thinking/docs/rubricar.htm Andrade, H.G. (2000). Using Rubrics to Promote Thinking and Learning. Educational Leadership 57(5), pp.13–18. Wilson, M. (2006). Rethinking Rubrics in Writing Assessment. Portsmouth: Heinemann.

  23. Questions? Presentation at the 2011 TESOL National Convention Please contact us with comments, suggestions, questions… BreanaBayraktar breana.bayraktar@gmail.com This presentation will be posted at: http://breanabayraktar.wordpress.com

More Related