Comprehensive English Language Learning Assessment (CELLA)

1. Comprehensive English Language Learning Assessment(CELLA) TRAIN THE TRAINER TRAINING SESSIONS -- January 2009

2. Welcome & Introductions • Lori Rodriguez, Chief, Bureau of Academic Achievement through Language Acquisition • Wendy Nucci, ETS CELLA Program Manager • Paul Rybinski, ETS Assessment Development Lead, English Language Learning • Zulma Torres, ETS CELLA Program Manager • Lorenzo Murolo, Inbound Operations Manager

3. Welcome & Introductions Review Content of Tests Who Should be Tested Functional Level Testing What is it? How is it administered? Administering Level A Administering Levels B-D Morning Agenda

4. What’s New for 2009 2009 Comprehensive Schedule Training Materials 2009 Supplemental Order Process Responsibilities Test Administrator School Coordinator District Coordinator 2009 DFAs and Scoring Guides Answer Docs and Pre-ID Labels Question and Answers Session Afternoon Agenda

5. What is CELLA? • CELLA provides… • evidence of program accountability in accordance to Title III of NCLB • data on students over time • information about the language proficiency of students across the State • diagnostic information about students’ strength and weakness in English • fair, reliable, and valid information about ELLs in grades K through 12

6. How was CELLA developed? • CELLA was funded by a grant from the U.S. Department of Education. • A collaboration with AccountabilityWorks and a consortium of five states: Florida, Maryland, Michigan, Pennsylvania, & Tennessee • As the grantee, AccountabilityWorks managed the work of the consortium • ETS was awarded the subcontract to develop the test, ancillary materials, conduct the field test and develop the scoring programs • The State provided substantial input and the majority of field tested students

7. CELLATest Content

8. Overview of CELLA • Test Sections: Listening Speaking Reading Writing • Test Levels: Level A (Grades K-2) Level B (Grades 3-5) Level C (Grades 6-8) Level D (Grades 9-12) • Test Format: • Listening: All Multiple Choice • Speaking: All Constructed Response • Reading: All Multiple Choice • Writing: Multiple Choice & Constructed Response

9. CELLA Item Types • Writing Multiple Choice • Grammar, structure& written expression • Paragraph choices • Recognizing errors • Reading Vocabulary • Synonym • Antonym • Idiom • Root & Affix • Listening • Listen & Match • Picture Description • Short Talks • Extended Listening • Speaking • Oral Vocabulary • Speech Functions • Personal Opinion • Story Retelling • Graph Interpretation • Reading Comprehension • Main Idea • Detail • Inference/Prediction • Reference • Rhetorical Elements • Vocabulary in Context • Constructed Response • Writing sentences • Writing paragraphs

10. Test Document Configuration(On Grade Level – ALL Sections) Level A: Grade K - 2 Level B: Grades 3 - 5 Level C: Grades 6 - 8 Level D: Grades 9 - 12

11. Testing Decision Tree (Whom Do You Test?)

12. Which Students Should be Tested? • All students in grades K-12 classified as ELLs, with codes of “LY” or “LP” as of May 31, 2008 and in membership in the district at the time of testing • Former ELLs (code of “LF”) that were exited from the ESOL program on or after May 31, 2008 and before April 14, 2009 • Any student that enrolls after January 16, 2009 and before April 3, 2009, and is classified as an ELL must be assessed

13. Who is NOT Tested? • Students enrolling on or after April 6, 2009, with affirmative responses on the Home Language Survey are NOT included in the 2009 administration of CELLA

14. Functional Level Testing

15. What is Functional Level Testing? • Testing students’ language ability according to what they can actually do and NOT according to what is expected at their grade level • Used only for literacy skills (Reading & Writing), which must be learned, and NOT for oral skills (Listening & Speaking), which are more naturally “acquired” • Students may take the Reading and Writing sections at a level lower than grade level

16. Why administer Functional Level Testing? • There is a often wide disparity in skill levels of ELLs (especially in the higher grades) • To get reliable and accurate test results for students at all proficiency levels within each grade-span would require more test items • Test administration time is kept short by giving students only those items most appropriate for their skill level • Without functional level testing, measures of growth are inaccurate for very low performing students

17. Benefits of Functional Level Testing to the Student • Frustration during testing is minimized. • Abilities are more accurately reflected by the scores. • Interventions are more effectively and appropriately. • Results can be compared across districts more accurately.

18. CELLA Results • Administering the CELLA Reading & Writing sections according to students' functional level does NOT result in an "off-grade" version of the assessment • “Off-grade" testing typically indicates that student results are scored and reported against a different, lower standard. This is NOT the case with CELLA • Every student in a given grade span (e.g., K-2, 3-5, 6-8, 9-12), regardless of which test level is administered, will be reported as "English Proficient" against the same standard (or "cut-score") on the assessment's vertical scale

19. How To Implement CELLA Functional Level Testing • Teachers and CELLA Test Administrators should use the following tools to determine the correct Reading & Writing level to administer to students classified as ELLs at the time of testing: • 2009 CELLA Testing Decision Flowchart, AND • 2009 CELLA Functional Level Testing Decision- Making Rubric, • -OR- • CELLA Online Locator Test

20. Which students should NOT be tested with Functional Level Testing? Former ELLs (LFs) who exited the ESOL program on or after May 31, 2008 must be tested in Reading & Writing based on their grade level

21. 2009 CELLA Functional Level Testing Decision-Making Rubric • Developed based on the CELLA test items and the content demands in Reading & Writing required of students to reliably determine their level of English proficiency in these two domains • Teachers use a check mark next to the level of the student’s attainment in each of the four English skill areas • Students with 3 or 4 check marks in one column should take the test indicated in that column

22. 2009 CELLA FUNCTIONAL LEVEL TESTING DECISION-MAKING RUBRIC

23. How to Administer Functional Level Testing

24. Test Material for Functional • Level Testing

25. FUNCTIONAL LEVEL TESTING – Example 1 • Example: A student in grade 9 will take Level D, Listening and Speaking The student is eligible to take the Level C, Level B, or Level A Reading & Writing test OR OR ALL student responses are recorded onto the Level B, C, D Answer Document ALL Student responses are recorded on the Functional Level A Test Book

26. FUNCTIONAL LEVEL TESTING – Example 2 • Example: A student in grade 5 will take Level B Listening & Speaking and is eligible to take the Level A Reading & Writing test AND ALL student responses are recorded onto the Functional Level A Test Book

27. Level ATraining

28. Level A Sections

29. Administering Level A • The test should be individually administered to students in kindergarten. • Grades 1 and 2 may take Listening, Reading and Writing in small groups. • The One-on-One section must be individually administered • Grade 2 will take additional “extension” items in Reading & Writing

30. Administering Level A “Core” and “Extension” • The Core Section should be taken by all students in Grades K-2 • Listening & Speaking sections are identical for all students in grades K-2 • The Extension contains more challenging Reading & Writing items and is administered only to students in Grade 2 • Students in grades K and 1 receive fewer Reading & Writing items than students in Grade 2

31. Administering Level A • STOPPING POINTSare available for individually administered sections of CELLA: • Test administrators should stop the administration of that section only IF the student cannot respond to five questions in a row AND it is clear that the questions are above the student’s proficiency level

32. Level A Student Responses • Please demonstrate how students should mark their responses in their test book: Correct Incorrect

33. Test Administrators One-on-One Scoring Section Page 38 of the Level A Student Test Book

34. Proficiency in English Thorough understanding of how rubrics “work” Knowledge of and experience with the specific skills being measured Practice with rubrics Objectivity CELLA Speaking/One-on-One Administration Knowledge & Skills Needed to be an Effective Evaluator of Language Proficiency

35. Administering Level A One-On-One SectionSpeech Functions • Measures a student’s oral response to a specific prompt • Criteria include: • Appropriateness of information • Grammatical accuracy

36. Administering Level A One-On-One Section Speech Functions Rubric Review Activity • There are a total of 22 training tracks for Speech Functions • Refer to the Directions for Administration &Scoring Guides • Rubric is on page 49 • Training Items are on page 56 • Answer Key is on page 63 • Activity • Listen to training CD Tracks 1-2 • Discuss why the response scored a 2, referring to the rubric • Listen to CD Tracks 3-4 • Discuss why the response scored a 1 and 0 • Make sure you LISTEN to student responses, rather than read the transcript

37. Administering Level A • One-On-One SectionPersonal Opinion • Measures students ability to orally state and defend an opinion • Criteria include: • Clarity of response • Adequate support • Good control of grammar & adequate vocabulary

38. Administering Level A One-On-One Section Personal Opinion Rubric Review Activity • There are a total of 9 training tracks for Personal Opinion • Refer to the Directions for Administration & Scoring Guides • Rubric is on page 50 • Training Items are on page 57 • Answer Key is on page 64 • Activity • Listen to training CD Tracks 23-24 • Discuss why the response scored a 2, referring to the rubric • Listen to CD Track 27 • Discuss why the response scored a 1 • Make sure you LISTEN to student responses, rather than read the transcript

39. Administering Level A • One-On-One SectionStory Retelling • Measures a student’s ability to hear a story (while looking at sequential picture cues) and to then retell it with detail • Criteria include: • Comprehensive response • Vocabulary • Grammar • Fluency

40. Administering Level A One-On-One Section Story Retelling Rubric Review Activity • There are a total of 16 training tracks for Story Retelling • Refer to the Directions for Administration & Scoring Guides • Rubric is on page 51 • Training Items are on page 58 • Answer Key is on page 64 • Activity • Listen to training CD Tracks 32-33 • Discuss why the response scored a 4, referring to the rubric • Listen to CD Track 35 • Discuss why the response scored a 2 • Make sure you LISTEN to student responses, rather than read the transcript

41. Administering Level A • One-On-One SectionReading Aloud for Fluency • Measures reading fluency • Criteria include: • Rate • Accuracy

42. Administering Level A • One-On-One SectionReading Aloud for Fluency • Administration - Types of Errors: • Substitution • e.g., bird instead of bear • Mispronunciation • e.g., fell instead of fall • Words pronounced with an accent are counted as correct if they cannot be confused with other English words • Omissions • i.e., skipped words • If the student stops or struggles with a word for 3 seconds, you may tell the student the word and count it as an error

43. Administering Level A One-On-One SectionReading Aloud for Fluency • There are a total of 16 training tracks for Reading Aloud for Fluency • Refer to the Directions for Administration & Scoring Guides • Rubric is on page 53 • Training Items are on page 62 • Answer Key is on page 66 • Activity • Listen to training CD Tracks 48-49 • Discuss why the response scored a 4, referring to the rubric • Listen to CD Track 53 • Discuss why the response scored a 0 • Make sure you LISTEN to student responses, rather than read the transcript • Note: • If the student makes repeated errors on the same word, count the error only once • Repetitions and self-corrections are not counted as errors

44. Level B-D Listening/Speakingfor Grades 3-12

45. Administering Levels B-DListening & Speaking • Grades: • Level B (3-5) • Level C (6-8) • Level D (9-12) • Test design is identical across levels for Listening & Speaking* • Listening all multiple-choice; Speaking all constructed-response • Listening is group administered; testing time is about 25 minutes • * Level B is the only exception with one additional item

46. CELLA B-D Listening & Speaking Sections • Listening • Listen and Match • Picture Description • Short Talks • Extended Listening • Materials Needed • Levels B, C & D Answer Sheet • Directions for Administration & Scoring Guides • Listening & Speaking test book • Listening CD • Speaking • Oral Vocabulary • Speech Functions • Personal Opinion • Story Retelling • Graph Interpretation • Materials Needed • Levels B, C & D Answer Sheet • Directions for Administration & Scoring Guides • Listening & Speaking Test Book • Training CD for Speaking

47. Test Level Selection Area

48. Level C Training

49. Administering Level C Speech Functions • Measures a student’s oral response to a specific prompt • Criteria include: • Appropriateness of information • Grammatical accuracy

50. Administering Level C Speech Functions Rubric Review Activity • There are a total of 45 training tracks for Speech Functions • Refer to the Directions for Administration & Scoring Guides • Rubric is on page 186 • Training Items are on page 194 • Answer Key is on page 204 • Activity • Listen to training CD Tracks 1-2 • Discuss why the response scored a 2, referring to the rubric • Listen to CD Tracks 3-4 • Discuss why the response scored a 1 and 0 • Make sure you LISTEN to student responses, rather than read the transcript