Adult ESL Literacy Impact Study: Instruction and Outcomes

1. Adult ESL Literacy Impact Study:Instruction and Outcomes Larry Condelli Stephanie Cronen American Institutes for Research, USA LESLLA Sixth Annual Symposium Cologne, Germany August 26, 2010

2. Overview of Presentation • Summary of Study Purpose and Design • Impact of the Intervention • Content of Instruction • Student Attendance • Discussion and Implications of Findings

3. Evaluation of an ESL Literacy Intervention: Sam and Pat • Structured language approach • Adapted from Wilson Reading System • Never before evaluated for ESL • Literacy activities organized around basal reader/workbook • Heavily phonics-based

4. Sam and Pat: Instructional Approach • Direct instruction, transparent • Rules explained, modeled, practiced • Controlled text, vocabulary and grammar • Words match phonics already learned • Sequential • Easy to hard in defined steps • ESL instruction to support literacy

5. Sam and Pat: Literacy and Language Skills Covered • Phonics for reading and writing • Sight words • Oral reading for accuracy and fluency • Reading comprehension • Vocabulary • Speaking and listening • Grammar

6. Research Questions • How effective is instruction using the intervention in improving the English reading, speaking and listening skills of low-literate adult ESL learners? • Is the intervention more effective for certain groups of students (e.g., language, literacy level)? • Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts?

7. Study Design • 10 adult ESOL centers across USA • Paired intervention and “normal” ESL literacy classes (34 total) • Random assignment of students and teachers • 1,344 students participated for one term • Minimum 5 hours/week 10-12 weeks instruction with approach • Other instruction also provided • Each class conducted twice over a year

8. Student Flow in the Study Intake NL Literacy • Student applies to center • Assessed for NLL • Recruited into study • Gives informed consent • Random assignment to class • Pretests administered • Instruction • Posttests administered Recruited into Study Informed consent Random Assignment Pre-test Post-test Instruction

9. Students Assessments • Phonics and decoding • Word attack • Letter/Word ID • Reading comprehension • Vocabulary (ROWPVT) • Listening, oral expression

10. Sam And Pat Teachers in the Study • All teachers randomly assigned • 3-day teacher training on curriculum • Follow-up visits by trainers • Classroom observations to monitor fidelity • Refresher webinar at start of second term

11. Students in the Study:“True” Literacy (LESLLA) • Little or no literacy in native language • Limited oral English • Education: 0-3 years • Languages: • Haitian-Creole • Spanish • Burmese • Others

12. Students in the Study: Non-Roman Alphabet Literate • Some Literacy in native language with non-Roman script • Mean education: 6 and more years • Limited oral English • Languages: • Armenian • Arabic • Farsi • Chinese • Others

13. Findings and Impacts: Student Test Results

14. Main Impact Analyses • No significant impacts on literacy or language outcomes for full sample (below) and subsamples • But all students gained on pre-post tests

15. Pre-post Differences, All Students

16. Lack of Impact of Sam and Pat • The study took place in a challenging environment that makes having/finding an impact difficult, but that represents reality • Low exposure and instructional time in adult ESL class • Short class duration • Limited instructional time and irregular attendance • Training teachers • Lack of specialized training in literacy • Short training time available

17. Findings and Impacts: Instruction: What is taught?

18. Classroom Observations • Literacy development instruction: • Pre-literacy • Phonics • Fluency • Reading strategies & comprehension • Writing • ESL Instruction: • Oral language development • Grammar, etc. • Vocabulary • Socio-cultural knowledge • Functional literacy

19. Instructional Content by Interval (%)

20. Instruction in Literacy Classes • In control (“normal”) classes, little literacy taught- Why? • Importance of oral language for daily life • Student expressed needs • Lack of teaching on how to teach LESLLA students • More diversity of instruction in control classes • More NL use and bringing in outside • Less constrained by curriculum and study needs

21. Findings and Impacts: Attendance

22. Attendance and Outcomes

23. Effect of Attendance on Test Scores • Instruction and outcomes correlated • Instruction appears to have an effect on learning • Relationship is weak • Larger effect on reading outcomes • May mean literacy gains more sensitive to gain, may be testing artifact

24. Observations and Discussion Implications of Possible Findings for Practice and Research for LESLLA Students

25. Implications for LESLLA Practice • Sam & Pat is no more effective than other types of ESL literacy instruction • Under what conditions, then, would Sam & Pat be a good choice for instruction? • Attendance relates to instruction, weakly • Is this what you expect? How could it be increased? • Teachers of literacy classes teach little literacy • How can we improve this --through curriculum and teacher professional development? • What are some potential explanations for these results from your perspective?

26. Thank you! • Contacts • lcondelli@air.org • scronen@air.org • Enjoy Koeln and LESLLA