Bringing Educational Innovation to Scale: Top-Down, Bottom-Up, or A Third Way?

1. Bringing Educational Innovation to Scale: Top-Down, Bottom-Up, or A Third Way? Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen, Don Compton, Chris Lemons, Wen Zhang Vanderbilt University Chris Schatschneider Florida State University R305G04104 Institute of Education Sciences

2. Big Q and Presentation Focus • If we can think of Grade 2-6 Peer-Assisted Learning Strategies (PALS) as a “best-evidence” reading practice, how do we scale it up? More importantly, how do we scale up any worthy educational innovation? • This was the general Q of an IES Goal 4 study conducted between 2004-2010. Years 1 & 2, at kindergarten; Years 3-5, at grades 2 thru 5. • Why this question is important practically and theoretically.

3. Presentation Overview • Purpose of PALS-Reading • Grade 2-6 PALS program • Challenges involved in scaling up • Re-thinking scaling up: “top-down” & “bottom-up” • Findings

4. Purpose of PALS • Supplements the general education core program • It is a peer-mediated program, and facilitates intensive practice of core academic skills • Implemented 3 times per wk in reading; twice in math • Creates a structure for teachers to differentiate instruction: many simultaneous peer-mediated lessons rather than one teacher-directed lesson • PALS-R: K, first grade, grades 2-6, high school • PALS-M: K, first grade, grades 2-6

5. PALS Research • Originally based on “Juniper Gardens Classwide Peer Tutoring” (e.g., Greenwood et al., 1989) • More than 15 yrs of school-based research • Title I and Non-Title I schools; urban and suburban • High, average, and low achievers • Students with LD • “Validated Practice” status (USDE, WWC, BEE)

6. Grades 2-6 PALS Partner Reading Paragraph Shrinking Prediction Relay

7. Partner Reading • Conducted for 11-12 minutes • Stronger reader reads aloud for 5 minutes • Weaker reader reads same text aloud for 5 minutes • Weaker reader retells story for 1-2 minutes • Readers read quickly, correctly, and with expression • Coaches listen, correct mistakes, and mark points (1 point for each correctly read sentence and 10 points for story retell)

8. Paragraph Shrinking • Conducted for 10 minutes • For 5 minutes: • Stronger reader reads new text aloud, summarizing paragraph by paragraph • Name the most important who or what (1 point) • Name the most important thing about the who or what (1 point) • Shrink it to 10 or fewer words (1 point) • For next 5 minutes: • Weaker reader reads new text aloud, summarizing paragraph by paragraph (as above) • Coach listens, corrects mistakes, and marks points

9. Prediction Relay • 5 minutes, stronger reader read new text • Makes prediction (1) • Reads half page (1) • Checks prediction (1) • States main idea (3) • Makes new prediction • Continues to read • 5 minutes, weaker reader continues on in new text, with the same activities • Coach listens, corrects mistakes, and marks points

10. Support to Teachers in K-PALS & Grade 2-6 PALS R&D • During program development and validation, research assistants provided in-class help 1x or 2x per wk. • Support is costly, finite, and typically disappears during scale up. Absent support, quality of implementation often suffers. • Obvious Q: How to successfully de-couple an effective intervention from a support system and “brand name” (culture)? How to sustain and scale-up?

11. Challenges to Scaling Up • Just described is the well known problem of moving an innovation from A to B. • Less well known problem: sustaining an innovation across time, or dealing with “the changing counterfactual,” • Case in point: K-PALS.

12. K-PALS:Data from 1998-99 StudyNashville

13. Phonological Awareness Effects by Achievement Level for 1990s Data

14. Word Reading Effects by Achievement Level for 1990s Data

15. Pseudo-word Reading Effects by Achievement Level for 1990s Data

16. Fluency Effects by Achievement Level for 1990s Data

17. Teacher-led phonological awareness Peer practice Sound-letter correspondence Sight words reading Word reading Sentence reading Book reading Points earned for doing activities Elements of K-PALS

18. Explaining Vanishing Effects Over Time: Segmenting

19. Explaining Vanishing Effects: Word Identification

20. Explaining Vanishing Effects: Pseudo-Word Reading

21. Explaining Vanishing Effects: Reading Fluency

22. Grade 2-6 PALS:Years 3, 4, & 5 of IES Study

23. PALS vs. Controls, 2006-2007

24. PALS vs. Controls, 2007-2008

25. Moment of Truth (for the melodramatically-inclined) • Fear…depression… • Thinking differently: Top-down vs. bottom-up • “Now at starbucks: A rebound”

26. Method

27. 4 Activities: Partner Reading (10 Minutes) Retell (2 Minutes) Paragraph Shrinking (10 Minutes) Prediction Relay (10 Minutes) 35 Minutes per Session Reciprocal peer tutoring A motivational peer reinforcement system Emphasis on fidelity – Using PALS the way it was designed, without modifications. Top Down PALS

28. Core Elements of PALS: 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking 35 minutes per session minimum A motivational peer reinforcement system Customized Elements of PALS Small “tweaks” Big changes Flexibility Bottom Up PALS

29. Participating Sites across 5 Years

30. Participating Teachers in Years 3-5 (Grade 2-6 PALS) Two cohorts of 3rd-, 4th-, and 5th-grade teachers: • Cohort 1 • Entered study in 2006-07 • Cohort 2 • Entered study in 2007-08 Two years of study participation: • Year 1 • Assigned randomly to PALS or Control • Year 2 • PALS Teachers selected Top Down or Bottom Up PALS • Control Teachers remained in Control group

31. Teachers by Year 2 Study Group

32. Students by Year 2 Study Group

33. Student Measures in Years 3, 4, & 5 Academic Measures • Test of Word Reading Efficiency (TOWRE) • Woodcock Reading Mastery Test-Revised (WRMT-R) • Word Identification Subtest • Wide Range Achievement Test (WRAT) • Letter and Word Identification Subtests • Comprehensive Reading Assessment Battery (CRAB; 2 passages) • Oral reading (1 min & 3 min) • Comprehension (10 open-ended questions) • CBM Maze Task (2 passages) • Correct maze choices made in 2.5 min • Iowa Test of Basic Skills (ITBS) • Reading Comprehension • Vocabulary Student Characteristics • Demographics • SWAN • Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses • Teacher ratings • Teachers rated each student’s effort in reading and behavior in the classroom

34. Teacher Measures in Years 3, 4, & 5 Classroom Measures • PALS Calendars • PALS Fidelity • Language Arts Observation • Classroom Atmosphere Rating Scale (Wehby) • Survey of Enacted Curriculum (SEC): English and Language Arts Teacher Characteristics • Demographics • Berends teacher survey (assesses school climate, teacher professional development, teacher efficacy, etc.)

35. Procedures • Pretesting (September-October) • PALS Workshops (September-October) • Year 1: All teachers attend same workshop • Year 2: Separate TD and BU workshops • PALS Implementation (~18 weeks) • Teachers implemented 3 times per week for 35-40 min • Weekly classroom visits from project staff • Three “booster” sessions for TD and BU PALS teachers • Two fidelity observations • Language arts observations in PALS and Control classrooms • 45-60 min • Momentary time sampling of a variety of reading instructional components • Supplementary field notes • Posttesting (March-May)

36. Analysis • Create latent pretest and posttest variables combining 5 reading measures into 1 • Create a latent change score • Produces an “error-free” change value • Run 2-level HLM analyses • Outcome: Latent change score • Variables: Treatment condition (TD, BU, Control); Site (TN, MN, TX); latent pretest score • Random effects: Level 2 teacher effects; ICC = .10 • Test comparability of groups on variables plausibly related to selection of TD or BU

37. Results

38. PALS vs. Controls across cohorts (2007-08, 2008-09), sites, and TD/BU

39. Results: Latent Change Score

40. Results by Student Rating:Latent Change Score

41. Did we successfully scale up? • Yes: PALS appears to be generally effective. • BU PALS classes appeared to outperform TD PALS and Controls under the following conditions: • Core elements of PALS including a 35 minute PALS session • Support provided by PALS staff

42. Potential Mediators:No Differences Between TD & BU • Fidelity year 1 (ALL Teachers TD) • Were teachers who later became TD or BU better PALS implementers in their 1st year? • ALL PALS activities compared. • Fidelity year 2 (TD vs. BU year) • When teachers chose between TD and BU PALS, did their PALS fidelity change? • Examined core elements of PALS.

43. Fidelity Across Years

44. Classroom-based Potential Mediators: No Differences • Latent Change-Year 1 • Indirect method of looking at overall teacher efficacy AND differential effect of PALS implementation in year 1 • Number of PALS Sessions • Did one group implement more PALS than the other? • Measure of Classroom Environment • Climate, management, behavior, etc.

45. Classroom-based Potential Mediators

46. Other Potential Mediators • No differences were found for: • Scale of attitude about student needs • Scale of attitude of about school programs • Scale of attitudes about self-efficacy • Scale of attitudes about principal leadership