460 likes | 694 Views
Bringing Educational Innovation to Scale: Top-Down, Bottom-Up, or A Third Way?. Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen, Don Compton, Chris Lemons, Wen Zhang Vanderbilt University Chris Schatschneider Florida State University R305G04104
E N D
Bringing Educational Innovation to Scale: Top-Down, Bottom-Up, or A Third Way? Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen, Don Compton, Chris Lemons, Wen Zhang Vanderbilt University Chris Schatschneider Florida State University R305G04104 Institute of Education Sciences
Big Q and Presentation Focus • If we can think of Grade 2-6 Peer-Assisted Learning Strategies (PALS) as a “best-evidence” reading practice, how do we scale it up? More importantly, how do we scale up any worthy educational innovation? • This was the general Q of an IES Goal 4 study conducted between 2004-2010. Years 1 & 2, at kindergarten; Years 3-5, at grades 2 thru 5. • Why this question is important practically and theoretically.
Presentation Overview • Purpose of PALS-Reading • Grade 2-6 PALS program • Challenges involved in scaling up • Re-thinking scaling up: “top-down” & “bottom-up” • Findings
Purpose of PALS • Supplements the general education core program • It is a peer-mediated program, and facilitates intensive practice of core academic skills • Implemented 3 times per wk in reading; twice in math • Creates a structure for teachers to differentiate instruction: many simultaneous peer-mediated lessons rather than one teacher-directed lesson • PALS-R: K, first grade, grades 2-6, high school • PALS-M: K, first grade, grades 2-6
PALS Research • Originally based on “Juniper Gardens Classwide Peer Tutoring” (e.g., Greenwood et al., 1989) • More than 15 yrs of school-based research • Title I and Non-Title I schools; urban and suburban • High, average, and low achievers • Students with LD • “Validated Practice” status (USDE, WWC, BEE)
Grades 2-6 PALS Partner Reading Paragraph Shrinking Prediction Relay
Partner Reading • Conducted for 11-12 minutes • Stronger reader reads aloud for 5 minutes • Weaker reader reads same text aloud for 5 minutes • Weaker reader retells story for 1-2 minutes • Readers read quickly, correctly, and with expression • Coaches listen, correct mistakes, and mark points (1 point for each correctly read sentence and 10 points for story retell)
Paragraph Shrinking • Conducted for 10 minutes • For 5 minutes: • Stronger reader reads new text aloud, summarizing paragraph by paragraph • Name the most important who or what (1 point) • Name the most important thing about the who or what (1 point) • Shrink it to 10 or fewer words (1 point) • For next 5 minutes: • Weaker reader reads new text aloud, summarizing paragraph by paragraph (as above) • Coach listens, corrects mistakes, and marks points
Prediction Relay • 5 minutes, stronger reader read new text • Makes prediction (1) • Reads half page (1) • Checks prediction (1) • States main idea (3) • Makes new prediction • Continues to read • 5 minutes, weaker reader continues on in new text, with the same activities • Coach listens, corrects mistakes, and marks points
Support to Teachers in K-PALS & Grade 2-6 PALS R&D • During program development and validation, research assistants provided in-class help 1x or 2x per wk. • Support is costly, finite, and typically disappears during scale up. Absent support, quality of implementation often suffers. • Obvious Q: How to successfully de-couple an effective intervention from a support system and “brand name” (culture)? How to sustain and scale-up?
Challenges to Scaling Up • Just described is the well known problem of moving an innovation from A to B. • Less well known problem: sustaining an innovation across time, or dealing with “the changing counterfactual,” • Case in point: K-PALS.
Phonological Awareness Effects by Achievement Level for 1990s Data
Pseudo-word Reading Effects by Achievement Level for 1990s Data
Teacher-led phonological awareness Peer practice Sound-letter correspondence Sight words reading Word reading Sentence reading Book reading Points earned for doing activities Elements of K-PALS
Moment of Truth (for the melodramatically-inclined) • Fear…depression… • Thinking differently: Top-down vs. bottom-up • “Now at starbucks: A rebound”
4 Activities: Partner Reading (10 Minutes) Retell (2 Minutes) Paragraph Shrinking (10 Minutes) Prediction Relay (10 Minutes) 35 Minutes per Session Reciprocal peer tutoring A motivational peer reinforcement system Emphasis on fidelity – Using PALS the way it was designed, without modifications. Top Down PALS
Core Elements of PALS: 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking 35 minutes per session minimum A motivational peer reinforcement system Customized Elements of PALS Small “tweaks” Big changes Flexibility Bottom Up PALS
Participating Teachers in Years 3-5 (Grade 2-6 PALS) Two cohorts of 3rd-, 4th-, and 5th-grade teachers: • Cohort 1 • Entered study in 2006-07 • Cohort 2 • Entered study in 2007-08 Two years of study participation: • Year 1 • Assigned randomly to PALS or Control • Year 2 • PALS Teachers selected Top Down or Bottom Up PALS • Control Teachers remained in Control group
Student Measures in Years 3, 4, & 5 Academic Measures • Test of Word Reading Efficiency (TOWRE) • Woodcock Reading Mastery Test-Revised (WRMT-R) • Word Identification Subtest • Wide Range Achievement Test (WRAT) • Letter and Word Identification Subtests • Comprehensive Reading Assessment Battery (CRAB; 2 passages) • Oral reading (1 min & 3 min) • Comprehension (10 open-ended questions) • CBM Maze Task (2 passages) • Correct maze choices made in 2.5 min • Iowa Test of Basic Skills (ITBS) • Reading Comprehension • Vocabulary Student Characteristics • Demographics • SWAN • Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses • Teacher ratings • Teachers rated each student’s effort in reading and behavior in the classroom
Teacher Measures in Years 3, 4, & 5 Classroom Measures • PALS Calendars • PALS Fidelity • Language Arts Observation • Classroom Atmosphere Rating Scale (Wehby) • Survey of Enacted Curriculum (SEC): English and Language Arts Teacher Characteristics • Demographics • Berends teacher survey (assesses school climate, teacher professional development, teacher efficacy, etc.)
Procedures • Pretesting (September-October) • PALS Workshops (September-October) • Year 1: All teachers attend same workshop • Year 2: Separate TD and BU workshops • PALS Implementation (~18 weeks) • Teachers implemented 3 times per week for 35-40 min • Weekly classroom visits from project staff • Three “booster” sessions for TD and BU PALS teachers • Two fidelity observations • Language arts observations in PALS and Control classrooms • 45-60 min • Momentary time sampling of a variety of reading instructional components • Supplementary field notes • Posttesting (March-May)
Analysis • Create latent pretest and posttest variables combining 5 reading measures into 1 • Create a latent change score • Produces an “error-free” change value • Run 2-level HLM analyses • Outcome: Latent change score • Variables: Treatment condition (TD, BU, Control); Site (TN, MN, TX); latent pretest score • Random effects: Level 2 teacher effects; ICC = .10 • Test comparability of groups on variables plausibly related to selection of TD or BU
PALS vs. Controls across cohorts (2007-08, 2008-09), sites, and TD/BU
Did we successfully scale up? • Yes: PALS appears to be generally effective. • BU PALS classes appeared to outperform TD PALS and Controls under the following conditions: • Core elements of PALS including a 35 minute PALS session • Support provided by PALS staff
Potential Mediators:No Differences Between TD & BU • Fidelity year 1 (ALL Teachers TD) • Were teachers who later became TD or BU better PALS implementers in their 1st year? • ALL PALS activities compared. • Fidelity year 2 (TD vs. BU year) • When teachers chose between TD and BU PALS, did their PALS fidelity change? • Examined core elements of PALS.
Classroom-based Potential Mediators: No Differences • Latent Change-Year 1 • Indirect method of looking at overall teacher efficacy AND differential effect of PALS implementation in year 1 • Number of PALS Sessions • Did one group implement more PALS than the other? • Measure of Classroom Environment • Climate, management, behavior, etc.
Other Potential Mediators • No differences were found for: • Scale of attitude about student needs • Scale of attitude of about school programs • Scale of attitudes about self-efficacy • Scale of attitudes about principal leadership