270 likes | 389 Views
Team Read Improving Literacy in the Seattle School District. PA 590C – Program Evaluation Fall 2007 Anne Debuyserie.
E N D
Team Read Improving Literacy in the Seattle School District PA 590C – Program Evaluation Fall 2007 Anne Debuyserie
“...a Seattle School district tutoring program dedicated to increasing the reading skills of elementary students through year-long coaching by high school students ... The program's goals for participating high-school tutors are to develop work experience, a sense of responsibility and accomplishment, and to learn the rewards of community service.” (Team Read Mission Statement, 1998)
What is Team Read? • Private-public partnership to improve literacy in Seattle Public Schools • Cross-age tutoring program using high school student as tutors • Started in March 1998 with a three-phase implementation • June 1999: 10 schools, 335 students, 300 coaches • June 2000: addition of 7 schools
Team Read Stakeholders • Craig McCaw (and wife Susan): Businessman • John Stanford: Seattle Schools Superintendent • Joan Dore: Seattle Schools Reading Specialist • Tricia McKay: Team Read Program Mgr • Team Read Advising Board: Seattle School District Staff, Alliance for Education pgm representatives, Mc Caws, community representatives. • Additional investors joined by year 2.
Who is Team Read? • Coaches • Strict selection guidelines • High expectations from Team Read leaders • Compensation • Readers • School selection comes first • Student selection based on reading level • Eligibility based on need • First-come, first-serve basis • Low socio-economic background
Who is Team Read? • Site Coordinators • Part-time, teachers or school staff • Meet informally with Tricia McKay for feedback • Volunteers • Needed due to amount of work • Partnership with VISTA/Americorps and U. of Washington Pipeline • Assist site coordinators
Team Read Session • Two 1 hr session per week during school year • Standardized session format and structure • 3.00 PM: coaches arrive • 3.15 PM: snack with reader • 3.30 pm: reading session begins • 4.15 PM: reading session ends/ coaches fill out paperwork • 4.30 PM: reading coaches leave • Opportunity for site coordinators to be creative
Team Read Evaluation Results • 1st year evaluation: not as positive as expected • 2nd year: promising but Team Read’s impact not as promising as hoped • Tricia McKay disappointed by results and wonders if findings provide her with the information needed for improvements • Questions about the accuracy of results.
Team Read Evaluations • Evaluation conducted by Margo Jones, statistician by trade. • 17 schools in the program by June 2000 but only 10 used in the evaluation. • 3 main objectives to show Team Read’s Impact: • Goal#1: Do the reading skills of the student readers improve significantly during their participation in the Team Read program? • Goal#2: How does the program affect the reading coaches? • Goal#3 What is working well, and what can be improved?
Evaluation Methods Goal #1 • Goal #1: Reading skills. Two-fold approach
Evaluation Methods Goal #1, Cont’d • 2nd method: determine proportion of Team Read participants who moved from below grade level test score to at-or above level compared to all students from entire district • Note: one of the selection criteria to be in Team Read is to be within bottom 25% of district reading test score
Evaluation Methods Goal #1 • Both methods for goal 1 are flawed. • Note from Jones in report confirming problems with different tests. • “The question then becomes ‘Is it reasonable to assume that the pre- and post-tests measure the same skills?’ The evaluator adopted a correlation criterion for answering this question. If the correlation between pre- and post-tests was near or above .8 (see footnote 5) (as was the case for the 2nd grade, where pre- and post-tests were essentially the same), the pre- to post-test change score was interpreted as a gain score. If the correlation coefficient was much less than .8, no analysis was performed.”
Evaluation Methods, Goals #1- Assessment • Visible problems with tests: • Pre and post-tests do not measure the same skills except in 2nd grade • All Team Read participants are within the bottom quartile of the reading test scores and can’t be compared with students district-wide. • Analysis could not be carried out for some grades (3rd) due to incompatibility of reading tests
Evaluation Method, Goal #2 • Method: questionnaire given to coaches during their last week of coaching. • Goal: to assess the impact of Team Read on coaches. • Identify 3 main areas of job satisfaction: • How positive is their experience with TR? • Extent to which the coaches feel that TR helps students readers • How supportive is TR’s site coordinators and staff?
Evaluation Method, Goal #2 - Assessment • Jones results showed that coaches were satisfied with their job and felt a sense of pride and accomplishment. • Test performed by Jones was successful in meeting its goals, unlike in Goal #1. • Some questions provided answers as to what to improve in the program. • Test is useful to the program manager
Evaluation Method, Goal #3 • Method: interviews with major stakeholders, one site visit, review of last 5 years’ research literature on cross-age tutoring and last 3 years research on best practices. • Goal: assess the overall progress of the program.
After the Evaluation • Margo Jones makes her recommendations based on her findings to improve the efficacy of Team Read. • Tricia McKay reads the evaluation but fail to see what the results are telling her about the program
Evaluation Problems • Academic v. Practitioner approach. • Result presentation. • Research design flaws. • Not all the data was analyzed • Comparison with dissimilar items • Statistical analysis stretched • Evaluation only covers 3 goals. • Lack of communication between the evaluator and the program manager.
Impact: 2005-2006 Reading Results From Team Read’s 2006 Annual Report. http://www.teamread.com/downloads/team_read_106903_report_proof.pdf
Team Read Update - 2006 • Program Evaluation Results for 2005-2006 School Year • 70% of 2nd graders and 52% of 3rd graders were reading at/above or approaching grade level • 54% of 2nd graders and 35% of 3rd graders gained greater than 1.5 grade levels in reading • 98% of the parents of 2nd & 3rd graders and 92% of their teachers reported increased reading skills as a result of participation in Team Read • 76% of 2nd & 3rd graders said that reading was more fun since joining Team Read