Implementation and Evaluation of Project REAL Rural Early Adolescent Learning Allen Murray ACRES March 14, 2009

1. Implementation and Evaluation of Project REAL Rural Early Adolescent Learning Allen Murray ACRES March 14, 2009

2. Key Personnel for Project Real • Principal Investigators Thomas Farmer, Ph.D. & Jill Hamm, Ph.D. • Investigators Kimberly Dadisman, Ph.D. & Linda Mason, Ph.D. • Intervention Staff Allen Murray, Intervention Director Kirsten Feil Abby Hoffman • Project Support Staff Mallory Vinson, Project Manager Jenny Westrick Kelli O’Brien Courtney Mann

4. What is NRCRES? • The National Research Center on Rural Education Support (NRCRES) was established in 2004 with funding from the Institute for Educational Sciences of the U. S. Department of Education. The center is based at the University of North Carolina at Chapel Hill. • NRCRES conducts a focused program of control-trial research that addresses significant problems in rural education. More than 40% of all American schools are in rural areas and 30% of all students attend rural schools. The research and development work of NRCRES will seek solutions that will improve the quality of rural education.

5. What is REAL • In short, Project REAL is a Professional Development Program for teachers in rural schools (with a focus on teachers of early adolescents). • It is designed to help them promote the academic, behavioral, and social adjustment of rural youth as they transition from childhood to adolescence. • The program is designed to help support youth who are most likely to experience difficulty in school, but the strategies and techniques have been carefully selected to promote the engagement of all students including typical learners and high achieving students. • A primary emphasis of this program is to help teachers establish classroom and instructional contexts responsive to a broad and diverse range of learning needs.

7. Overview ofProjectREALSchools • Common Challenges: • Poverty and changing employment base; • Financial impact of large geographic distances; • Shrinking population of communities and schools; • Changing economic foundation of communities. • Teacher recruitment, retention, and licensure issues; • Common Strengths: • Faculty are a part of the community; • Schools are focal point for community; • Home ownership rates above the national average • Generational affiliation with local school and/or school system; • Social indicators (church, civic clubs) show strong community ties, • Higher rates of participation in extra-curricular activities

8. Training and Research Sites • Pilot Sites: • 2 Appalachian states, 4 middle school transition schools • Full Implementation Sites: • Northern Plains: 4 k-12 schools (completed) • Appalachia: 4 middle school transition schools (completed) • Midwest: 2 k-8, 2 middle school transition schools (completed) • Southwest: 4 middle school transition schools (completed) • Deep South: 4 k-12 schools (completed) • Southeast: 2 k-8, 2 middle school transition schools (5/09) • Far West, 4 middle school transition schools (5/09) • Pacific Northwest, 4 k-8 schools (5/09) Total Number of Districts/Schools: 28 districts/56 schools Total Number of Teachers: 392 Total Number of Students: approximately 4000

9. Rationale for Project Design • The wide variances in grade level Standard Course of Study made a content-specific focus and a universal design mutually exclusive. • So the focus is on addressing issues that impact academics for all populations and across curricula. • Small school size means the grade-level team is often very small (and may consist of only one teacher). • Small teams can cause teachers to work in isolation, with little opportunity to see others model new teaching techniques, curricula, or management strategies. • The program combines a topic-focused training model with a teacher-focused consultation model.

11. Training The agenda is owned by the trainer. Trainer has information to give others. Trainer’s job is to get that information out. A good model when many folks need the same content. Consultation The agenda is owned by the teacher. The teacher needs help in finding a solution. The consultant’s job is problem-solving. A good model when individuals or schools have unique needs. Trainingvs.Consultation

12. Training is available to local schools; Consultation may be the greater need • In many ways it is easier to provide training than consultation. Training is more time efficient, can be done in larger scale, can be made universal, and can be individually pursued (self-instructional modules, etc.). • Consultation, by its very nature, is a person to person process; it is not adaptable to self-instructional models (by reading, videos, on-line, etc.). It is time-intensive, requires knowledge of both the teacher and community, and needs a face-to-face component.

13. TrainingConsultation Sets the stage for • Our training is not designed as stand alone, • Instead, it is designed to set the stage for consultation, • Consultation is the “real goal”, especially for special educators or teachers of students with disabilities, • Consultation is student specific, with hope that it will generalize.

15. Project REAL Delivery • Project Real combines in-person training with a technology enhanced (distance delivery) consultation to both regular education teachers and specialists who have “at-risk” students in their class • Focus is on how the transition to early adolescence impacts academic performance, both in class-wide issues and for “at-risk” students (as defined by teacher). • Prior to the beginning of intervention, relationships are formed during site visits. These relationships, and the in-person summer institutes, keep the video-conferences from feeling “artificial”. • Site visits are made during the intervention year to observe classrooms and follow-up on training topics.

16. Universal Components Site Visits, Needs Assessments, and Summer Institutes Training modules on topics including: Early Adolescent Development Motivation and Academic Engagement Instruction for low-achieving students School and classroom social dynamics Information processing Literacy support and Writing Targeted Components Video-conference consultation with Project REAL staff A forum for intervention specialists to facilitate discussions of: Training modules Strategies for at-risk students Strategies for class-side issues Project REAL Intervention

17. Sample Training Unit • Each module typically consists of some or all of these components: • Narrated PowerPoint Presentation, • Brief (non-academic) article, • Further readings on the topic for those teachers interested in a more in-depth look at the topic. • A video illustrating a teaching technique or topic in adolescent development. • An activity that teachers will complete with each other or in class.

19. Research Design for Project REAL • Intervention schools matched with control schools • 1/2 with middle school transition configuration • 1/2 alternative configuration (e.g., k-8, k-12) • Schools are matched closely on demographic data • Baseline student data is collected in the spring of 5th grade and fall of 6th grade. • Student outcome data on school adjustment and academic achievement collected in spring of 6th grade. • A follow-up year of data is collected. (7th grade, fall & spring)

20. Plan of Analysis Hierarchical Linear Regression Models • Control variables: • Baseline, fall scores as controls • Ethnicity (Native American, White), Gender • Intervention Main Effect • Intervention by Ethnic Group, Intervention by Gender effects Tests for improvement to fit of model for block of variables; Test for significance of beta weight within significant block of variables

21. Three Key Outcome Domains • Perceptions of Social-Academic Context • Peer norms for effort and achievement(Hamm, 2001) • Emotional risk of participation (Hamm & Faircloth, 2005) • Bullying Context (Song, 2008) • Affective Relationships withSchooling • Sense of Belonging (Hagborg, 1994; 1998) • School Valuation (Voelkl, 1996) • Achievement • End-of-year state-level standardized test (school records) • Grades (school records)

23. Findings for Intervention Effects: Northern Plains Site Site Description • Recruited from all 4th, 5th, & 6th grade classrooms of four public k-12 schools in a state in the Northern Plains • 72% agreed to participate, N=165 student participants • 45% Native American, 55% White • 92 boys, 73 girls • Schools were eligible for U.S. Department of Education’s Rural and Low-Income School Program (RLISP) • locale code 7 or 8 and at least 20% of students are from families living below the federal poverty level • In our sample, 64.6% of students were eligible for subsidized meals through the National School Lunch Act

24. Results After controlling for the previous year scores on each outcome, and for student demographic characteristics, students in Project REAL intervention schools, compared to their peers in matched control schools reported . . . • More supportive peer norms for effort and achievement; • More peer protection from bullying; • A greater sense of school belonging; • Greater valuing of school; • Higher year-end grades.

25. Intervention effects were particularly pronounced for Native American (versus White) students for: • State-level standardized test scores • Sense of belonging • Perceived peer norms for effort and achievement

33. Hierarchical Regression Analysis of Indicators of Students’ Academic Achievement Note: Standardized Betas reported: +p <.06, *p < .05, **p < .01, and ***p < .001 (one-tailed).

34. Hierarchical Regression Analysis of Indicators of Students’ Affective Relationship with Schooling Note: Standardized Betas reported: *p < .05, **p < .01, and ***p < .001 (one-tailed).

35. Hierarchical Regression Analysis of Indicators of Students’ Perceptions of the Social Academic Context of Classrooms and School Note: Standardized Betas reported: *p < .05, **p < .01, and ***p < .001 (one-tailed).

36. Equivalence of Intervention and Comparison Groups on Selected Child and School Attributes Note. There were 165 participants (105 intervention and 60 control students) and 4 schools (2 intervention and 2 control schools).