Using a Learning Information System to Monitor Student Progress and Enhance Instructional Outcomes

1. Using a Learning Information System to Monitor Student Progress and Enhance Instructional Outcomes Jim Ysseldyke, PhD, NCSP University of Minnesota

2. Website to Access PPT www.tc.umn/edu/~,jim

3. Learning Information System • In this presentation LIS refers to the continuous assessment of student progress and the provision of information on student progress to both the student and the teacher for purposes of modifying instruction and enhancing student outcomes.

4. Some Initial Observations • A current focus in schools on enhancing student outcomes (overall achievement, especially as reflected in numbers of students meeting standards). • A major bottleneck to improving teaching and learning is lack of systematic, usable information on individual student performance and progress at the classroom level.

5. Some Initial Observations • Publishers are producing comprehensive technology-enhanced progress monitoring systems that provide teachers with the data they need to differentiate instruction, group students on the basis of comparable goals, and manage/adapt instruction based on student performance.

6. Some Initial Observations • The incredible diversity of academic performance in today’s classrooms makes differentiated instruction virtually impossible, unless one is able to enlist the assistance of technology. • Example: 6th grade math class in MPS has an 11 year range in tested math skills.

7. Curriculum-Based Measures • Have been shown to be reliable and valid for enhancing the level of information educators need to modify individual instruction for students (Black & Wiliam, 1998; Deno, 1985; Fuchs & Fuchs, 1988).

8. Curriculum-Based Measures • Overall goal of CBM: to frequently assess ongoing work, monitor individual progress, provide informative feedback to students, adapt instruction as needed, and ultimately improve student overall performance.

9. A Study of the Efficacy of Accelerated Math • Accelerated Math is a Learning Information System; a progress monitoring systems that provides teachers with the information they need to differentiate instruction for individual students in general education settings. • Accelerated Math is a technology-enhanced system for monitoring student response to instruction.

10. Six Renaissance Principles • Increased time to practice essential skills • Match of instruction to student skill level • Direct and immediate feedback to teachers and learners • Personalized goal setting • Use of technology to process, store and report information • Universal success

11. Match to Algozzine/Ysseldyke Model of Effective Instruction • The A/Y Model is based on a systematic review of research on evidence-based principles, components, strategies and tactics for effective instruction. • The A/Y Model underlies Functional Assessment of Academic Behavior (Ysseldyke & Christenson, 2002), Strategies and Tactics for Effective Instruction (Algozzine, Ysseldyke & Elliott, 1997), and the Special Education Module Series (Ysseldyke & Algozzine, in press)

12. Components • Instructional Match • Relevant Practice • Direct and Immediate Informed Feedback • Checking for Student Understanding • Personalized Expectations • Academic Engaged Time • Student Motivation • Adaptive Instruction

13. What is Accelerated Math (AM) ? • An integrated, computerized continuous monitoring system. • Initial Assessment of student skills and competence (STAR Math). • Uses computerized adaptive testing model. • Increased precision and improved placement decisions.

14. What is Accelerated Math (AM)? • Teacher informed assignment of students to math “libraries” or graded sets of math objectives-AM. • Enhances ability to provide individualized instruction and practice to students. • Teacher is “hands-on” in the assignment and organization of math objectives for the classroom and individual students.

15. AM Classroom Process • The computer prints pages of problems for students to work on • Students record their answer on a bubble sheet and scan their answers • Students receive immediate feedback (TOPS report)

16. AM Classroom Process • Next Loop: • Each student is provided with a new set of practice problems. • Mastered objectives are spiraled back for skill maintenance (distributive practice). • In addition, mastered objectives are “banked” for testing student competence. • Non-Mastered Objectives are paired with new objectives for additional practice and instruction.

17. AM Classroom Process • Source of student work is not a standardized set of probes • Underlying AM is an item bank with millions of items aligned with specific objectives, and objectives aligned with specific math curricula • Computer uses algorithms to generate problem sets, and these will differ for individual students working on the same objective (this facilitates use of peer-assisted learning)

18. Reports • Individual student feedback (TOPS) • Diagnostic report (snapshot of every student and class as a whole) enables normative peer comparisons • Goal history report (report for each student showing progress toward individual goals) • Status of Class Report (alerts teacher to students who need help) • Standards Mastery Report (mastery information for a group of students)

19. Leadership Implementation Reports • These show administrators or related services personnel the progress of every student, by teacher, by grade, along with the amount of math practice students are doing. • Enables user to identify immediately the extent to which the program is being implemented. • E.g. our Minneapolis Data

20. Our Previous Studies of AM • Students in a mandated summer school program gained an average of 5.75 NCE on the NALT (Spicuzza & Ysseldyke). • AM resulted in significant gains for high, middle, and low functioning students, and increased the in-classroom incidence of behaviors known to enhance academic outcomes (as measured by E-BASS) (Spicuzza, Ysseldyke, Lemkuil et al)

21. Our Previous Studies of AM • Students enrolled in classrooms where AM was used to enhance the Everyday Math curriculum significantly outperformed those who had Everyday Math only (Ysseldyke, Spicuzza Kosciolek et al) • In a study of more than 2000 students, those who participated in AM significantly outgained those who did not (Ysseldyke & Tardrew).

22. Our Previous Studies of AM • Students whose teachers implement AM as it is intended to be implemented gain nine times as much as the control group (18 percentile points) in a single semester of intervention (Ysseldyke & Tardrew)

23. Research Questions • To What Extent are their differences in teacher implementation of the continuous monitoring and instructional management system? • To what extent do students whose teachers are high, moderate and low implementers of the system differ in gains on standardized tests.

24. Subjects • 133 classrooms in 9 schools in 8 school districts in 8 states. • The 133 classrooms were randomly assigned to experimental and control groups in elementary school. In middle school teachers taught one experimental and one control classroom. • Experimental group teachers used AM as an enhancement to their regular math instruction

25. Instruction in Mathematics With and Without CBM

26. Formulation of an Index of Intervention Integrity • We counted the number of objectives the students mastered. Overall range was 0-197. • To control for the impact of outliers, we converted scores by taking their square roots before forming groups.

27. Dependent Measures • STAR Math pre and post • Math subtests of the Tera Nova

28. Results • Surprise Result. Teachers did not implement AM with 1046 of the 2645 students in the experimental group (39.5%) • We found no systematic reason for failure to implement (gender, race, skill level, etc). • Six teachers did not implement the program with any of their students, and there was nothing “special” identifiable about these teachers.

29. Use of Residualized Change Scores • We used Residualized Change Scores as these allow for differences in pretest scores and give more conservative estimates of effects.

30. Results • Significant difference (p<.001) in the amount of gain in NCEs for students in the no-, low-, and high-implementing groups on both STAR Math and Terra Nova

31. Conclusions • Use of a continuous progress monitoring and instructional management system significantly enhances instructional outcomes in math.

32. Conclusions • High amount of non-implementation. It is imperative to monitor intervention implementation. • Shmoker notes that teachers are confronted with “initiatives du jour”, and unless there is explicit monitoring of implementation and some reward for doing so, teachers do not do so. • Goodlad noted that “Innovation is not enough, behind the classroom door even teachers who think they are implementing an innovation are often only twisting it right back into what they have always done” (p. 72).

33. Conclusions • Accelerated Math is a highly successful continuous monitoring and instructional management program.