Findings from the Indiana Acuity Efficacy Study: Under What Conditions Do Benchmark Assessments Impact Teaching and Lea

1. Findings from the Indiana Acuity Efficacy Study: Under What Conditions Do Benchmark Assessments Impact Teaching and Learning? Terry Spradlin June 29, 2012 2012 National Conference on Student Assessment

2. About the Center for Evaluation & Education Policy • The Center for Evaluation & Education Policy (CEEP) is a client-focused, self-funded research center associated with the School of Education at Indiana University • CEEP promotes and supports rigorous program evaluation and nonpartisan policy research primarily, but not exclusively, for education, human service and non-profit organizations • In the area of P-20 education policy, CEEP’s mission is to help inform, influence and shape sound policy through effective, nonpartisan research and analysis • www.ceep.indiana.edu

3. Contents • Overview of Indiana’s Comprehensive Assessment Plan • 2011-12 Testing Schedule (interim and summative) • Key Terms • Objectives of Indiana Acuity Efficacy Study • Study Design and Methods • Preliminary Findings of Comparison-Group Quasi-Experimental Study • Findings of Qualitative Analysis on High-Improvement Schools

4. I. Indiana’s Comprehensive Assessment Plan • Adopted by the Indiana State Board of Education on November 1, 2006 • Assessment Plan called for moving the state’s summative assessment ISTEP+ (Indiana Statewide Testing for Educational Progress-Plus) from the fall to spring and the creation and implementation of “formative/diagnostic” assessments • These changes and the new components were implemented during the 2008-09 school year and included: • Wireless Generation’s mClass Reading 3D and Math (Grades K-2) • CTB/McGraw-Hill’s Acuity Assessment System (Grades 3-8) • Phase-out of the Graduation Qualifying Exam (GQE) • Class of 2011 last to be required to pass GQE • replaced with end-of-course assessments in core subject areas • Moved ISTEP+ from fall to spring • (Students in grades 3-10 were tested twice during the 2008-09 school year; now testing in grades 3-8)

5. II. 2011-12 Fall Testing Schedule (Interim and Summative)

6. II. 2011-12 Spring Testing Schedule (Interim and Summative)

7. III. Key Terms • Assessment Definitions: • Formative Assessments – are short-cycle assessments that are administered regularly and often informally; allow for continual evidence collection of student knowledge; flexible and child-specific; can customize assessment to each child • Interim(/Benchmark) Assessments – administered multiple times during a school year (typically quarterly), usually outside of instruction, to evaluate students’ knowledge and skills relative to a specific set of academic goals. They inform educator decisions at multiple levels; results may be reported in a manner allowing aggregation across students, occasions or concepts; can help with the evaluation of effectiveness of remediation and intervention programs. *Approx. 14 states include interim assessments as optional component in their assessment programs • Summative Assessments – Rigid in content and administration, least frequently given, often carry highest stakes, used to inform policy and for accountability purposes; not designed to provide diagnostic information about individual students or to inform instructional decisions in short-term

8. III. Key Terms (cont.) • Acuity Predictive Assessments – is used to measure student growth and progress toward academic standards; are designed to mirror ISTEP+ blueprint (state summative test); predict likelihood of student passing ISTEP+ • Acuity Diagnostic Assessment – designed to reflect the curriculum that is anticipated to be taught in the classroom prior to its administration; intended to provide educators with detailed info for targeting and personalizing instruction

9. IV. Objectives of Acuity Efficacy Study Objectives of CEEP Study: • How can the system be improved? • Information intended to inform CTB and the IDOE about the changes and support needed to make the implementation and use of Acuity most effective during subsequent school years • To what degree does Acuity impact teaching and learning? • Evaluate the effects of the Acuity Assessment System on instructional practice and student achievement, particularly ISTEP+ • Under what conditions do benchmark assessments impact teaching and learning most? • Looked at success in high-improvement Acuity schools

10. V. Study Design and Methods A Mixed Methods (Qualitative and Quantitative) Research Study • Spring Statewide Online Survey of Acuity Schools - 2009-11 • About 4,000 respondents in total over three survey administrations • Examined educator opinions regarding Acuity Assessment Program content, technology/user experience, PD, and customer support after use of the system for a full school year or more • The primary objectives of the survey were to obtain suggestions for improvement of the program and to gauge views regarding the impact of the program on: 1) classroom instruction; 2) general student achievement; and, 3) student achievement on ISTEP+

11. Study Design and Methods • 2. Intensive Case Study in 12/15 schools - 2008-2010 • Determine what factors make a difference in the effective implementation of Acuity and use of Acuity data • Examined extent to which schools have implemented the Acuity Assessment Program and identify obstacles and challenges encountered • Examined extent to which Acuity has altered or informed classroom instruction, and impacted general student achievement as well as ISTEP+ performance • included one-on-one, face-to-face interviews with 34 principals, testing coordinators, and Acuity trainers as well as focus groups with 6-10 teachers in all 15 schools (109 teachers total); conducted 2/03/10 through 4/06/10

12. Study Design and Methods 3. Completion of a Quasi-Experimental Comparison-Group Study • Methodology • Control schools were matched as closely as possible to Acuity schools on the basis of their demographic variables and baseline ISTEP+ scores, within each grade level • Demographics included: school information (locale, size, expenditures, etc.); student information (SES, race, grade, gender, etc.); teacher credentials and experience; and baseline ISTEP+ scores • Treatment schools were grouped based on years using Acuity: 2008-2011 (three years), 2008-10 (two years), and 2009-10 (one year) • Matched groups separated by Math and English/language arts

13. Study Design and Methods Comparison Group Study, cont. • Treatment and Control Matches Per Cohort

14. Study Design and Methods Comparison-Group Study, Cont. • Methodology, cont. • A Linear Mixed Model (LMM) was performed on ISTEP+ scores for ELA and Math (separately) and by years in program, yielding six separate analyses. • Type III Tests for Fixed Effects were conducted to determine if statistically significant differences between Acuity schools/students and Control schools/students could be detected for several demographic factors and Acuity test types. • Estimates for comparative gain scores for those statistically significant interactions were then prepared from LMM and Fixed Effects regression results.

15. Study Design and Methods 4. Supplemental Focus Group Work – Spring 2011 and 2012 • 28 educators participated in focus groups on March 10-11, 2011, to further gauge opinion on Acuity Assessment Program and impact • 10 educators from high-improvement Acuity schools participated in focus groups on May 29 and June 5, 2012 (8 principals, 1 Acuity coordinator, and 1 curriculum coordinator), to examine “Under what conditions do benchmark assessments impact teaching and learning most?” • High-Improvement Acuity schools were the schools using Acuity all three school years of the program and their students demonstrated the highest scale-score gains on ISTEP+ over the same period of time.

16. VI. Findings of Comparison-Group Quasi-Experimental Study

17. Findings

18. Findings

19. Findings

20. Findings

21. Findings

22. Findings

23. Analysis • Effect sizes were very small across grade levels and subjects, suggesting effects may not have practical significance • In many instances the ELA and math means are higher in the Acuity group; however, the size of the difference is small as indicated by the negligible effect sizes and figures using ISTEP+ scale scores • Increased teacher usage of Acuity Assessment Program components (reports, Instructional Resources, etc.) does have a statistically significant positive effect on ISTEP+ scale score gains • Effects of benchmark assessment system is not definitive in either direction • Qualitative data indicates Acuity may have more effect in subsequent years. It takes time, but how much longer?

24. VII. Findings from Qualitative Analysis • Findings illustrate Acuity schools are progressing through the “Stages of Assessment Enlightenment:” 1) Concern: “test fatigue,” validity of results and utility of information 2) Confusion: interpretation and making meaning of the data – “How do we do it and what do we do with the information to provide instructional focus?” 3) Consciousness: evidence that reflective conversation and shared learning occurring; data-driven decisions; informing and altering instruction – at this stage we should begin to see larger impact on achievement

25. A. Spring Survey Results: Findings, Opinions, and Suggestions

26. What impact did Acuity have on classroom instruction? (3 years)

27. What impact did Acuity have on student achievement?

28. Anticipated impact of Acuity on ISTEP+ performance

29. Discussion of Third Research Objective • Under what conditions do benchmark assessments impact teaching and learning the most?

30. High-Improvement Acuity Schools:Focus Group Findings • Acuity has an impact on instruction • The accuracy of Acuity data has prompted teachers to increasingly utilize data to alter instruction – accuracy termed “eye-opening” • Acuity helps teachers determine where students are performing and underperforming – “helps us get on the front end with intervention efforts” • The data often informs teachers to establish groups by ability or mastery and identify needed remediation; administrators shifting resources and personnel to assist with efforts including remediation periods and extra time for reading/ELA; data could be more user friendly – esp. student reports • Re-teaching and remediation plans based on Acuity help to target students at all ability levels; intervention tier grouping • Schools are aligning curriculum to state curriculum maps and paying closer attention to teaching of Indiana Academic Standards to ensure instruction is standards driven

31. High-Improvement Acuity Schools:Focus Group Findings • One principal used analogy of the benefit of Acuity as “small races instead of one long race” in that it provides a clarity of focus on teaching and learning in shorter sequences; teachers can focus on improving 1 or 2 things and then move forward • Some of the high-improvement Acuity schools use instructional or data coaches to guide PD and PLC activities; student literacy coaches helpful, too

32. High-Improvement Acuity Schools:Focus Group Findings • Acuity has a perceived impact on student achievement • Acuity is perceived to be very influential in raising students’ ISTEP+ scores • Instructional Resources are popular, but there are many ISTEP+ sub-standards without sufficient correlated Instructional Resource items; make more interactive • Students taking ownership of Acuity results to improve their performance on ISTEP+; they like seeing the Student Growth Reports • Individual conferences with teachers regarding individual student performance lead to positive student attitudes towards Acuity • “…[Our students] get really excited about [the Student Growth Report] because they see a [progression] line, and they see it going up or down, so that’s been good for us and with the kids, knowing where they are, where they’re going” • “…what gets measured gets done, and if we measure something and we think it’s important as a staff, the students are going to take it seriously.”

33. Professional Development Critical for Success • Professional development is essential not only to inform on “assessment literacy” issues of relevance, validity, content, administration, and access to reports, the “technical usage” issues, but then to help with interpretation and use of data to inform and alter instruction, as well as ongoing PD on modifications or new features to the system. • High-improvement Acuity schools are setting aside or building time in school day for data analysis and discussion in professional learning communities • If present, these PD practices will lead to greater teacher buy-in and support of the system.

34. Suggestions Next Steps/Needs • More expansive Instructional Resources needed • Move Diagnostic 1 up earlier in the school calendar • More time is needed between tests and Predictive C needs to be pushed back • Auto reassignment feature in IR activities • Adequate computer access in schools still an issue and limits more regular use of IR activities – teachers want remediation tools that don’t require computer • Sharing of Acuity data with parents is not widespread and a parent-friendly report is desired • Utilize open-ended/constructed response questions and improving scoring process for this component • Schools would be interested in seeing state-level Acuity data • Some educators expressed the sentiment that the Acuity Assessment Program should replace ISTEP+ altogether

35. Concerns/Observations • Lack of use of open-ended constructed-response questions on Predictive Assessments and none included in Diagnostic Assessments; teachers must score using rubric – a disincentive because of time • Educators question, “Are Acuity Assessments as rigorous as the interim assessments that are being developed by SMARTER Balanced Assessment Consortium and PARCC?” • Will and how will results from Acuity be used to judge teacher quality under new teacher evaluation system required by Indiana? • Narrowing of curriculum and not educating “whole child”

36. Indiana Acuity Schools Evolution Reflective of Experiences Elsewhere • From study of SDP use of interim assessments, Blanc et al. (2010) writes that interim assessments “are most likely to contribute to improved student learning if there are also concomitant attention to developing strong school leaders who promote data-driven decision making within a school culture focused on strengthening instruction, professional learning, and collective responsibility for student success.”

37. CEEP Contact Information Terry E. Spradlin, MPA Director for Education Policy/Project Manager tspradli@indiana.edu Dingjing Shi & Rod Whiteman Graduate Research Assistants 1900 East Tenth Street Bloomington, Indiana 47406-7512 812-855-4438 Fax: 812-856-5890 http://ceep.indiana.edu Stephanie Dickinson & Lijiang Guo Senior Statistician/Statistician Indiana Statistical Consulting Center