Summative Data, Formative Program Review

1. Summative Data, Formative Program Review Joseph L. Zawicki1, Kathleen Falconer2 and Dan MacIsaac3 1Department of Earth Sciences and Science Education, Buffalo State College, Buffalo, New York 14222. 2Department of Elementary Education and Reading, Buffalo State College. 3Department of Physics, Buffalo State College. Abstract Summative item response data should inform classroom practice. Data from the 2011 New York State Physical Setting: Physics was analyzed by key idea and performance indicators; selected items were reviewed form item difficulty and response pattern. Overall student understanding was assessed and items were used to inform the subsequent planning of units, including the use of selected items as formative prompts. This is part of a long term project to inform current physics teachers of the response data and analysis. Every spring since 2004, an analysis of the previous year's physics exam has been presented and discussed at the Western New York Physics Teachers Alliance (WNYPTA). Subsequent studies will explore the impact of the instructional instruments on student achievement. The Study Context Summative student data (test scores) have been used to provide evidence of student learning. Educational conventions suggest that students scoring an 85% or higher have essentially mastered the concepts and skills embedded in the curriculum. Students in the scoring in the 65-84% range are proficient, but have not attained mastery. Students scoring between 55-64% are close to reaching proficiency, and those earning scores below 55% have not successfully addressed the bulk of the particular curriculum. Recent legislation, such as the No Child Left Behind Act, mandate that districts provide remediation (through Academic Intervention Services), for students falling in the latter two categories. While teachers often focus their attention on students scoring at Level 1 ( < 55%) or Level 2 (55 – 64%), a thorough program review might suggest approaches to address student performance at all four of the demonstrated levels of competency. Data Collection Anonymized item responses from students in districts participating in the Data Warehouse supported by the Western New York Regional Information Center (WNYRIC), were complied for the June 2011 Physical Setting: Physics assessment (n=6,963). The substantial number of responses presents a compelling picture of conceptually challenging concepts. Summary data (item difficulties and response patterns) are presented in Tables 1 & 2. Table 1. Constructed Response Item Difficulties and Response Patterns Table 2. Multiple Choice Item Difficulties and Response Patterns Program review, conducted by teachers, utilizes a variety of resources, including, but are not limited to: prior discussions of assessment data, concepts recognized as challenging based upon discussions with colleagues and other educational professionals, and published formative assessment resources including the works of Bernhardt, Holcomb, Keeley and Liu, et. al., Love, Minstrell, and Popham. Data Comparison Inertia: One Example Inertia (Newton’s First Law) is often a challenging concept for students. The June 2003 assessment included the following item: Figure 1. June 2003, Physical Setting: Physics Exam, Item #15 15. Which person has the greatest inertia? (1) a 110-kg wrestler resting on a mat (2) a 90-kg man walking at 2 m/s (3) a 70-kg long-distance runner traveling at 5 m/s (4) a 50-kg girl sprinting at 10 m/s An analysis of 955 student papers supported the conclusion that 0.34 (34%) of the students were able to answer the question correctly. (324 selected the first response, 13 students selected the second response, 42 selected the third response, and 576 selected the fourth response.) (The 2003 exams were among the first assessments offered that targeted the new core curriculum.) This topic was addressed through yearly exam discussions at annual conference presentations at the Science Teacher’s Association of New York State, as well as during regular presentations with the Western New York Physics Teachers Alliance (WNYPTA). Activities and approaches to address the concept were developed. Data from the 2011 assessment, as shown in Table 2., indicates that fully 82% (0.82) of the student respondents, were able to successfully answer this question. This is particularly significant since the number of students participating in the 2011 assessment was just under 7,000 (6,961). While further analysis is pending, this initial analysis suggests that overall nature of student understanding has been changing over the past decade. Figure 2. June 2011 Physical Setting Physics Exam, Item #13. Future analyses will focus on the development of additional concepts over the course of the exam administration. New changes will be forthcoming with the development of Common Core Standards in Science; the Frameworks that will be used for developing the standards were just released this past summer (2011). ~~~~~~~~~~ References Cited Bernhardt, V. L. (1998). Data analysis for comprehensive school-wide improvement. Larchmont, NY: Eye Education. Diagnoser.com. (www.diagnoser.com/diagnoser/index.jsp) A NSF Project developed by J. Minstrell. Holcomb, E. L. (1999). Getting excited about data: How to combine people, passion and proof. Thousand Oaks, CA: Corwin Press. History of Regents Examinations, 1865 to 1987. Retrieved from http://www.emsc.nysed.gov/osa/ assesspubs/repubs.html. Last accessed on October 19, 2011. Love, N. (2002). Using data/getting results: A practical guide for school improvement in mathematics and science. Norwood, MA: Christopher-Gordon Publishers. Liu, X., Zawicki, J., & Arnold, J. (2009). Using data to reform science instruction. In Gess- Newsome, J., Luft, J., and Bell, R., eds. Reforming Secondary Science Instruction. Arlington, VA: NSTA Press. Popham, J. W. (1990). Modern educational measurement: A practitioner’s perspective. Englewood Cliffs, NJ: Prentice Hall. The No Child Left Behind Act of 2001 (Public Law 107-110).