rcet/twd/index.html

1. Preliminary Results from the Thinking With Data Project: A Cross-Curricular Approach to the Development of Data Literacy Mark van ‘t Hooft, Annette Kratcoski, Dale Cook, Kent State University RCET Karen Swan, University of Illinois, Springfield Phil Vahey, Ken Rafanan, Louise Yarnall, SRI International www.rcet.org/twd/index.html

2. Thinking with Data (TWD) Primary Goal • Develop middle school students’ deep understanding of data literacy across the curriculum. “We use data every day—to choose medications or health practices, to decide on a place to live, or to make judgments about education policy and practice. The newspapers and TV news are full of data about nutrition, side effects of popular drugs, and polls for current elections. Surely there is valuable information here, but how do you judge the reliability of what you read, see, or hear? This is no trivial skill—and we are not preparing students to make these critical and subtle distinctions.” -- Andee Rubin The research reported on in this paper was funded by the National Science Foundation under Grant NO. NSF ESI-0628122. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the National Science Foundation.

3. TWD Unit Context • Context: • Water situation in the Tigris/Euphrates & 8 US watersheds • U.N. convention states that international water courses should be used in “an equitable and reasonable manner”

4. TWD Modules & Materials • Four 2-week, integrated replacement modules • For implementation in 7th grade social studies, mathematics, science, and English Language arts classes • Core domain content (using real-world data) • SS: water sharing among Turkey, Syria, & Iraq • Math: proportional reasoning • Science: impact of technology on water availability and quality • ELA: persuasive arguments • Modules were implemented sequentially, with no other requirements for coordination • Assessments included: • An overall data literacy assessment (TWD and comparison) • Math and Science assessments (TWD only) • ELA final projects (TWD only)

5. TWD and Data Literacy • The modules address issues of data representation, proportional reasoning, and argumentation using real data in discipline-specific problem-solving contexts. Data literacy skills used include: • Formulating and answering data-based questions • Using appropriate data, tools, and representations • Solving real problems and communicate their solutions • Anchor learning in the idea of fairness: • Of comparisons, e.g. water distribution • Of measures, e.g. per capita distribution • Of arguments, e.g. accurate, relevant, complete

6. Preparation for Future Learning • Students are more likely to learn when they have recognized the existence of a problem before being presented with a solution. • In this framework students first prepare by investigating a set of problems that are designed to highlight the structure of an important concept. Instead of creating complete solutions, students come to understand the structure of the concept, and internalize key dimensions of the situation. • Students then engage in a formal learning activity in which they are introduced to a standard solution, and which they then practice and apply in a variety of contexts.

7. Research Questions • Are teachers able to effectively implement the cross-disciplinary TWD curriculum modules? • Do students who engage in the modules increase their understanding of cross-disciplinary data literacy at the end of the project versus the beginning of the project? • Do students who engage in the modules also increase their understanding of the required disciplinary content, particularly in Mathematics and Science? • Can we expect, based on teacher and principal input, that this program can be scalable to a wide number of schools?

8. Procedures and Data Sources • Procedures: • Pilot Test (2007-2008) with 42 7th grade students and 2 teachers • Module revisions made and materials posted online • Field Test (2008-2009): 2 middle schools; experimental (n=114) and control (n= 462) • Data Sources: • Data Literacy Pre/Post Assessment • Math and Science Subject Assessments • Classroom Observations • Teacher Interviews

9. Findings • Question: • Are teachers able to effectively implement the cross-disciplinary TWD curriculum modules? • Findings: • Yes, if implemented with the appropriate administrative support and professional development. • Most difficult module is social studies (fit with existing 7th grade curriculum and lack of closure)

10. Findings • Question: • Do students who engage in the modules increase their understanding of cross-disciplinary data literacy at the end of the project versus the beginning of the project? • Findings: • On average, TWD students (n = 114) had a gain score that was three points higher on a 15-point test than non-TWD students (n = 462): t(156.273) = 10.750, p < .001, d = 1.24 (very large effect). • TWD students at School 2 seemed to have learned more than those at School 1 with mean gain scores at 3.69 and 2.03, respectively. At School 2, the mean difference in gain scores between the TWD group and their 7th grade classmates was 3.135, as compared to a mean difference of 2.270 at School 1. • When considering questions individually, student scores improved the most on those items that required higher order thinking skills.

11. Findings • Question: • Do students who engage in the modules also increase their understanding of the required disciplinary content, particularly in Mathematics and Science? • Findings: • Pre/post testing in math and science showed statistically significant gains inlearning disciplinary content: • Due to inconsistencies in test administration, only two items could be scored on the math test for School 1; both items showed statistically significant gains (Z = 3.16 and Z = 4.70 respectively). Students in School 2 showed statistically significant gains across the entire math test, t(24) = 4.899, p < .001, d = .56 (medium effect). • In science, students in both schools showed statistically significant gains, t(84) = 12.665, p < .001, d = 1.36 (very large effect) for School 1; and t(27) = 4.441, p < .001, d = .83 (large effect) for School 2.

12. Findings • Question: • Can we expect, based on teacher and principal input, that this program can be scalable to a wide number of schools? • Findings: • Yes; with the exception of Social Studies, teachers felt that the TWD unit fit into the 7th grade curriculum. • There were concerns about the tensions created when modules are taught using PFL, especially at the end of the Social Studies module (lack of closure). • A small number of teachers were concerned that some of the materials were too advanced for their students, and these concerns are addressed in our final set of materials by including alternative options for lower-achieving students. • Finally, teachers expressed the importance of staff development, working as a team of teachers so each knew what the other three were doing in their modules, and alignment with content standards.

13. Conclusions • Preparing students for data literacy learning can occur in one curricular context with the learning activity occurring in another, strengthening plausibility that PFL uncovers a general mechanism of transfer. • Subsequent application and communication of learning may amplify learning that happens when using PFL, extending the framework. • Study provides a theoretically and empirically grounded basis for increasing the use of real-world data and developing students’ data literacy across the curriculum. • The TWD Projectprovides a scientific basis for conducting school-based data literacy activities that cut across disciplines; provides a set of assessment tools that can used to investigate students’ formative reasoning as they engage in cross-disciplinary data literacy; and provides a collection of materials and related data sets that can be used by others in the implementation of cross-disciplinary data literacy

14. Future Work • Focus on more closely analyzing our existing data to investigate the specific conditions that allow for our PFL-based approach to be effective, and on investigating how students and teachers make cross-disciplinary links when using the materials. • In short, we know that the TWD materials were effective; we need to investigate why and how they worked as well as they did. This work will especially focus on student work and reflections.

15. Related AERA Presentations Exploring the Efficacy of a Cross-Curricular Application of the Preparation for Future Learning Framework: PFL+ 44.066. Frameworks for Problem-Based Learning. SIG-Problem-Based Education; Paper Session Colorado Convention Center, Street Level, Room 606 Sunday, May 2, 12:25 pm to 1:55 pm Thinking with Data: A Cross-Disciplinary Approach to Teaching Data Literacy and Proportionality 45.068. Student Learning in Mathematics and Design of Learning Tasks. SIG Research in Mathematics Education; Paper Session Colorado Convention Center, Street Level, Room 708 Sunday, May 2, 2:15 pm to 3:45 pm

16. www.rcet.org/twd/index.html mvanthoo@kent.edu kswan4@uis.edu philip.vahey@sri.com