Defining Data Literacy for Educators

1. STATS-DC 2012 Data Conference July 12, 2012 Defining Data Literacy for Educators Ellen Mandinach, WestEd Edith Gummer, Education Northwest and NSF A Collaborative Project of WestEd and Education Northwest Sponsored by the Bill & Melinda Gates Foundation

2. Problem Space • No common definition of data literacy • Professional development providers • Teacher/Administrator preparation • Confusion between data literacy and assessment literacy • Lack of understanding of what it means to be data literate and use data effectively

3. The Project • Bring together a diverse group of expert stakeholders who know data-driven decisionmaking • Scope out a common definition of data literacy • Discriminate between data literacy and assessment literacy, understanding the requisite skills and knowledge • Perform a content analysis of major professional development books and models. Identify the key knowledge and skills. • Identify commonalities, differences, and perform a gap analysis.

4. The Stakeholder Groups • Experts from: • Research • Professional development • Government • Policy • Funding agencies • Assessment • Other stakeholders

5. Data Sources • Pre-conference data definitions from experts • Professional development text analyses • A hands-on activity to distinguish between data literacy and assessment literacy • Transcripts from the conference

6. Pre-Conference Definitions • Really interesting and diverse • Some succinct; others quite involved • Some quite philosophical • Focus on data; focus on assessment • Plethora of skills and knowledge components

7. A Wordle for Data Literacy

8. Some Overarching Thoughts and Select Quotes from Pre-Conference • Differentiation between declarative, procedural, schematic and strategic knowledge; the content versus the performance/process components • Data literacy is more than the sum of the component skills and knowledge • Continuum of novice to expert • “Defining data literacy will require careful consideration of the level of ability necessary to define literacy – somewhere between basic skills to complete fluency.”

9. Conflicting Perspectives(in one quote) “At the state department and college/university levels we are doing a damn lousy job of teaching data literacy.  Too few states demand courses on testing and measurement.  Too few colleges have those courses taught (when taught at all) by people who could plausibly be described as psychometrically competent.  We spend too much time on the statistics of data and too little on the ‘how to make constructive use of data’ and the ethics of data gathering and use.”

10. A Hands–On Activity • Purpose – to differentiate between data and assessment literacy • Outcomes • Data literacy is seen as broader than assessment literacy – but contested • Assessment literacy is a component of data literacy – but has unique components • The question remains what are the unique components – open quandary

11. Analysis of the Materials • Purpose – to characterize the terrain of professional development in data literacy • What knowledge and skills are suggested by the materials as being the core of what educators need to know and be able to do? • Representative rather than exhaustive search • Provide an initial operational framework

12. Analysis of the Materials • Process for analysis • Had to start somewhere – 2 texts we knew were being used extensively in professional development • Love, Stiles, Mundry, & DiRanna (2008) The data coach's guide to improving learning for all students. • White (2011) Show me the proof: Tools and strategies to make data work with the Common Core Standards. • Read through the texts identifying the key concepts and skills being presented on each page – tons of post-it notes

13. Analysis of the Materials • Grain size decisions – 30,000 feet versus weeds • Examined commonality in language across the 2 texts • Distilled 4 pages of codes to 8 key terms – negotiated process • Sorted remaining codes into 8 key terms

14. Analysis of the Materials • Coding process • 2 research associates used the coding spreadsheets – negotiated practice • Code values • Mentioned – page # and m • Whole page addressed code – page #-1 • More than one page addressed code page # + • Multiple pages addressed code – page #-page # • Relative emphasis of the text material

15. Findings – Common Elements Collaborative processes Multiple forms of data – triangulation Key focus on student achievement Little emphasis on statistical knowledge General analysis processes described with examples Templates for examining and displaying data and making interpretations Connection to instruction overly general

16. Findings – Variations • Level of description and instruction – a number of texts provide broad general description and few details • Data use versus formative assessment • Focus on multiple forms of data beyond achievement and student performances • Educator versus teacher

17. Analysis of the Conference Transcripts • Overarching theme: It’s complex • Data literacy is more than just its definition • There is an entire educational contextual landscape that surrounds the definition • The issue is systemic

18. Working Definitions for Data Literacy • Data Literacy is……… • Complexity • 95/5 issue – importance of the 5% • Consensus definition

19. Complexity of Defining Data Literacy • Literacy – generic to specific • Messaging • Language • Role or stakeholder influence/perspective • Systems nature • Technical knowledge and skills • Data • Pedagogical/administrative

20. Knowledge and Skills95%/5% • Problem Focus • Identify purpose • Frame questions • Understand context • Apply data to instructional or administrative action • Data Focus • Select the right data • Know how to access data • Design instruments • Collect data • Organize data • Use multiple sources of data • Understand data properties • Understand different kinds of data (formative, summative, diagnostic) • Use a variety of data sources (qualitative, quantitative)

21. Knowledge and Skills95%/5% • Data Focus - continued • Use of statistics and measurement concepts - understand reliability, validity, sources of error • Understand scaled scores, percentiles, performance levels • Drill down to different levels of data • Summarize data • Synthesize information • Analyze data • Analyze different levels of data – cohort, course, grade • Examine patterns and trends • Make interpretations • Troubleshoot data

22. Knowledge and Skills95%/5% • Process Focus • Formulate hypotheses • Engage in collaborative inquiry • Transform data into actionable knowledge • Know how to use data systems, tools, and applications • Comprehend data displays and data reporting • Draw inferences • Consider consequences • Test assumptions • Evaluate outcomes • Critique arguments • Habits of Mind - Dispositions

23. Educational Context • The need for a conceptual model • Account for organizational factors • Issues around data properties and adequacy for use • Issues around professional development • The need for best practices and models • The role-based nature of data literacy • Roles of schools of education, state agencies, and policymakers • The SLDS • Other technologies and supporting resources • The role of research • The need for instrumentation • The role of funding organizations • The need for collaboration • Messaging

24. A Road Map for Next Steps • For the researchers • For the professional development providers • For the policymakers • For the funders