Delivering on the Promise of Data in Education

1. Delivering on the Promise of Data in Education Winter Forum February 2013 Jack Buckley Commissioner National Center for Education Statistics

2. A Really Simple Theory of the Tradeoff Between Quality and Accountability Data Quality What we wish were true How it is Accountability Pressure

3. The “Chowky Dar” Region: Increased Accountability Pressure Improves Quality Data Quality Accountability Pressure

4. Chowky Dar? • "The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the chowkydar(village watchman in India), who just puts down what he damn pleases.” –Josiah Stamp • No accountability = no incentive for accurate reporting • Some accountability (even “soft”) can yield improvements in quality through reduction of “noise” • Example: Last year’s U.S. News and World Report high school rankings and the quality of school-level CCD data

5. The High-Stakes Region: Too Much Pressure Can Distort the Data Data Quality Accountability Pressure

6. High-Stakes • Here the threat is not usually noise—it’s intentional misreporting. • As the stakes get higher, a percentage of individuals involved in the system will subvert it. • Example: the intentional cheating by some Atlanta Public Schools teachers and administrators on the Georgia state assessment. • Some ways to reduce accountability pressure are sampling (NAEP versus state tests) or confidentiality through aggregate reporting, but not an option for every application.

7. Can We Get to the Middle? Only if We Control How the Data Are Used. Good Luck with that. Data Quality Accountability Pressure

8. Q: If Policy Makers Set the level of Accountability Pressure, what Can We Do? A: Find Ways to Shift the Whole Curve Up Data Quality Accountability Pressure

9. So How Can We Shift the Curve? • Cooperation, training, technical assistance, sharing best practices. In other words, the Forum and related efforts. • Improvements in collection technology (broadly defined) can help—SLDSs, better edit checks, smarter tools, automation, integrated systems, methods for detecting distortion, common data standards. • This is the basic strategy behind NCES’s activities in administrative data: Build a strong community, invest in better systems, develop common standards, and improve our tools and technology. • In short, if we can shift the curve, data quality can theoretically be improved at all levels of accountability.

10. How NCES is Shifting the Curve Today and Tomorrow • CEDS V 3.0 end of January 2013. On time and almost under budget • Continued development work on CEDS Align and Connect Tools • Assisting OCR with redesign of the CRDC collection tool • Working with EDFacts and state partners to integrate from CEDS to data groups • Maintaining and improving our Forum, MIS/Summer Data, SLDS conferences to work with the field • Working with our partners to reexamine and improve the back end of collection systems like CCD, CRDC, EDFacts, and IPEDS.

11. But Data Quality Is not the Only Challenge to Usefulness • Some uses of the data are obvious and benefit obviously from quality improvement—simple descriptive statistics, monitoring and early warning systems, feedback reports. • But other uses—prediction, causal inference, evaluation—can only be improved so much through shifting the curve. Even with perfectly measured data, there are many threats to valid inference. • Large scale IT systems, shiny technology, “Big Data,” aren’t substitutes for principles of careful scientific research design. There are no easy answers to difficult questions. • The surest way to end the new era of data in education is through overpromising and large-scale failure. “Those who ignore Statistics are condemned to reinvent it.” –Brad Efron.