Damian Bebell , Ph.D. Assistant Research Professor

1. Strapping jet engines to the stage coach: Using technology topush the boundaries of educational measurement Damian Bebell, Ph.D. Assistant Research Professor Center for the Study of Testing, Evaluation and Educational Policy Lynch School of Education Boston College Data Mining for Educational Assessment and Feedback ASSESS 2014 KDD 2014

2. Background • Defining success in schools • Purpose of School • Examples from research • Measurement matters • DVs • IVs • Scaling • Leveraging digital technology • Data visualization • Feedback loops • Big data • Learner analytics • APIP • Redefining success

3. “For want of records, much useful knowledge is continually lost. Though many individuals have derived advantages to themselves from experiments, but few have recorded them. Even those who make experiments [are] liable to forget them, so as to give incorrect representations of them when they try to relate them.” Lathrop, Leonard E. (1826). Remarks On the General Condition of Agriculture: The Farmer’s Library, Second Edition. Windsor, VT: Wyman Spooner.

4. Background and perspective • Limited knowledge of computer science, pedagogy, or instructional technology • Background in psychometrics, measurement, and evaluation • Leveraging technology to advance the state of the art in educational research and measurement…

5. What is the technology that drives so much of what we do in educational research and measurement today?

6. IBM 805 Test Scoring Machine 1937

9. “the assessment of students and schools in the US began as early as 1969, with the first National Assessment of Educational Progress (NAEP) test. The test has developed over the years, but it is still based on the same methodology that was initially enabled by the IBM 805: multiple choice tests with automated scoring.” http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/testscore/transform/

10. The IBM Optical Scanner allowed the bubble sheet and the rise of the 5 response question. • To this day, the artifacts of this 1930’s computer technology restrict imagination and practice in how we design surveys and tests. How can we use new and emerging technologies to advance the state of the art In our own research and evaluation work? Bebell, D., O’Dwyer, L. M., Russell, M., & Hoffman, T. (2010). Concerns, considerations and new ideas for data collection and research in educational technology studies. Journal of Research on Technology in Education, 43(x).

11. Leveraging Research and data Formative Assessment Summative Assessment The information you collect about your program/initiative can serve to inform the day-to-day operation of your program (formative information) which over time can provide evidence of success/goals met (summative information).

12. Why do schools bother collecting data or conducting research? • Organize school stakeholders/constituents around project goals • Determine if educational investments are “paying off” • Better understand how teaching and learning is evolving in educational system (and where) • Measure progress toward specific project and strategic goals • Support action planning with data • Provide greater community evidence of “success” • Provide the greater educational community evidence and examples of what works (and what doesn’t) • Develop a culture and community of data-informed practices

13. What is the purpose of school?

14. Categorize and quantify the wide spectrum of reasons we have schooling in the US. Use content analyses to focus is on the perspective of schools themselves. Mission statements are a common “data” point for most schools. • Constitutional perspective • Legal perspective • Business leaders perspective • Parent perspective • What about school’s perspective? www.purposeofschool.org

15. Mission Statement/School purpose and outcome identifier: Cognitive/Academic Development Social Development Emotional Development Civic Development Physical Development Vocational Development Integrate Into Local Community Integrate Into Global Community Integrate Into Spiritual Community Safe/Nurturing Environment Challenging Environment

16. Mission Statement/School purpose and outcome identifier:

19. It is not adequate to simply focus exclusively on low hanging academic and cognitive outcomes, if the overall mission or program has a broader reach. Severe limitations of letting methodological limitations define success www.purposeofschool.org

20. Time to Know (NYC) Study PS-- Brownsville, Brooklyn • 724 students • 98% non-White • Highest concentration of public housing in USA Pre/Post Comparison Design Study Outcomes: • Evolution of teaching practices • New pedagogical opportunities • Student engagement • Student retention and attendance • Student achievement • Digital Citizenship 1:1 laptop and Digital teaching platform across 4th and 5th grade

21. T2K Study: Increased Student Attendance Compared to past averages and those students in the non-1:1 Brooklyn setting, 1:1 students exhibited a statistically significant difference of nearly six school days in the number of missed school days during the 2010/2011 school year.

22. T2K Study Outcomes Year 1: • Increase in variety of instructional strategies/resources employed by teachers • 79% of students reported more engagement in class. • Skype partnership with middle-eastern schools

23. T2K Study Outcomes Year 2: • NYC DOE’s official measures of success is based on students’ ELA and Math test scores • Teacher names were published with average student score gains in NY Times • High Stakes results for school leadership and teachers

24. Who defines your success? Project Ceibal-Uruguay • 1st country wide 1:1 program (2007) • 350,000 students, 16,000 teachers • 1,000,000th student devices was distributed in Fall 2013 • X0-1 device, wireless Internet connection http://www.youtube.com/watch?v=SnLo-j8GvPM • How do measure the success of Ceibal? • Who is impacted by this program? • How do we know it is working?

25. “Plan Ceibal…does not see itself as an 'education project' per se, but rather as a project to help transform larger society, with the education system as just the initial vector through which the project hopes to infect all of Uruguay with a new level of 'connectedness'.” -Michael Trucano World Bank

26. Who defines your success?

27. Who defines your success? Since Ceibal: “Encompassing the video game companies, software development in Uruguay has evolved into a $600 million industry, making the country Latin America’s leader in per-capita software exports.” Feb. 22, 2013 http://www.nytimes.com/2013/02/22/world/americas/uruguays-video-game-start-ups-garner-attention.html?pagewanted=all&_r=0

28. Who defines your success? Project Ceibal-Uruguay • 1st country wide 1:1 program (2007-2009) • 350,000 students, 16,000 teachers • X0-1 device, wireless Internet connection http://www.youtube.com/watch?v=SnLo-j8GvPM

29. How we define and measure something mattersSmall decisions can have big impactsMeasuring the frequency of teachers technology use example**2-3 min. small group breakout**

30. Quick discussion activity • Why do some teachers use given resources while other teachers do not? • How do we illuminate the relationship between teacher practices and educational impacts? (impact of IV on DV)? • How do we measure teachers practices to illuminate why some teachers adopt and use digital technology and others do not? • Oh, and we have 2500 teachers across 20 schools in 4 continents (they have internet connection- and we have little operating budget) • HOW DO WE OPERATIONALLY DEFINE AND MEASURE TEACHER PRACTICES?

34. 7 Scales of Teacher Technology Use

35. Examining Teacher Survey Data: Why multiple measures are more revealing (1)

36. Examining Teacher Survey Data: Why multiple measures are more revealing (2)

37. Traditional Survey Question(8 response choices):

38. Sequential Assigned Values

39. Scaling the Assigned Values

40. Slider with Full Ratio Scale

43. Accessible Portable Item Protocol Standard (APIP) IMS Global Learning Consortium, Measured Progress, US State Consortia developed Data model for standardizing the interchange file format for digital test items 1) APIP allows digital tests and items to be ported across APIP compliant test item banks. 2) It provides a test delivery interface with all the information and resources required to make a test and an Item accessible for students with a variety of disabilities and special needs. Interoperability across assessment items across students

44. Accessible Portable Item Protocol (APIP) Data model for standardizing the interchange file format for digital test items Interoperability across assessment items across students Major step forward in utilizing technology to advance assessment What are we really trying to measure? Who own the assessments?

45. Thinking ahead, questioning assumptions, and staying in front of the curve “All of us… must consciously break the habits we bring to thinking about the computer. Computation is in its infancy. It is hard to think about computers of the future without projecting onto them the properties and the limitations of those we think we know today. And nowhere is this more true than in imagining how computers can enter the world of education.” —Seymour Papert, Mindstorms, 1980

46. www.asbunplugged.org/collaborativewww.intasc.orgwww.purposeofschool.orgwww.bc.edu/T2Kwww.bc.edu/nps21www.edx.orghttp://www.measuredprogress.org/apip-standardwww.asbunplugged.org/collaborativewww.intasc.orgwww.purposeofschool.orgwww.bc.edu/T2Kwww.bc.edu/nps21www.edx.orghttp://www.measuredprogress.org/apip-standard

47. Questions/reflections bebell@bc.edu Thank you!