110 likes | 200 Views
One State’s Implementation of Computer-based High-stakes Assessments in Public Education. Sarah Susbury, Director Office of Test Administration, Scoring & Reporting Virginia Department of Education. Conceptual Framework and Methods.
E N D
One State’s Implementation of Computer-based High-stakes Assessments in Public Education Sarah Susbury, Director Office of Test Administration, Scoring & Reporting Virginia Department of Education
Conceptual Framework and Methods • Participant-observer case study of state perspective (implementation, assessment-related, technology-related, organizational, financial, political) • Interviews with state-level individuals associated with the Web-based Standards of Learning Technology Initiative • Review of all existing documentation (legislative documents, procurement documents, project plans, status reports, risk mitigation reports, meeting minutes, memos, emails, etc.) • Constant comparative methodology with procedural additions (Glaser, B. G., & Strauss, A. L. (1967), Lincoln, Y. S., & Guba, E. G. (1985)Maykut, P., & Morehouse, R. (1994))
Findings • Technology Initiative with an assessment focus • Proof of Concept or Demonstration Phase • Existing high-stakes assessment program transitioned to computer-based program • Leadership and multi-level organizational change • Phased Implementation with change management • Implementation models for paper/pencil versus computer-based high-stakes testing
Conclusions • Pairing technology and assessment • Supporting technology readiness and assessment readiness • Phased implementation overall and at different school levels • Change management • Leadership • Training and communication
One successful state’s perspective on factors that affect the extent of adoption of technology-enabled assessments for K12 statewide accountability • There is a global need for assessments to measure skills that cannot be measured with paper and a pencil but the practical challenges of school capacity make the ideal model of simultaneous testing for all students in a statewide computer-based accountability test almost impossible. • There are factors associated with these challenges that could be addressed by experts once they are known and understood; this study aims to define these factors. Diana W. Cano Rutgers, The State University of New JerseyGraduate School of New BrunswickNew Brunswick, New Jersey
Conceptual Framework and Methods • What does adoption mean in the context of testing and technology? • General adoption. The innovation adoption cycle is most often used by technologists to understand how organizations go about making changes that are innovative. • Adoption by middle level of a federal system. Three categories of factors interact with each other in the management of our policies on education assessments: district and school capacity; state policy; national policy. • What are the state factors that influence adoption? • There are four broad categories: standards and accountability; school capacity; teacher quality; network strength. • Research has shown that the setting of standards leads to better student performance on accountability assessments. • There are negative effects of high accountability on a school’s capacity to improve its academic performance. • Teachers play a critical role in the successful adoption of innovation. • A school’s adoption of technology may also indicate a willingness to adopt innovative educational solutions. • Using social network analysis, studies have been done to describe the characteristics of a policy network that enable or inhibit innovation adoption. • Methods • Sampling. One state was purposely chosen because of their extensive experience with computer-based testing. • Recruitment. Interviewees were the state’s director of testing and two team members. • Interviews. These interviews lasted approximately 40-60 minutes and were digitally audio-recorded.
Findings • How do communications and autonomy factors constrain or facilitate adoption? • communications • “If they don't do well, then we're not going to do well” • “we'd set them up, and we'd be right there.” • “If there were seats and they wanted to come, we let them come. And we figured the more information we could get out there, the better.” • “what I was hearing from someone in the field and I could share that with someone else. So that, that flow of information was important.” • school migration path • “beef up your network” and “grown into the testing” and “phased it in” and “building a capacity to do it” • “they could choose the subject. It didn't matter to us.” • How do the factors of national policies, state policies and school capacity constrain or facilitate adoption? • state policy, school capacity and state organizational structure and dynamics factors influence the adoption of TEAs • state policy and school capacity are dependent upon each other • communication between these stakeholders affects adoption
Conclusions • Key factors affecting adoption of TEAs for statewide accountability • State and district communications • Schools defining their own migration schedule • State policy and school capacity dependencies • State policy, organizational structure and school capacity • States use the Internet to gather technical information • The data illustrate that the factors most affecting innovation adoption are rooted in state-level bureaucrats respecting the districts through substantive communications and autonomy over how they can accomplish the migration from paper-based tests to TEAs. • This conclusion is especially important as states rely on two national consortia to set timelines for implementation, regardless of district readiness and school capacity. • PARCC states will be expected to be online in the 2014-2015 school year • The state has spent years successfully migrating all schools to TEAs through state leadership and the acknowledgement that this capability needs to be built locally at a reasonable rate and funded accordingly. • They are 100% online this year (exceptions only for accommodations) • They have a strong network of district leaders who manage year-round testing • Recommendations for Future Research • Further research on factors influencing adoption • Further study within the state from other perspectives in the districts • Additional research between states as a cross-state comparison I am recruiting states… please let me know if you are interested!
Showcase: From Research to Reality: What States Need to Do to be Successful with K12 Adoption of Computer-Based Accountability Assessments Jennifer Dugan Minnesota Department of Education “Leading for educational excellence and equity. Every day for every one.”
Online Participation There are ~63K students/grade. education.state.mn.us
Next Steps education.state.mn.us • Scheduling is main concern impacting current participation in online testing • Pursue delivering tests on mobile devices with smaller screens; districts changed their approach from their original advice • What is best path for items that currently utilize all of the 11.6 inch screen real estate in moving to 9.7 inch screen? • Planning for increased need for on-site technical assistance