1 / 14

Thanos Patelis Center for Assessment Presentation at the

Beyond Peer Review: Developing and Validating 21st-Century Assessment Systems Is it time for an audit?. Thanos Patelis Center for Assessment Presentation at the National Conference on Student Assessment New Orleans, June 25, 2014. Overview. What is being reviewed?

samuru
Download Presentation

Thanos Patelis Center for Assessment Presentation at the

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Beyond Peer Review: Developing and Validating 21st-Century Assessment SystemsIs it time for an audit? Thanos Patelis Center for Assessment Presentation at the National Conference on Student Assessment New Orleans, June 25, 2014

  2. Overview • What is being reviewed? • Assessment Quality Audits • What are the criteria & how? • Requirements: • Theory of Action & Evidence • Sound methodology implemented by experts with transparency and external review • Recap Patelis – NCSA - 06/25/2014

  3. The need today for a mechanism to assure the quality of tests and their valid use flows, therefore, out of the tension between the critical consequences of present high-stakes tests for test takers and institutions and the public’s need for the protection testing affords. George Madaus, 1992, p.26 Patelis – NCSA - 06/25/2014

  4. The need for some sort of social intervention to monitor, audit, or review high-stakes testing programs is critical because the demands of test validation often clash with the commercial nature of applied testing. George Madaus, 1992, p. 27 Patelis – NCSA - 06/25/2014

  5. What is being reviewed? • The reviews are of the assessment system. • While assessment systems are components of other systems, these other systems will include other criteria and evidence. • Accountability systems • Educational evaluation systems • Because they are assessment systems, the review must go beyond the technical aspects of the instrument. • Context • Purpose and use • Score reporting • Data systems • Operational features that interact with the assessment Patelis – NCSA - 06/25/2014

  6. Assessment Quality Audits Peer Review: “To determine whether States have met NCLB standards and assessments requirements, the U.S. Department of Education (ED) will use a peer review process involving experts in the fields of standards and assessments. The review will evaluate States' assessment systems only against NCLB requirements.” USDOE, 2007, p. 7 Patelis – NCSA - 06/25/2014

  7. Assessment Quality Audits The idea of having experts in the fields of standards and assessments is sound. USDOE, by only looking at compliance with the law, is missing the opportunity to implement a full review of the quality of assessments. While the USDOE should require quality, it is not in the best position to do it. Similarly, federal granting agencies require independent evaluators to evaluate the initiatives that they fund. But, they do not do the evaluations! Patelis – NCSA - 06/25/2014

  8. Criteria and Methodology CCSSO published a set of assessment criteria “for high-quality state summative assessments aligned to college- and career-readiness standards” (CCSSO, 2013, p. 1) Patelis – NCSA - 06/25/2014

  9. Overview of Assessment Criteria • The criteria bring together and utilize • Standards of Educational and Psychological Testing (1999), • the Operational Best Practices for Statewide Large-Scale Assessment Programs (2010), • and a variety of recommendations of quality that implicitly allude to quality of data as seen from the Data Quality Campaign and principles of score reporting (Zenisky & Hambleton, 2012). • Examples of undertaking full reviews or audits of assessments programs can be found and are currently done. • Buros Center, ETS. • The operationalization of quality criteria is not trivial. • Examining a toolkit provided by Achieve partnering with CCSSSO and Student Achievement Partners recently for evaluating alignment of instructional and assessment material to the Common Core State Standards illustrates the hard work and expertise that is needed to implement the reviews effectively, fairly and truthfully. Patelis – NCSA - 06/25/2014

  10. Overview of Assessment Criteria • Meet overall assessment goals and ensure technical quality (7) • Align to standards – English language arts/literacy (9) • Align to standards – Mathematics (5) • Yield valuable reports on student progress and performance (2) • Adhere to best practices in test administration (1) • State specific criteria (varies) • Samples: • Linkage to higher education • Articulated assessments • Interoperability Patelis – NCSA - 06/25/2014

  11. Requirements • Assessment Program: • Theory of action • Evidence • Reviewers: • Criteria • Operationalization of the criteria • Description • Evidence needed • Exemplars • Rubric • Rating collection forms • Transparency of methodology including multiple external reviews • Review panels • Publication and dissemination Patelis – NCSA - 06/25/2014

  12. Recap • As new, important assessments are being launched, there is a heightened awareness for evaluating and ensuring quality of these assessments and all their components. • The USDOE is not in a position to do this, but should require that assessments show evidence of quality and be evaluated by an independent party. • Substantial effort is needed to operationalize the criteria that have been offered by CCSSO. • These criteria go beyond the technical quality of the instrument. • The methodology developed must realize the nature of the criterion. • The same old methodology will not work. Patelis – NCSA - 06/25/2014

  13. References AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington, DC: AERA. Council of Chief State School Officers & Association of Test Publishers (2010). Operational best practices for statewide large-sale assessment programs. Washington, DC: Council of Chief State School Officers. [ISBN: 978-1453716281] Council of Chief State School Officers (2013). Criteria for procuring and evaluating high-quality assessments. Washington, DC: Author. http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdf Madaus, G. F. (1992). An independent auditing mechanism for testing. Educational Issues: Measurement and Practice, 11(1), 26-31. DOI: 10.1111/j.1745-3992.1992.tb00225.x Student Achievement Partners, CCSSO, Achieve (March, 2014). Toolkit for evaluating alignment of instructional and assessment materials to the Common Core State Standards. Washington, DC: Achieve. http://www.achieve.org/files/CompleteToolkit_March2014_highRes.pdf U.S. Department of Education (2009). Standards and assessment peer review guidance: Information and examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: Author. Zenisky, A. L. & Hambleton, R. K. (2012). Developing test score reports that work: The process and best practices for effective communication. Educational Measurement: Issues and Practice, 31(2), 21-26. Patelis – NCSA - 06/25/2014

  14. Questions and Comments? Center for Assessment www.nciea.org ThanosPatelis tpatelis@nciea.org Patelis – NCSA - 06/25/2014

More Related