1 / 31

Update on the Revisions to the Standards for Educational and Psychological Testing

Update on the Revisions to the Standards for Educational and Psychological Testing. Antonio E. Puente, Ph.D. University of North Carolina Wilmington Joint Committee for the Revision to the Standards for Educational and Psychological Testing www.apa.org/science/programs/testing/standards.aspx.

jessamine-castro
Download Presentation

Update on the Revisions to the Standards for Educational and Psychological Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Update on the Revisions to the Standards for Educational and Psychological Testing Antonio E. Puente, Ph.D. University of North Carolina Wilmington Joint Committee for the Revision to the Standards for Educational and Psychological Testing www.apa.org/science/programs/testing/standards.aspx

  2. Mission & Brief History • “The purpose for publishing the standards is to provide criteria for the evaluation tests, testing practices, and the effects of tests use”. (1999) • Established in 1954; revised in 1955, 1966, 1999 and currently (2009-2011) • American Psychological Association, American Educational Research Association, National Council on Measurement in Education Update on Revisions to the Test Standards

  3. Update on the Revisions to the Standards for Educational and Psychological Testing: Overview of the Revision Process

  4. Scope of the Revision Based on comments each organization received from invitation tocomment Summarized by the Management Committee in consultation with the Co-Chairs Wayne Camara, Chair, APA Suzanne Lane, AERA David Frisbie, NCME Update on Revisions to the Test Standards 4

  5. Joint Committee Members Lauress Wise, Co Chair, HumRRO Barbara Plake, Co Chair, University of Nebraska Linda Cook, ETS Fritz Drasgow, University of Illinois Brian Gong, NCIEA Laura Hamilton, Rand Corporation Jo-Ida Hansen, University on MN Joan Herman, UCLA Update on Revisions to the Test Standards 5

  6. Joint Committee Members Michael Kane, ETS Michael Kolen, University of Iowa Antonio E. Puente, UNC-Wilmington Paul Sackett, University of MN Nancy Tippins, Valtera Corporation Walter (Denny) Way, Pearson Frank Worrell, Univ of CA- Berkeley Update on Revisions to the Test Standards 6

  7. Five Identified Areas for the Revisions Access/Fairness Accountability Technology Workplace Format issues Update on Revisions to the Test Standards 7

  8. Timeline First meeting January, 2009 Three year process for completing text of revision Release of draft revision following December 2010 JC meeting Open comment/Organization reviews Projected publication Summer, 2012 Update on Revisions to the Test Standards 8

  9. Anticipated Review Process • Draft to be released December, 2010 • 90 Day (+/-) Review Period • Public Portal for Submitting Comments • Sponsoring Organizations with Publicize Opportunity for Comments • Committee will Summarize Key Issues from Reviews and Provide Response(s) Update on Revisions to the Test Standards

  10. Publication Options Management Committee responsibility Goal is for electronic access Pursuing options for Kindle, etc. Concerns about retaining integrity and financial support for future revision efforts May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 10

  11. Update on the Revisions to the Standards for Educational and Psychological Testing: Fairness for Diverse Populations

  12. 1999 Approach Standards related to fairness appear throughout many chapters Concentrated attention in: Chapter 7: Fairness in Testing and Test Use Chapter 8: Rights and Responsibilities of Test Takers Chapter 9: Testing Individuals of Diverse Linguistic Backgrounds Chapter 10: Testing Individuals with Disabilities Update on Revisions to the Test Standards 12

  13. Committee Charge Five elements of the charge focused on accommodations/modifications Impact/differentiation of accommodation and modification Appropriate selection/use for English learners (ELs) and examinees with disabilities (EwD) Attention to other groups, e.g., pre-K, older populations Flagging Comparability/validity of accommodated scores One element focused on adequacy and comparability of translations One element focused on Universal Design Update on Revisions to the Test Standards 13

  14. Revision Response Fairness is fundamental to test validity: include as foundation chapter Fairness and access are inseparable Same principles of fairness and access apply to all individuals and regardless of specific subgroup From three chapters to a single chapter that describes core principles and standards Examples drawn from ELs, EwD, and other groups (young children, aging adults adults, etc) Comments point to applications for specific groups More particular standards retained where appropriate (e.g., test translations) Update on Revisions to the Test Standards 14

  15. Overview to Fairness Chapter Section I: General Views of Fairness Section II: Threats to the Fair and Valid Interpretations of Test Scores Section III: Minimizing Construct Irrelevant Components Through the Use of Test Design and Testing Adaptations Section IV: The Standards Update on Revisions to the Test Standards 15

  16. Four Clusters of Standards Use test design, development administration and scoring procedures that minimize barriers to valid test interpretations for all individuals. Conduct studies to examine the validity of test score inferences for the intended examinee population. Provide appropriate accommodations to remove barriers to the accessibility of the construct measured by the assessment and to the valid interpretation of the assessment scores. Guard against inappropriate interpretations, use, and/or unintended consequences of test results for individuals or subgroups. Update on Revisions to the Test Standards

  17. Update on the Revisions to the Standards for Educational and Psychological Testing: Other Issues

  18. Other Issues in Committee Charge • Increased use of tests for accountability • Impact of technology on testing • Format and style issues Update on Revisions to the Test Standards

  19. Increased Use of Testing for Accountability Use of tests for accountability has expanded Most notably in education but also in other areas such as behavioral health Facilitated by increasing availability of data and analysis tools Recent and impending federal and state initiatives will likely lead to further expansion Use of test scores in policy and program evaluations continues to be widespread Reinforced by groups that fund and evaluate research (e.g., IES, What Works Clearinghouse) May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 19 19

  20. Organization of Accountability Material Chapter on policy uses of tests focuses on use of aggregate scores for accountability and policy Chapter on educational testing addresses student-level accountability (e.g., promotional gates, high school exit exams) and interim assessment Validity, reliability, and fairness standards in earlier chapters apply to accountability testing as well May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 20

  21. Impact of Technology Technological advances are changing the way tests are delivered, scored, interpreted and in some cases, the nature of the tests themselves The Joint Committee has been charged with considering how technological advances should impact revisions to the Standards As with the other themes, comments on the standards that related to technology were compiled by the Management Committee and summarized in their charge to the Joint Committee Update on Revisions to the Test Standards 21

  22. Key Technology Issues Included in our Charge Reliability & validity of innovative item formats Validity issues associated with the use of: Automated scoring algorithms Automated score reports and interpretations Security issues for tests delivered over the internet Issues with web-accessible data, including data warehousing Update on Revisions to the Test Standards 22

  23. Reliability & Validity of Innovative Item Formats What special issues exist for innovative items with respect to access and elimination of bias against particular groups? How might the standards reflect these issues? What steps should the standards suggest with regards to “usability” of innovative items? What issues will emerge over the next five years related to innovative items/test formats that need to be addressed by the standards? Update on Revisions to the Test Standards 23

  24. Automated Scoring Algorithms What level of documentation/disclosure is appropriate and tolerable for automated scoring developers/vendors? What sorts of evidence seem most important for demonstrating the validity and “reliability” of automated scoring systems? What issues will emerge over the next five years related to automated scoring systems that need to be addressed by the standards? Update on Revisions to the Test Standards 24

  25. Automated Score Reports and Interpretation Use of computer for score interpretation “Actionable” reports (e.g., routing students and teachers to instructional materials and lesson plans based on test results) Documentation of rationale Supporting validity evidence Update on Revisions to the Test Standards 25

  26. Format Issues Consideration of ways to identify of “Priority Standards” More parallelism between chapter Tone Complexity Technical language Organization of Chapters May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 26

  27. Possible Ways to Identify “Priority Standards” Clustering of Standards into thematic topics Over-arching Standards/ Guiding Principles Application Chapters Connection of standards to previous standards May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 27

  28. Organization of Chapters 1999 Testing Standards Three sections Foundation: Validity, Reliability, Test Development, Scaling & Equating, Administration & Scoring, Documentation Fairness: Fairness, Test Takers Rights and Responsibilities, Disabilities, Linguistic Minorities Applications: Test Users, Psychological, Educational, Workplace, Policy May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 28

  29. Possible New Organization Section 1: Core Principles Validity, Reliability, Fairness Testing Operations Test Design and Development, Scaling & Equating, Test Administration & Scoring, Documentation, Test Takers, Test Users Section 2 : Applications Psychological, Educational, Workplace, Policy and Accountability May 1, 2010 Update on Revisions to the Test Standards Update on Revisions to the Test Standards 29

  30. Additional Important Information • Documentation • Third Party Observers • Standardized Administration/Scoring • Norms • Technicians • Supervision • Diverse Populations Update on Revisions to the Test Standards

  31. Feedback Requested • Feedback Information • Web-based • Necessary due to its impact • Feedback Timetable • Comment period- Early 2011 • Revision period- Mid to late 2011 Publication expected- 2012 (APA) Next revision probably in 15 or so years Update on Revisions to the Test Standards

More Related