Wallace Judd APA Testing Standards : RFC

1. Wallace Judd APA Testing Standards: RFC Wallace@WallaceJudd.com 1

2. The APA Testing Standards: Request for Comment Wallace Judd, Ph.D. President of Authentic Testing Corporation President-Elect of the PTC Webinar Sponsored by the PTC and by Authentic Testing Corporation. 2

3. The APA Testing Standards: Request for Comment Comments due October 15, 2007 http://www.TestStandards.org 3

4. TestStandards.org 4

5. Webinar is organized in 2 parts: Part I. 20-minute intro to the Standards: • How they were formulated. • Why they’re important. • How they’re likely to impact your program. • Problems with the current Standards. 5

6. Webinar is organized in 2 parts: Part II. Forty minutes of discussion… • Are these issues relevant to you? • What (if anything) do you want to do? • What are Next Steps? 6

7. NCME Session Title 7

8. What The Standards Address: Importance of the Standards – • 30,000 copies sold. • Represents testing practice worldwide. • Basis for ANSI ISO 17024 enforcement • Basis for NCCA Accreditation 8

9. Why Standards Are Important • ANSI – required by DOD for security certifications • NCCA – required by Maine for state accreditations • More coming… 9

10. Who Created the Standards? 75 Organizations provided input, volunteer time, editing, ideas. They were from: • Original Sponsors: AERA, APA, NCME • Academic Institutions • Non-Profit Associations • State & Federal Government Agencies • Test Publishers 10

11. Not a single sponsor or reviewer was from corporate America. 11

12. Current Standards • Reviewed, revised 1996-1998. • Adopted 1998. • Published, disseminated 1999. 12

13. Contents of the Standards • The Standards were written • for the professional and • for the educated layperson 13

14. The Standards address professional and technical issues of test development and use in • education, • psychology and • employment. (APA.org, home page) 14

15. The Standards are the Bible of Testing in the U.S. 15

16. Standards = The Bible 16

17. Contents of theStandards This book is a vitally important reference for • professional test developers, • sponsors, • publishers, • users, • policymakers, • employers,and • students (http://www.APA.org, home page) 17

18. But the Standards clearly target educational & psychological testing. 18

19. Few Job-Related Standards • Educational – measuring and predicting student achievement • Psychological – measuring and evaluating psychological traits • Sections on Performance Testing and corporate / industrial testing – minimal. 19

20. Initial Issues 20

21. Problem: Bylaws Standard 2. A. The certifying program must show that the governance structure, policies, and procedures that have been established protect against undue influence that could compromise the integrity of the certification process. 21

22. Problem: Governance Structure Standard 2. B.The governance structure, policies, and procedures must provide forautonomy in decision making regarding important aspects of the certification program, such as eligibility standards; the development, administration, and scoring of the assessment instruments; selection of personnel; and operational processes. 22

23. Problem: No Training & Testing In the Same Organization • Standard 2. D. To avoid conflicts of interest between certification and education functions, the certification agency must not also be responsible for accreditation of educational or training programs or courses of study leading to the certification 23

24. Problem: Board Composition • Standard 3. The certification board or governing committee of the certification program must include individuals from the certified population, as well as voting representation from at least one consumer or public member. 24

25. Problem: No Place for Gating Items Gating Items – Items Required to Pass Test • Pilot’s Test – Land the plane • Veterinarian’s Test – Spay a cat • Crane Operator’s Test – Put ball in barrel • Word Test – Save a file • System Admin. Test – Add a new user 25

26. Problem: No Performance Items • Standard 3. (p. 39) When [using IRT], the fit of the model is evaluated … by evaluating unidimensionality, local independence, speededness, and equality of slope parameters. 26

27. Problem: Expert Judges • Standard 3.6 The qualifications, relevant experience, and demographic characteristics of expert judges should also be documented. 27

28. Problem: Content Domain • Standard 3.11 Test developers should document the extent to which the content domain of a test represents the defined domain and test specifications. 28

29. Problem: Publish Scoring Criteria Standard 3.20When appropriate, …criteria for scoring …should be provided to the test takers prior to the administration of the test. 29

30. Why Do I Care? • I’ve developed tests since 1981 at Xerox, Oracle, Honeywell, McGraw Hill, Kelly Services, and my tests were the foundation for Microsoft’s MOUS tests. • I believe that skills testing and certification deserves as much recognition and validation as psychological or educational testing. 30

31. Why Does the PTC care? • It has many members who have testing programs, and who want them to exemplify best practices in testing. • PTC meets October 17-19 in Scottsdale. • PTC will formulate its own response to the RFC. • PTC is at PerformanceTest.org 31

32. How important is compliance with the Standards to your program? • Insignificant • A small factor • Significant • A large factor • Critical 32

33. How important is NCCA accreditation to your program? • Insignificant • A small factor • Significant • A large factor • Critical 33

34. How important is ANSI accreditation to your program? • Insignificant • A small factor • Significant • A large factor • Critical 34

35. We have two routes to go. What should we do? • Formulate separate standards for corporate-sponsored testing • Work with AERA, APA, NCME, to revise the Standards. • Both 35

36. Which would you participate in? (You can vote for more than 1.) • Individual corporate response • Industry-specific group response • Response representing corporations with testing programs 36

37. Plug: The PTC • Where: Scottsdale, Arizona • When: October 17– 19, 2007 • Who: • Graham Brent – Crane Operators’ Exam • Ric Luecht – Engineering Assessments • Brian Rice – How Virtualization Works http://www.PerformanceTest.org 37

38. Plug: Negotiating the Standards • Workshop October 16th, PTC Summit in Phoenix. 38

39. Plug: AuthenticTesting.com • Automated Job Task Analysis • Automated Exam & Item Analysis • Performance Test Design 39

40. Wallace Judd The Performance Testing Council • Wallace@WallaceJudd.com • Julie@PerformanceTest.org 40