Using tests to predict job performance CALSWEC - 2007

1. Using tests to predict job performanceCALSWEC - 2007 Cynthia Parry, PhD C.F. Parry Associates Michelle Graef, PhD Center on Children, Families & the Law – University of Nebraska, Lincoln Todd Franke, PhD Department of Social Work – University of California at Los Angeles Henry Ilian, PhD New York City Administration for Children’s Services James Satterwhite Academy for Child Welfare Training

2. Using tests to predict job performanceCALSWEC - 2007 • Goals: • Define high-stakes testing • Review technical complexities and professional standards • Discuss the risks, rewards & consequences of high-stakes testing • Anticipated Outcome • A phone call to one or more of the presenters prior any venture in high-stakes testing development

3. Using tests to predict job performanceCALSWEC - 2007 Presentation in 4 parts: 1. Overview of high stakes testing 2. Validity considerations 3. Validity issues in high stakes testing and training of child welfare workers 4. The ethics of using high-stakes testing and the ethics of not using high stakes testing

4. Part 1 of 4:Overview of High Stakes Testing Cindy Parry, PhD

5. What do we mean by high stakes? • Any use of a test that might affect • Your ability to get or keep a job • Your chances for promotion or raises • Your supervisor’s evaluation of your work • Testing out of training requirements or needing further coaching or other interventions • Such as • Licensing exams • Civil service hiring tests • Certification/credentialing tests • Any testing where individual results are reported to a supervisor/manager or used to predict future competent performance

6. What is the Law and Policy Context? • 14th Amendment Requirements • Equal protection • Due process • Civil Rights Acts • Disparate treatment of protected group • Disparate impact on a protected group • Business necessity • ADA • Accommodations for test takers Taken from Mehrens, W. Legal and Professional Bases for Licensure Testing. In Impara, James (Ed.). (1995) Licensure Testing: Purposes, Procedures and Practices. Buros Institute of Mental Measurements: University of Nebraska, Lincoln.

7. Considerations in Developing a High Stakes Test • Use professional standards to guide test development • The Standards for Educational and Psychological Testing (AERA, APA, NCME) • Uniform Guidelines on Employee Selection Procedures (EEOC) • Principles for the Validation and Use of Personnel Selection Procedures (SIOP)

8. Key Considerations in Validation • Validity is the most fundamental consideration in developing any test • We validate the use of a test score for a specific purpose — not a test or an item • Content validity is the most common type of evidence cited in child welfare applications • Requires full specification of the domain or construct it is intended to measure (1.6) • Standards require complete description of procedures used to develop test content (1.6) • Use of subscales requires validation of the inferences made from subscale scores as well as overall score (1.12) Standards cited from: American Educational Research Association, American Psychological Association and National Council on Measurement in Education. (2002). Standards for Educational and Psychological Testing. American Educational Research Association: Washington, DC.

9. Other Considerations for High Stakes Testing • Adequate reliability for overall scores and any sub-scores that will be used in decision making • Investigations of differential item functioning and test fairness • Comparability of alternate forms or tests given over time • Documentation of the method of setting the cut score and the qualifications of judges • Notice of testing and due process procedures • Policy and procedures for accommodating persons with disabilities • Policy and procedures for test security and identification of test takers

10. Part 2 of 4:Validity Considerations Michelle Graef, PhD

11. Can we use our training evaluation knowledge test to make decisions about candidates’ ability to perform the job? Three points to consider: • Used alone, a written knowledge test is unlikely to be adequate to make inferences about the full range of KSAOs needed to perform child welfare work. • The target of your inferences indicates the appropriate content domain to be sampled. The typical training evaluation knowledge test (“post-test”) items have been sampled from the curriculum domain, which may not accurately represent the job domain. • Even if you have strong evidence of content validity, evidence derived from other validation strategies (criterion-related validity) is recommended to support an employment decision. The implied inference is prediction of future job performance.

12. Job Domain (KSAOs): Determined through a Job Analysis Other Characteristics (values, beliefs, attitudes, personality) Skills Knowledge Abilities

13. Training Curriculum Content [K,S,O] Test Item Content [K] Test Item Content [K] Sampling for development of a training evaluation knowledge test designed to assess trainees’ “mastery of training content”

14. Job Domain [K,S,A,O] Training Curriculum Content [K,S,O] Test Item Content [K] Example of a training evaluation knowledge test that is inappropriate for assessing individuals’ “ability to perform the job”

15. Job Content [K,S,A,O] Test Item Content [K,S,O] Training Curriculum Content [K,S,O] IDEAL: Appropriate sampling for training evaluation test designed to assess individuals’ “ability to perform the job”

16. Conceptual basis for test validation strategies Criterion Measure (e.g., supervisor rating, case review scores) Predictor Measure (e.g., test) Criterion-related validity Content validity Construct validity Predictor Construct Domain (e.g., knowledge of child maltreatment, critical thinking skill, conscientiousness) Criterion Construct Domain (e.g., job performance) Adapted from: Standards for Educational & Psychological Testing (1999)

17. Criterion-related validation study • Demonstrate relationship between predictor and criterion using statistical significance testing • Feasibility depends upon: • Availability of appropriate criterion measures • Representativeness of research sample • Adequacy of statistical power • Variety of designs: • Predictive • Concurrent • Use of incumbents or job applicants • Development of predictor and criterion measures that are relevant, uncontaminated, not deficient, free from bias, and demonstrate reliability • Refer to Principles for the Validation and Use of Personnel Selection Procedures (2003)

18. Recommendations Strategies: 1) retrofit: map your existing knowledge test to the job domain and supplement it with other assessments (e.g., skills) to more adequately represent the job domain 2) start over: develop a test specifically for this purpose that directly samples the job domain, such as a work sample test, simulation, or assessment center 3) gather evidence of criterion-related validity in support of the use of your existing training evaluation knowledge test for employment decisions

19. Part 3 of 4:Validity issues in high stakes testing and training of child welfare workers Todd Franke, PhD

20. Validity issues in high stakes testing and training of child welfare workers Are we measuring what is important?

21. Los Angeles Context • Evidence based practice • Evidence based training • High stakes in Los Angeles • Minimally competent CPS workers • What would one look like?

22. Construct Validity • Translation validity -- Focuses on whether the operationalization (i.e., measure) is a good translation of the construct • Face Validity -- On itsface, does the operationalization look like a good translation of the construct? • Content Validity -- Operationalization is checked against the relevant content domain for the construct • Criterion-related Validity • Predictive Validity -- Operationalization’s ability to predict something it should theoretically be able to predict (L3) • Concurrent Validity • Convergent Validity • Discriminant Validity

23. So what constitutes the domains relevant in training child welfare workers?What should be trained/assessed?It depends who you ask! • Knowledge of • Policies ( over 384 policies in LA-DCFS) • Procedures • submitting mileage • cellular phone reimbursement • use of logs (court calendars, visitation) • new case to-do’s • cubicle organization and systems

24. Examples of current topics • Adult substance abuse • Child neglect • Concurrent planning • Legal foundations • Risk assessment • Sexual abuse • Legal foundations • Legal permanence • Social work values • Worker safety • Physical abuse • Appreciating multiculturalism • Family preservation/Alternative response • Adoption assessments • Home assessments • Team Decision Making • Going to court/testifying • Court report writing • SDM tools • Basic interviewing

25. What else might be important?Suggestions from supervisors/trainees • Writing • Self-reliance • How to deal with emotional aspect of job • Critical thinking • Issues related to burn-out • Anything that links directly to case work practice • Readiness to learn • Ability to multi-task • Willingness to accept supervision • Ability to make hard decisions • Ability to transfer knowledge to field • Social support • Problem solving Are these training issues or hiring criteria?

26. What’s next? • How should an assessment be used? • Is knowledge only one of the domains that define the construct? • Does it have any predictive validity? • Is it a valid measure of future job performance?

27. Part 4 of 4:The Ethics of Using High-Stakes Testing and the Ethics of Not Using High Stakes Testing Henry Ilian, PhD

28. High-stakes Tests Accomplish Three Things • Screen out trainees who cannot demonstrate a specified level of mastery • Compel Studying • Enforce Fidelity to the Curriculum

29. A Comparison from NYC Compel Studying N Items Pretest Posttest CW Common Core 74 80 39.70% 86.10% (High Stakes) Core Essentials for 64 32 44.30% 63.40% Experienced CWs (Not High Stakes) Enforce Fidelity to the Curriculum Core Essentials for Experienced CWs Posttest 1 64 15 72.80% Posttest 2 64 17 55.10%

30. Using and Not Using High-Stakes Tests Each Has Consequences • Using High-Stakes Tests • The requirements for a professionally developed testing program require significant agency resources • Some people with the potential to be good child protective workers may not pass • Not Using High-Stakes Tests • Risk to families, children, self and colleagues • Difficulty maintaining employee standards and agency morale

31. The Major Ethical Issue is the Potential for Harm • There is a need to balance potential harm • to caseworker trainees • to children and families served by CPS agencies • to co-workers and the agencies themselves

32. Concerns for the Evaluator • The organization may be reluctant/unable to make available the time and resources to develop and validate instruments • Testing decisions are often made in a political or administrative context • Administratively imposed testing may not meet professional standards