Test Validation Litigation:

Test Validation Litigation: Legal Requirements and Lessons Learned

Federal Law and Regulations • Title VII of the Civil Rights Act of 1964, 42 USC 2000e • Uniform Guidelines on Employee Selection Procedures • Cases interpreting the foregoing

Title VII • Title VII of the Civil Rights Act of 1964 (as amended in 1991) prohibits basing employment decisions on race, gender, ethnicity, religion, or national origin • Employment decisions include: Recruitment, Hiring, Promotion, Transfer, Wages, Leave, Training, Discipline, and Termination

Title VII Has been interpreted to require that an employer’s selection procedures not result in disparate impact against a protected group, unless the procedure is demonstrated to be “valid and consistent with business necessity”

Title VII • Selection procedure that results in adverse impact is presumed to be discriminatory, unless it is shown to be valid pursuant to the Uniform Guidelines.

Uniform Guidelines • Issued jointly by the EEOC, DOJ, OFCCP and other federal agencies in 1978 • Set forth standards for determining adverse impact • Set forth standards for demonstrating validity of selection procedures • Include important record-keeping requirements

Litigation • Plaintiff or class of plaintiffs allege that selection procedures have adverse impact and are not valid • Plaintiffs must show, usually in a statistical fashion, that the proportion of the protected class hired or promoted was significantly less than would be expected • Raises a rebuttable inference of discrimination

Litigation • In general, plaintiffs can establish a prima facie case of adverse impact by relying upon: • The “80% rule” or • A statistically significant difference, usually expressed as standard deviations

Litigation • Disputes over statistical evidence • Applicant pools and applicant flow • Aggregation of data across jobs, facilities • Inclusion or exclusion of variables • What statistical tests to apply based on sample sizes, etc.

Litigation • Disputes over applicant data will be multiplied after adoption of the new Q&A: • Whether the employer acted to fill a position • Whether the putative applicant complied with employer’s application process • Whether the advertised requirements were valid • Whether criteria mentioned in advertisement were covered requirements

Internet Applicants • EEOC and OFCCP have proposed new Uniform Guidelines Q&A to define “Internet Applicant” • OFCCP has just issued a new rule defining “Internet Applicant” and implementing new recordkeeping requirements • May have broad impact on adverse impact disputes

Proposed Q&A • In order for an individual to be an applicant in the context of the Internet and related electronic data processing technologies, the following must have occurred: • The employer has acted to fill a particular position; • The individual has followed the employer's standard procedures for submitting applications; and • The individual has indicated an interest in the particular position.

Proposed Q&A • Q: Are all the search criteria that employers use subject to disparate impact analysis? • A: Yes. All search criteria used are subject to disparate impact analysis. Disparate impact analysis can be based on Census or workforce data. If a disparate impact is shown, the employer must demonstrate that its criteria are job-related and consistent with business necessity for the job in question. 42 U.S.C. 2000e-2(k).

EEOC Guidance • Search parameters: • An employer who uses specific qualifications to search a database in order to narrow the field will be required to assess whether those qualifications have an adverse impact

OFCCP Rule • The “expression of interest” indicates that the individual possesses the “basic qualifications • Advertised qualifications • Qualifications established in advance

OFCCP Rule • Basic qualifications must be: • Non-comparative • Objective • Job related

Litigation • Once plaintiffs establish adverse impact, burden shifts to employer to demonstrate validity • Expert testimony in support of existing validity evidence • New retrospective validation study

Litigation • Plaintiffs counter with expert testimony attacking the validity evidence • Compliance with Uniform Guidelines • Compliance with professional standards • Alternatives with lesser adverse impact

Litigation • Aspects of the selection process that are subject to challenge • Recruiting and advertising • Application process and applicant definition • MQ or other preliminary screening • Examination, as a whole and by item • Scoring • Use of scores • Final decisionmaking process

Litigation • Aspects of validation that are subject to challenge • Sampling plan and selection of SMEs • Qualification of SMEs • Interaction with SMEs • Analysis of SME input • Qualifications of development staff • continued

Litigation • Design of test instrument and items • Item ratings and item bias • Consideration of alternatives • Compensatory vs. non-compensatory • Alternative use of scores • Consideration of alternative procedures • Documentation • More documentation

Reynolds v. ALDOT • Litigation concerning the validity of minimum qualifications for several job classes • Pursuant to consent decree, defendants had the burden of demonstrating validity in response to objections made by plaintiffs

Reynolds v. ALDOT • Disputes concerning overall MQ development process • Disputes concerning particular job classes • Engineering jobs (including mgmt) • “Right of Way” jobs

MQ Development Process • Gather and analyze background information • Prior selection procedures for same job • Data from other employers • O*NET

MQ Development Process • Select subject matter experts • Job incumbents and supervisors with a minimum level of experience in the job • Representation of • Race • Gender • Functional areas • Geographic locations • Supervisors and incumbents

MQ Development Process • Draft MQ Development form • Used by SMEs as a tool for organizing thoughts about tentative MQ statements

MQ Development Process • SME Meeting • Review list of qualifying KSAs from job analysis • Define “minimum qualification” and discuss purpose • Develop and discuss tentative MQ statements

MQ Development Process • SMEs are instructed that • MQs are designed to identify the barely acceptable applicant • MQs are the minimal levels of education, training, prior work experience or other attribute that would be necessary to acquire the KSAs needed to perform at a minimally acceptable level on Day 1

MQ Development Process • SME-developed tentative MQ statements are “bracketed” with statements requiring lesser and greater amounts of the same type of qualification

MQ Development Process • SME rating session • SMEs rate the proposed MQ statements for suitability and link them to the surviving KSAs

MQ Development Process • Suitability rating • 0: not at all • 1: not enough to expect • 2: appropriately defines • 3: too much to expect

MQ Development Process • Linkage ratings • Dichotomous (yes/no) scale • Can the particular K, S, or A be acquired from the experience described in the proposed MQ statement?

MQ Development Process • SME supplemental information questionnaire • Solicits suggestions for any alternative MQs • Solicits SME input concerning substitution of experience for education, or vice versa

MQ Development Process • SME ratings analyzed by I/O Psychologists • Mean ratings closest to 2.0 • Number of 2.0 ratings • Number of KSAs to which statement linked at .50 or higher • Mode of combined ratings • Adverse impact data • Supplemental SME info

MQ Development Process • Final MQ statements selected using professional judgment after consideration of all these factors

Challenges • Plaintiffs claimed • Education and experience MQs are not capable of content validation • The MQs were not sufficiently specific • “Compound” MQ statements are not permissible • SME ratings and rating scales were improper • continued

Challenges • Necessary at entry ratings didn’t include those who said they didn’t use the K, S, or A • Use of the same SMEs to develop and rate tentative MQ statements was improper • Documentation did not satisfy the requirements of the Guidelines • continued

Challenges • Defendants failed to properly consider alternatives with lesser adverse impact • MQ development form was improper • Low pass rates indicate MQs too difficult • SMEs not properly instructed on how to identify “barely acceptable” applicant

Defense • Lay the foundation • Project manager testifies to establish procedures • Individuals responsible for development work testify about what was done • All necessary documentation introduced

Defense • Documentation is crucial • Demonstrates compliance with Guidelines and professional standards • Corroborates lay witness testimony as to what was done and why • Documentation should include professional literature supporting the methodology

Defense • Expert Witnesses • Experts file report before trial supporting validity • Following lay witnesses, experts testify that procedures and work done comport with Guidelines and professional standards • Experts stress the exercise of professional judgment

Findings • The MQ development process is consistent with the requirements of the Guidelines and leads to content valid MQs

Findings • The Uniform Guidelines operate, of necessity, at a general level. The specific mechanisms for complying with the requirements of the Guidelines are left to the professional judgment of those responsible for their implementation.

Findings • In making determinations concerning compliance with the Uniform Guidelines, it is appropriate for the Court to rely on the testimony of experts

Findings • Defendants documented their MQ development efforts in a manner consistent with the Guidelines

Findings • It was appropriate to validate “compound” MQ statements, where the requirements are designed to work together as a single statement.

Findings • It was appropriate to allow SMEs to rate more than one tentative MQ as “appropriately defining” the barely acceptable applicant, because the SMEs could judge that there was a range of acceptable MQs in the series

Findings • Alternative selection procedures must be considered only where they have • Lesser adverse impact, AND • Substantially equal validity

Findings • Plaintiffs must proffer specific alternatives that they claim should have been considered • Plaintiffs have the burden of showing that any alternative proffered by them would be of substantially equal validity and have lesser adverse impact

Findings • It was appropriate to exclude from the calculation of necessary at entry ratings those SMEs who said they did not currently use a K, S, or A

Test Validation Litigation: