PSY 6430 Unit 6

1. Lecture: Wednesday, 3/13 ME1: Monday, 3/18 Lecture: Wednesday, 3/20 Exam: Monday, 3/25 PSY 6430 Unit 6 Last day to withdraw: Monday, 3/18 Task Analysis Workshop: Wednesday, 3/27 Validity Determining whether selection instruments are job-related

2. Task Analysis Workshop 3/27 • Purpose: To ask me questions about your task statements and KSAs/WRCs for your project • I will not review spelling/grammar or formatting of the task analysis document itself • Come prepared with specific questions • I will meet with you individually or in small groups • Assign times just so you don’t have to wait around too long • Email me no later than 3/20 if you plan on attending; I will give you a schedule on 3/25 (completely optional) (also welcome, of course, to come to my office hours)

3. Task Analysis Workshop 3/27 • You can also, of course, come to my office hours and meet with me, if you prefer • The task analysis is due on Monday, 4/08 if you want your grade by 4/17 in order to determine if you want to take ME2 • The final due date is Monday, 4/22 at 5:00 pm • Look at the syllabus or U4 ppt for contingencies regrading lateness

4. SO1: NFE, Validity, a little review • Predictor = test/selection instrument • Use the score from the test to predict who will perform well on the job • Possible confusion (again) • You need to determine the validity of the test based on your current employees • Then you administer it to applicants and select employees based on the score (a few students had a problem distinguishing between validity and reliability on E5, example next)

5. SO1: NFE, Validity, example • Administer a test to current employees • Obtain measures of how well they perform on the job • Correlate the test scores with the performance measures • Assume: The correlation is statistically significant • Assume: Current employees who score 50-75 also are performing very well on the job • Now you administer the exam to applicants, predicting that those who score 50-75 will also perform well on the job

6. SO2: Reliability vs. Validity • Reliability Operational Definition: Is the score on the measure stable, dependable, and/or internally consistent? Conceptual Definition: Are you actually measuring what you want to be measuring? • Validity Is the measure related to performance on the job?

7. SO3: Relationship between reliability and validity • A measure can be reliable, but not valid • However, a measure cannot be valid unless it is reliable • *Reliability is a necessary but not sufficient condition for validity • Text gives a perfect example You can reliably measure eye color, however, it may not be related to job performance at all *key point

8. Types of validation procedures • Content: expert judgment • Criterion-related: statistical analyses (concurrent & predictive) • Construct (but not practical-not covering this) • Validity generalization (transportable, no local validity study – jobs are similar) • Job component validity (not covering this) • Small businesses: Synthetic validity (not covering it, not very relevant now –content validity) (main types are the two kinds of criterion-related and content validity; construct really a hold over from test construction - not very relevant - I have only seen this used by a few organizations – create their own tests; cover validity generalization, but right now while validity generalization has excellent professional support, may not be legal - professional guidelines depart from legal; in one case, 6th Circuit Court ruled it illegal as a matter of law based on Griggs/Duke and Albermarle - 1987)

9. SO5 NFE but related to 7A which is: Difference between content and criterion-related validity • Content validity relies on expert judgment and a match between the “content” of the job and the “content” of the test • Expert judgment refers to • the determination of the tasks , KSAs and WRCs required to perform the job via a task analysis and task inventory/questionnaire • linking the KSAs and WRCs to selection procedures that measure them (actually covered this in U4, job analysis, but..)

10. SO5 NFE but 7B is: Difference between content and criterion-related validity • Empirical or criterion-related validity relies on statistical analyses (correlation of test scores with measures of job performance) • Measures of job performance = criterion scores • Two basic types of criterion-related validity • Concurrent validity • Predictive validity (content next slide)

11. Intro to content validity • You do NOT use statistical correlation to validate your tests • Validation is based “only” on your job analysis procedures and descriptively linking the KSAs and WRCs to selection measures • It is much more widely used than criterion-related validity • Particularly since Supreme Court ruled it was OK to use for adverse impact cases (1995) (again, to emphasize)

12. SO6: Two reasons why content validity is often used • It can be used with small numbers of employees • Large sample sizes are required to use criterion-related validity due to the correlation procedures • The text later when talking about criterion-related validity indicates you may need over several hundred • Dickinson: usually 50-100 is adequate • How many companies have that many current employees in one position or select that many employees for one position? • You can do a study over a couple of years but that obviously delays the results of the study (small number of incumbents)

13. SO6: Two reasons why content validity is often used • Many organizations do not have good job performance measures • You need good performance criterion measures to do a criterion-related validity study because you correlate the test scores with job performance measures

14. SO7B: Content vs. criterion-related validity and the type of selection procedure • If you use content validity you should write the test, not select an off-the-shelf test • If you use criterion-related validity, you can do either • It is much easier and less time consuming to use an off-the-shelf test than to write one! (Covered SO7A; VERY IMPORTANT!; book waffles on this a bit, indicating that emphasis should be placed on constructing a test, But only in rare situations would I recommend selecting off-the-shelf test with content validity - legally too risky; why, next slide)

15. SO7B: Why should you write the test if you use content validity? (this slide, NFE) • Content validity relies solely on the job analysis • The KSAs must be represented proportionately on the selection test as indicated in the job analysis in terms of: • Their relative importance to the job • The percentage of time they are used by the employees • It is highly unlikely that an off-the-shelf test will proportionately represent the KSAs as determined by your job analysis • In some discrimination court cases, the judge has gone through the test item by item to determine whether the items were truly proportional to the KSAs as determined by the job analysis • Thus, there is both a professional measurement reason and a legal reason to write the test rather than using an off-the-shelf test (not covering SO8: straightforward)

16. SO9: Major steps of content validity - very, very specific requirements for the job analysis • Determine the tasks for the job • *Determine the criticality and/or importance of each of the tasks • Specify the KSAs required for EACH task • KSAs must be linked to each task (NFE) *Now because of ADA, is it an essential function? (cont. next slide)

17. SO9: Major steps of content validity, cont. • Determine the criticality and/or importance of each KSA* • Describe the relationship between each KSA and each task statement • You can have KSAs that are required for only one or two tasks, or you can have KSAs that are required to perform several tasks • The more tasks that require the KSAs, the more important/critical they are • Describe the complexity or difficulty of obtaining each KSA (formal degree, experience) • Specify whether the employee must possess each KSA upon entry or whether it can be acquired on the job (cannot test for a KSA if it can be learned within 6 months) • Indicate whether each KSA is necessary/essential for successful performance of the job *Only the first major point will be required for the exam, but I want to stress how detailed your job analysis must be for content validity (cont on next slide)

18. SO9: Major steps of content validity, cont. • Link important job tasks to important KSAs* (FE) • Reverse analysis; you have linked the KSAs to the tasks, now you must link the KSAs to the tasks (NFE) • KSA # 1 may be relevant to Tasks 1, 6, 7, 10, 12, & 22 • KSA # 2 may be relevant to Tasks 2, 4, & 5 • Etc. • (NFE) Develop test matrix for the KSAs • If you want see how you go from the task analysis to the actual test, turn back to pages 91-96, Figures 3.17, 3.18, 3.19

19. SO10: When you can’t use content validity according to the Uniform Guidelines • When assessing mental processes, psychological constructs, or personality traits that cannot be directly observed, but are only inferred • You cannot use content validity to justify a test for judgment, integrity, dependability, extroversion, flexibility, motivation, conscientiousness, adaptability, or any personality characteristic • The reason for that is that you are basing your job analysis on expert judgment - and judgment is only going to be reliable if you are dealing with concrete KSAs such as mechanical ability, arithmetic ability or reading blue prints • The more abstract the KSA, the less reliable judgment becomes • If you can’t see it, if you can’t observe it, then the leap from the task statements to the KSAs can result in a lot of error • Per U4, you can rewrite the trait making it objective/observable (review from U4text mentions three; I am having you learn the first one and one I added in the SOs -- these are the two that are most violated in practice; the second one is relevant to BOTH content and criterion-related so shouldn’t be listed under when you can’t use content validity: cannot test for KSAs that can be learned on the job)

20. SO10: When you can’t use content validity according to the Uniform Guidelines, cont. • When selection is done by ranking test scores or banding them (from U1) • If you rank order candidates based on their test scores and select on that basis, you cannot use content validity - you must use criterion-related validity • If you band scores together, so those who get a score in a specified range of scores are all considered equally qualified, you cannot use content validity - you must use criterion-related validity • Why? If you use ranking or banding, you must be able to prove that individuals who score higher on the test will perform better on the job - the only way to do that is through the use of statistics The only appropriate (and legally acceptable) cut-off score procedure to use is a pass/fail system where everyone above the cut-off score is considered equally qualified (only relevant if adverse impact)

21. SO11: Conflict between the Uniform Guidelines and selection specialists (293,2 not 293,1) • As indicated previously the UG state that content validity may not be used to support mental processes-cognitive abilities-such as “intelligence” • However, there is general agreement among selection specialists that content validity can be used for general cognitive ability • In fact, more recent standards, specifically, the Standards for Educational and Psychological Testing and the Principles for the Validation and Use of Personnel Selection Procedures, state that content validity can be used to support the use of some cognitive ability tests (another one of the discrepancies between the law and the profession; remember the law/judges will follow what is in the UG)

22. SO11: Why the conflict? • Remember that the Uniform Guidelines were passed/agreed to by all of the agencies involved in selection and placement in 1978 • Since that time, the courts have used the Uniform Guidelines as a checklist for what is and what is not accepted • Once precedents are determined in case law through court rulings (in this case based on the Uniform Guidelines), current and future decisions are based on those precedents/past rulings • New knowledge has accumulated since the Uniform Guidelines were approved, however • This new knowledge is reflected in the two professional documents, the Standards and Principles, which are updated by the relevant professional organizations periodically (American Educational Association and SIOP)

23. Criterion-related validity studies:Concurrent vs. predictive • SO12A: Concurrent validity Administer the predictor to current employees and correlate scores with measures of job performance Concurrent in the sense that you have collected both measures at the same time for current employees • SO16A: Predictive validity Administer the predictor to applicants, hire the applicants, and then correlate scores with measures of job performance collected 6-12 months later Predictive in the sense that you do not have measures of job performance when you administer the test - you collect them later (comparison of the two, SO12A, describe concurrent validity; SO16A, describe predictive validity)

24. SO12B: Steps for conducting a concurrent validity study • Job analysis: Absolutely a legal requirement • Discrepancy between law and profession (learn for exam) • Law requires a job analysis (if adverse impact & challenged) • Profession does not as long as the test scores correlate significantly with measures of job performance • Determine KSAs and other relevant requirements from the job analysis, including essential functions for purposes of ADA • Select or write test based on KSAs (learn for exam) • May select an off-the-shelf test or • Write/construct one

25. SO12B: Steps for conducting a concurrent validity study • Select or develop measures for job performance • Sometimes a BIG impediment because organizations often do not have good measures of performance • Administer test to current employees and collect job performance measures for them • Add this step: Correlate the test scores with the job performance measures • Add this step: Determine whether the correlation is statistically significant at the .05 level You can then use the test to select future job applicants

26. SO13A: Advantage of concurrent validity over predictive validity • Because you are using the test data and performance data from current employees, you can conduct the statistical validation study quickly – in a relatively short period of time • Remember, that with predictive validity, you must hire applicants and then wait 6-12 months to obtain measures of job performance (post-training, after they have learned the job)

27. SO13B: The basic reason that accounts for all of the weaknesses with concurrent validity • All of the weaknesses have to do with differences between your current employees and applicants who apply for the job in the future • You are conducting your study with one sample of the population (your employees) and assuming conceptually that your applicants are from the same population • However, your applicants may not be from the same population - they may differ in important ways from your current employees • Ways that would cause them (as a group) to score differently on the test or perform differently on the job, affecting the correlation (job relatedness) of the test (text lists several weaknesses and all of them really relate to one issue; dealing with inferential statistics here; as per the SO, only partial credit for the first bullet))

28. SO14: Restriction in range • This is the term used for the statistical/mathematical reason why the differences between your current employees and applicants affect validity • It also explains from the last unit, why reliability is generally higher when • Your sample consists of individuals who have greater differences in the ability for which you are testing: High school students, community college students, vs. engineering majors in college who take a math test • The questions are moderately difficult – about 50% of test takers answer the questions correctly – rather then when the questions are very easy or very difficult (so I have talked about this before, I just haven’t named it)

29. SO14: Restriction in range • With criterion-related validity studies the ultimate proof that your selection test is job related is that the correlation between the test scores and job performance measures is statistically significant • A high positive correlation tells you • People who score well on the test also perform well • People who score middling on the test are also middling performers • People who score poorly on the test also perform poorly on the job • In order to obtain a strong correlation you need • People who score high, medium, and low on the test • People who score high, medium, and low on the performance measure

30. SO14: Restriction in range, cont. • That is, you need a range of scores on BOTH the test and the criterion measure in order to get a strong correlation • If you only have individuals who score about the same on the exam, and some perform well, middling, and poorly, you will get a zero correlation (makes sense, you can’t predict performance from the test) • Similarly if you have individuals who score high, medium, and low on the test, but they all perform about the same, you will get a zero correlation (again, it makes sense, you can’t predict performance from the test)

31. SO14: Restriction in range, cont. • Any procedure/factor that decreases the range of scores on either the test or the performance measure • Reduces the correlation between the two and, hence, • Underestimates the true relationship between the test and job performance • That is, you may conclude that your test is NOT valid, when in fact, it may be

32. SO14: Restriction in range, cont. • Restriction in range is the technical term for the decrease in the range of scores on either or both the test and criterion • Concurrent validity tends to restrict the range of scores on BOTH the test and criterion, decreasing the correlation between the two, and hence underestimating the true validity of a test • In other words, to be repetitive, this increases the possibility that you will conclude that your test is not valid, when in fact, it is (finally - the definition; stress the either or both; cont on next slide)

33. SO14: Restriction in range, cont.Also related to SO15 • Why? You are using current employees in your sample • Your current employees have not been fired because of poor performance (only ”good” employees are left) • Your current employees have been doing the job for a while and thus are more experienced • All of the above would be expected to • Result in higher test scores than for the population of applicants • Result in higher performance scores than for the population • Thus, restricting the range of scores on both the test and the performance criterion measure (diagrams on next slide)

34. High Performance Low Low High Test Scores High Performance Low Low High Test Scores SO14: Restriction in range, cont. • Top diagram • No restriction in range • Strong correlation • Bottom diagram • Restriction in range • Test scores and • Performance scores • Zero correlation (extreme example, but demonstrates point - concurrent validity is likely to restrict range on both, underestimating true validity)

35. SO16: Predictive validity • SO18A: Predictive validity (review) Administer the predictor to applicants, hire the applicants, and then correlate scores with measures of job performance collected 6-12 months later Predictive in the sense that you do not have measures of job performance when you administer the test - you collect them later, hence, you can determine how well your test actually predicts future performance

36. SO16B: Steps for a predictive validity study • Job analysis: Absolutely a legal requirement • Determine KSAs and other WRCs from the job analysis, including the essential functions for purposes of ADA • Select or write test based on KSAs* • You may select an off-the-shelf test or • Write/construct one • Select or develop measures for job performance *Learn this point for the exam (first four steps are exactly the same as for a concurrent validity study)

37. SO16B: Steps for a predictive validity study • Administer the test to job applicantsand select randomly or using the existing selection system • Do NOT use the test scores to hire applicants (I’ll come back to this later) • After a suitable time period, 6-12 months, collect job performance measures • Add this step: Correlate the test scores with the performance measures • Add this step: Determine whether the correlation is statistically significant and if it is, your test is valid

38. SO17: Two weaknesses of predictive validity (299,2 not 299,3) • Time it takes to validate the test • Need appropriate time interval after applicants are hired before collecting job performance measures • If the organization only hires a few applicants per month, it may take months, a year, or even more than a year to obtain a large enough sample to conduct a predictive validity study (N=50-100)

39. SO17: Two weaknesses of predictive validity • Very, very difficult to get managers to ignore the test data (politically very difficult) • Next to impossible to get an organization to randomly hire - some poor employees ARE going to be hired • Also difficult to convince them to hire using the existing selection system without using the test score (but much easier than getting them to randomly hire and doable) I love the reported conversation between one of the authors and a VP of personnel included in the text. (I don’t blame them; it would be like us randomly accepting students into the graduate program)

40. Predictive Validity: Three basic ways to do it • Pure predictive validity: by far the best Administer the test to applicants and randomly hire • Current system: next best, more practical Administer the test to applicants, use the current selection system to hire (NOT the test) • Use test to hire: bad, bad, bad both professionally and legally Administer the test, and use the test scores to hire applicants (going to come back to these and explain the evaluations; text lists the third as an approach! Click: NO!!)

41. SO18: Predictive validity designs • Figure 8.5 lists 5 types of predictive validity designs • Follow-up: Random selection (pure predictive validity) • Best design • No problems whatsoever from a measurement perspective; completely uncontaminated from a professional perspective • Follow-up: Use present system to select • OK and more practical, but • It will underestimate validity if your current selection system is valid; and the more valid it is the more it will underestimate the validity of your test • And, why will it underestimate the validity? (answer not on slide)

42. SO18: Predictive validity, selection by scores • Select by test score: Do NOT do this!!! • Professional reason: • If your selection procedure is job related, it will greatly underestimate your validity - and, the more job related the selection procedure is, the greater it will underestimate validity. • Why? If your test is valid, you are severely restricting the range on both your test and your job performance measures! • Thus your correlation will be much smaller than it should be • And, you are likely to conclude that your test is not valid when in fact it is (professional and legal reasons not to do this)

43. SO18: Predictive validity, selection by scores • Legal reason: • If adverse impact occurs you open yourself up to an unfair discrimination law suit • You have adverse impact, but you do not know whether the test is job related There is a caveat (NFE): Some courts have ruled that adverse impact is OK if a validation study is in progress. However, I see this as being way too risky legally (particularly given the technical problems with this method).

44. SO18: NFE, Further explanation of types of predictive validity studies from Figure 8.5 • Hire, then test and later correlate test scores and job performance measures • If you randomly hire, this is no different than pure predictive validity: #1 previously, Follow-up: Random selection • If you hire based on current selection system, this is no different than #2 previously, Follow-up: Select based on current system (both are perfectly acceptable; one more slide on this)

45. SO18: NFE, Further explanation of types of predictive validity studies • Personnel file research - applicants are hired and their personnel records contain test scores or other information that could be used as a predictor (i.e., perhaps from a formal training program). At a later date, job performance measures are obtained.

46. For exam:Rank order of criterion-related validity studies in terms of professional measurement standards 1. Predictive validity (pure) - randomly hire 2.5 Predictive validity – use current selection system 2.5 Concurrent validity 4. Predictive validity – use test scores to hire

47. SO19: Which is better: Predictive vs. concurrent, research results • Data that exist suggest that: • Concurrent validity is just as good as predictive validity for ability tests (most data) • May not be true for other types of tests such as personality and integrity tests • Studies have shown differences between the two for these type of tests – as well as for structured interviews and biographical data inventories - so proceed with caution! (Conceptually, predictive validity is better, it has more fidelity with, is more similar to the actual selection procedure; test applicants, select, and see how well they do on the job later)

48. SO20: Sample size needed for a criterion-related validity study (review) • Large samples are necessary • The text doesn’t give a “magic” number, understandably • A sample of 100 is usually adequate – and even 50 is possible (but risky): learn Dickinson’s number • What do companies do if they do not have that many employees? • They use content validity • They could possibly also use validity generalization, but even though this would be professionally acceptable, at the current time it is still legally risky

49. Validation procedures • Content: expert judgment (no statistical analysis) • Criterion-related: statistical analyses (concurrent & predictive) • Concurrent: Correlate test scores with performance measures for current employees • Predictive: Correlate test scores for individuals you hire with performance measures for those individuals 6-12 months later • Construct (but not practical-not covering this on the exam) • Validity generalization (transportable, no local validity study – jobs are similar)

50. SO21: NFE, Construct validity • Every selection textbook covers construct validity • I am not covering it for reasons indicated in the study objectives • Basic reason for not covering it is that while construct validity is highly relevant for test construction, very, very few organizations use this approach - it’s too time consuming and expensive • First, the organization develops a test and determines whether it is really measuring what it is supposed to be measuring • Then, they determine whether the test is job related