Reliability

1. Reliability • Consistency of measurement. • Reliability coefficient: correlations among repeated applications of same measure. • Range from -1.0 to 1.0 (lowest reliability approaches zero) • Forms of Reliability: • test-retest • parallel forms • internal consistency (raters/items)

2. Errors of Measurement • Observed score = true score + error score • observed = test score • true score = perfect reliability • error score = extraneous factors • Greater reliability = observed score approximates true score. • Standard error of measurement: range of variation around score due to error variance. • Measures vary in reliability: high (physical characteristics); moderate (abilities, skills); low (personality).

3. Validity • Does the test measure what we think it measures? • Validity coefficient: correlation between predictor (test) and criterion (job performance, gpa). • Range from -1.0 to +1.0 • Criterion used (job performance) must be valid: no point in developing a great test to predict kaka. • Reliability is a “prerequisite” to validity: • Can have reliable measure (astrology) that is not valid, but cannot have a valid measure that is not reliable.

4. Types of Validation Strategies • Criterion-Related Validity • Predictive • Concurrent • Content Validity • Construct Validity

5. Predictive Validity • Uses applicant pool to validate test. • Steps: • Administer new test to group of applicants. • Use old test to select. • After time, collect criterion data. • Compute validity coefficient. • Advantage: • Represents applicant pool. • Preferred by EEOC. • Disadvantage: • Hire unqualified employees. • Time.

6. Concurrent Validity • Uses current employees to validate test. • Steps: • Administer predictor to existing employees. • Collect criterion data immediately. • Compute validity coefficient. • Advantages: • Convenient, efficient, cheap. • Saves time. Preferred by employers. • Disadvantages: • Current employees may differ from applicants: diversity, motivation/anxiety, and on-the-job learning. • Restriction of range on criterion.

7. Content Validity • Content of test reflects KSA’s needed for job. • Work sample test (typing, in-basket, driving). • Not quantitative: no validity coefficient. • Matter of judgement. • Steps: • Test based on job analysis. • KSAs are observable, not abstract mental constructs. • KSAs used in test are important and frequently used in job. • KSAs aren’t learned on the job. • Small difference between test and job content. • Use for absolute cut-off (can/can’t do job) rather than ranking. • Strong “face validity.”

8. Construct Validity • Construct:unobservable attribute (leadership, aggression) • Steps: • Test related to other established measures of construct. • New leadership test related to LBDQ. • Test unrelated to irrelevant measures (Aggression). • New measure 1 --------(unrelated)---------Aggression 1 • + + • Established measure----(unrelated)--------Aggression 2 • EEOC: new test must be related to established measure of construct AND established measure must have criterion validity (related to effective performance). • Construct incorporates criterion-related validity.

9. Selection Methods • Application Blanks • Weighted Application Blanks • Training and Experience Evaluations (.18) • Biodata (.32-.46) • Reference Checks (.26) • positively biased; law suits for slander vs. law suits for negligence in hiring violent employees. • Cognitive Ability Tests (.53) • (GATB General Aptitude Test Battery) • “g” factor; validity generalization (cont’d)

10. Selection Methods (cont’d) • Personality Tests (.20-.30). • Self-report (MMPI; CPI; 16PF). • Projective (inkblots: Rorschach; pictures: Thematic Apperception Test). • Work Samples. • Performance Tests. • Motor/verbal on-the-job behaviors. • Assessment Centers (.30-.60). • multiple measures of key job dimensions. • panel of assessors. • in-basket, leaderless group discussion, management games, case analyses, written tests & interviews.

11. Interviewing • Most popular technique; low validity (.14). • Interviewing used: • assess social skills. • assess ability to work with individual. • face validity. • expected by interviewer and interviewee. • Problems with interviews: • snap judgements; first impression determines final decision; one characteristic overshadows others (ie clothes) • interviewer bias: • “like-me bias” • stereotype; increase with lack of information. • sex role expectations. (Cont’d)

12. Interviewing (cont’d) • Effective Interviewing • Structured Interview: standardized questions asked of all applicants based on job analysis. • Situational Interview • how would you handle xyz? • Behavioral Description Interview • How did you handle xyz in the past? • Multiple questions for each KSA, formally scored. • Team Interview • planned roles. • Train the Interviewer • asking good questions, listening, controlling bias, documentation.

13. Proctor & Gamble Interview Protocol • 5 factors: • Stamina and agility. • Willingness to work hard. • Working well with others. • Learning the work. • Initiative. • “Describe relevant experiences that may have occurred in either a work or non-work setting for each of the 5 factors.” • Trained interviewers use structured interviews. • Applicants rated on 7-point scale for each factor. • Use “evidence organizer”: worksheet with a section for each factor; contains questions and reminders to guide interviewers.