Decision making: Utility and assessing return on investment

1. Decision making: Utility and assessing return on investment • Efficiency of linear models • Alternative prediction models: multiple regression, multiple cut offs, multiple hurdle approaches • Classical validity approach: selection ratio, base rate • Decision-making accuracy • Utility models: Taylor-Russell, Naylor-Shine, Brogden-Cronbach-Gleser models • Alternative methods of estimating SDy in Rands: the 40% rule, Schmidt-Hunter Global Estimation, Cascio-Ramos Procedure • Integration of selection utility with capital budgeting models

2. Classical validity approach • Primary emphasis on measurement accuracy (reliability) and predictive efficiency (predictive validity) – correlation between predictor and criterion; emphasis on psychometric criteria • Goal of selection process is to capitalise on individual differences and to select those individuals who possess the greatest amount/quantity of the particular characteristics judged important for job success • Simple regression: predicting each individual’s criterion status based on predictor information • Multiple regression – compensatory as low scores on one predictor offset by high scores on another

3. Efficiency of linear models in prediction • General linear model: y = a + bx • Weights (b) derived by least squares regression procedure, subjective/intuitive, unit weights • These models appropriate and accurate for a wide range of situations

4. Moderator variables • Differential prediction exists: the correlation between predictor (assessment/test) and criterion varies as a function of classification on a third variable (gender, race, education, age) • Problems: (i) utility rarely assessed – cross validation - show that selecting based on moderator combined with assessment score superior to assessment scores alone (ii) subgroups often small – statistical power not enough to detect significant differences (iii) meta- analyses – sample sizes – Type I and II errors (iv) sub-grouping/-clustering – similar profiles enhances understanding of kinds of individuals for whom specific assessment predicts accurately

5. Suppressor variables • Lack of correlation with criterion but high inter-correlation with one/more of the predictor variables – increases multiple R but not prediction • Model of multiple regression assumes that each predictor variable correlates with criterion but not with each other, thus predicts unique part of criterion, contributes to multiple R • In practice suppressor variables found in complex model – particularly when using aggregate data

6. Criticism of classical validity approach • Largely ignores external parameters (minimum requirements, costs) of the situation that largely determine the overall worth of the selection instrument • Makes unwarranted utility assumptions • Fails to consider the systemic nature of the selection process (linear, mechanical)

7. Alternative prediction models • Although multiple regression basis of traditional prediction approach, it has to be compared with alternative – depending on chosen strategy, different employment decisions may result • Consider advantages/disadvantages of multiple regression and compare with multiple cut-off and multiple hurdle approaches

8. Multiple regression approach • Y’ = a + b1X1 + b2X2,+ … + bkXk • Assumptions of linearity, trait additivity • Values of predictors (X1, X2, X3,…Xk) vary across individuals, regression weights (b1,b2,…bk) constant for sample • Thus possible for individuals with different predictor scores to obtain identical predicted criterion scores – compensatory model (high scores on highly weighted predictor can compensate for low scores on predictors with lower weights

9. Advantages of the multiple regression model • If assumptions are met and sample is large: errors in prediction minimised; predictors are combined optimally, to yield most efficient estimate of criterion status • Model flexible: mathematically can be modified to handle nominal data and non-linear relationships, linear and non-linear interactions; equations for number of jobs generated using same predictors, different weights, or different predictors

10. Multiple cut-off approach • In some selections proficiency in one predictor cannot make up for deficiency in another (e.g. visual acuity drivers, pilots) • Minimal level of proficiency in one/more variables essential for job success - no substitutes allowed, failure on these predictors disqualifies applicant completely • As this approach non-compensatory, assumes curvilinearity in predictor-criterion relationships (above cut-off, higher predictor score not necessarily predicts higher criterion score)

11. Multiple cut-off approach … • Curvilinear relationships can be accommodated in regression approach, in practice multiple regression and multiple cut-off approaches may lead to different decisions (e.g. Cascio (1991), p. 287) • No satisfactory solution to finding cut-off scores • If organisation knows number of posts available and number of applicants, can determine hiring rate

12. Multiple Regression and Multiple Cutoff Model X1 cutoff A D R B X2 cutoff Predictor X2 R C R Multiple regression cutoff Predictor X1

13. Setting cut-off scores • No single best way of setting cut-off scores for all situations • Begin with job analysis that identify critical competencies and relative levels of proficiency • Validity and job-relatedness of assessment critical • Where possible data on actual relation between assessment and criterion considered carefully • Cut-off scores high enough to meet minimum standards of job performance • Cut-off scores consistent with normal expectations of proficiency within the work force

14. Expectancy tables • Depicts likelihood of successful criterion performance to be expected for any given level of predictor scores for an organisation (Cascio (1991), Figure 13.5, p. 288) • Also for individuals • Expectancy charts useful way of illustrating the effect of the validity coefficient on future hiring decisions • Optimal: initially use multiple cut-off scores to identify individuals who meet minimum requirements, then multiple regression with remaining predictors

15. Organisational expectancy chart

16. Multiple hurdle approach • Multiple regression and multiple cut-off models are single stage (not sequential) decision strategies • In multiple hurdle (sequential) cut-off scores on predictors used to make investigatory decisions • Cut-off scores on assessments or composite multiple regression used to make decisions • Applicants provisionally accepted and assessed further • Several staged approach appropriate when subsequent training is long, complex and expensive

17. Multiple hurdle approach … • In best interest of individual and organisation to reach a decision as quickly as possible • Essential decisions as accurately as possible given available information • Forecasting accuracy increases as candidate clear the various hurdles but so do costs • As candidates are eliminated range restricted and validity underestimated

18. Evaluating selection efficiency • Comparison of decision based on assessment to choosing candidates at random • Index should indicate proportion of saving by using assessment to predict success