PSI Division of Work and Occupational Psychology CPD Seminar February 2008 Recruitment and Selection: Implementing

1.  PSI Division of Work and Occupational Psychology CPD Seminar � February 2008 Recruitment and Selection: Implementing New Approaches �ine Gray Public Appointments Service

3. Overview Realistic Job Simulations Situational Judgment exercises Video based tests Internet Based Testing Case Study: Using the Internet to assess people with disabilities

4. Realistic Job Simulations Growing body of research evidence that realistic job simulations are effective in measuring job relevant skills Also known as situational tests: Anastasi and Urbina (1997) define a situational test as �one that places the test taker in a situation closely resembling or simulating a �real-life� criterion situation� Generally fairer than traditional cognitive tests, except where the job has a heavy cognitive load Gives candidates a realistic job preview Credible with candidates and the media

8. What is a Situational Judgment Exercise (SJE)? Also known as Situational Judgement Test, Job Simulation or Situational Judgement Inventory First used in 1920�s Present candidates with job related situations/ dilemma Adopt a multiple choice format May be bespoke for a particular role or off-the-shelf for particular grade e.g. managerial role

9. Experience of Public Appointments Service Situational Judgement Exercises used since mid 1990�s Used for a range of roles including entry level posts (Garda Trainee, Executive Officer, Junior Diplomat) and middle to Senior Management Positions (Assistant Principal & recently introduced at Principal Officer level) Used as part of a competency based approach to recruitment to assess a range of competencies Particularly valuable because of gender performance Almost all SJE�s are bespoke for the role / grade and set in a Civil Service Context Designed in-house with job incumbents, line mangers and other SME�s, trialling and item level analysis of test performance

10. You are on duty alone in a housing complex which has a reputation as a trouble spot. While you are on patrol, a youth who is one of a group throws an empty mineral can which hits you on the head. You are not physically injured, but are annoyed. The group jeers at you in a threatening way. What do you do? A Notify the Station by radio that a situation is developing and then approach the group yourself to apprehend the can-thrower. B Approach the group and warn them that they will be arrested if they continue being abusive. C Choose not to respond to this incident as you are not injured by the can and continue on your beat. D Approach the youths and calm the situation down by asking them to move on. Decide to deal with the issue of the can-throwing later on when the situation has calmed down. E Make a mental note of who threw the can, radio for assistance and wait for back-up.

11. What does an SJE measure? Job Knowledge

12. What does an SJE measure? Contd. Best viewed as a measurement method and not a measure of a single construct (O� Connell et al 2007) � outputs may be a single score or range of scores across constructs/ competencies Can assess �the integration of cognitive and affective based behaviours� Lewis et al 2006, cited in Creighton & Scott 2006.

14. Considerations in the use of SJE�s Benefits; Effective assessment methods Can balance out the potential adverse impact of cognitive tests Flexibility; Can measure a range of required constructs High levels of face validity for candidates Can be administered efficiently to large numbers of candidates May be designed to replicate specialist jobs effectively � useful realistic job preview

15. Considerations in the use of SJE�s Challenges; Challenges with Candidate feedback Significant investment required to devise exercises Balance needed when using exercises on a confined and open candidate pool In confined competitions, a neutral context is needed rather than one favouring candidates from policy/ operational areas Research suggests faking: Is easier when items are more transparent Is more difficult when there is a heavier cognitive load Is more likely with behavioural instructions � when asked what would you do, people will respond with what they feel is best answer

19. Recent developments � Emergency Medical Controller Emergency Medical Controllers take emergency medical calls and dispatch assistance Job Analysis demonstrated requirement for attention to detail and very specific interpersonal skills when dealing with callers and other health care professionals Sourced video based assessment; Call taker test; replicated a call-centre. Candidates required to take notes, and after an excerpt from the phone calls, choose which of a series of options the call taker should follow Dispatch test � candidates had to dispatch emergency vehicles according to a set of priorities

21. Validity Sourced from IPMA HR Because of US legal context, very good validity evidence e.g: Correlation between call-taker test and academy performance was .30 Correlation with supervisors ratings was .20 Correlation with overall pass/fail on probation was .28 Number in sample: 627

22. Case Study � EO Disabilities Campaign 2007 Stage 1 On-line testing Verbal Reasoning test Skills and Experiences Questionnaire Equality Monitoring Form Stage 2 Paper and Pencil testing Verbal Reasoning test Numerical Reasoning test Job Simulation Exercise Stage 3 Competency-based interview and Work-sample test

23. Why Internet based testing?  Evolution from supervised computer based testing Convenient for candidates Applicant population generally familiar with technology More efficient than face-to-face testing

25. Case Study � EO Disabilities Campaign 2007 No. Applied 575 Sat Stage 1 on-line 311 Sat Stage 1 off-line 81 Passed Stage 1 281 Sat Stage 2 250 Passed Stage 2 124 Stage 3 October 2007 People currently being assigned to jobs

26. Stage 1 � Feedback survey from candidates Telephone survey conducted with over 100 candidates � 45 responded 84% felt that doing the tests on line is a good idea for people with disabilities 78% preferred to do the tests on-line vs in a supervised environment �more relaxed on-line� �the test was much easier on a PC� 90% found it convenient to do the tests in their own time 100% of candidates use computers everyday

27. References Creighton, P & Scott, N. (2006). An Introduction to Situational Judgement Inventories. Selection and Development Review 22, 3-6. O Connell et al (2007). Incremental Validity of Situational Judgement Tests for Task and Contextual Job Performance. International Journal of Selection and Assessment. 15, 19-29. McDaniel et al (2007) Situational Judgement Tests, Response Instructions and Validity: A Meta-analysis. Personnel Psychology 60, 63-91. Lievens, F., Peeters, H & Schollaert, E. (In Press) Situational Judgment Tests: A Review of Recent Research. Personnel Review

29. Thank You!