1 / 17

TODAY’S JOURNEY

Nadia Brits Supervisor: Prof. Deon Meiring ACSG Conference 16 March 2011 EVOLUTION OF THE CONSTRUCT-VALIDITY DEBATE Department of Industrial Psychology University of Stellenbosch. TODAY’S JOURNEY. SACKETT & DREHER (1982). CONSEQUENCES OF INVALID CONSTRUCTS. DESIGN FIXES.

andie
Download Presentation

TODAY’S JOURNEY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nadia BritsSupervisor: Prof. Deon MeiringACSG Conference 16 March 2011EVOLUTION OF THE CONSTRUCT-VALIDITY DEBATEDepartment of Industrial PsychologyUniversity of Stellenbosch

  2. TODAY’S JOURNEY SACKETT & DREHER (1982) CONSEQUENCES OF INVALID CONSTRUCTS DESIGN FIXES DON’T TAKE AWAY MY DIMENSIONS ABANDON DIMENSIONS DESIGN FIXES: TRIED & TESTED EPIC OF THE CV DEBATE CRITICS OF LANCE (2008) ACs AT A CROSSROADS

  3. BACKGROUND • Sufficient evidence for CV exists (Arthur et al., 2000; Thornton & Gibbons, 2008) • The validity of ACs is questioned • CONSTRUCT VALIDITY PUZZLE (Lievens, Chasteen, Day & Christiansen, 2006) • CONSTRUCT-RELATED VALIDITY PARADOX (Arthur, Day & Woehr, 2008) • SO-CALLED CONSTRUCT VALIDITY PROBLEM (Howard, 1997)

  4. ORIGINS OF THE CONSTRUCT-VALIDITY DEBATE • Sackett & Dreher (1982) • Expectations: • CONVERGENT VALIDITY • DISCRIMINANT VALIDITY • ratings cluster according to DIMENSIONS, not exercises (which became known as EXERCISE EFFECT) • Similar research followed

  5. CONSEQUENCES OF INVALID CONSTRUCTS • SELECTION of applicants based on AC performance ratings • FEEDBACK based on AC results • WASTE OF TIME AND MONEY • Practitioners provide MISLEADING services to companies who appoint them to design and run ACs

  6. DESIGN FIXES • DEFINITION of dimensions • NUMBER of dimensions • TRANSPARENCY of dimensions to candidates • Behavioural CHECKLISTS • Type of SCORING METHOD • ASSESSOR training • TYPE of assessor (Lievens & Klimoski, 2001; Gaugler et al., 1987) • EXPERIENCED assessors (Kolk et al. 2002; Thornton & Rupp, 2005).

  7. DESIGN FIXES: TRIED AND TESTED • Design fixes show LITTLE, INSIGNIFICANT IMPROVEMENT • Controlling assessor variance has only a MARGINAL EFFECTS on construct-validity • EXERCISE EFFECTS STILL DOMINATE • What else can we do?

  8. ABANDONING DIMENSIONS • Move from dimensions-based ACs to task-based AC • Why? • Recurring exercise effects • Exercise factors show positive correlations with external performance criteria

  9. DON’T TAKE AWAY MY DIMENSIONS • useless to learn a task that the participant might never encounter again • Human performance is multidimensional • cannot capture the full complexity of a real job (novel, non-repetitive tasks) • Only dimensions will allow generalisation of AC results • Research supports dimensions (Connelly et al., 2008; Melchers & Konig, 2008; Bowler & Woehr, 2006) • knowledge about exercise-based ACs still lacking(Lievens, 2008).

  10. RESULT : 3 LINES OF RESEARCH • EXERCISES and DIMENSIONS explained same amount of variance – 34%(Lievens & Conway, 2001) • EXERCISE EFFECT were larger than dimension effects – 52%(Lance et al., 2004) • EXERCISES explained most variance – 33%, DIMENSIONS ALSO explained substantial amount - 22% (Bowler & Woehr, 2006)

  11. EPIC OF THE HEATED DEBATE • LANCE (2008): “ACs do not work they way they are supposed to” • NOT SUFFICIENT EVIDENCE for 3 requirements of CV • Supports the EXERCISE-BASED MODEL • DISCARD DESIGN FIXES • Exercise effects represent cross-situational specificity in candidate performance, not method bias...TRUE VARIANCE

  12. COMMENTS/CRITICS TOWARDS LANCE (2008) • ignores evidence of both dimension and exercises account for variance in AC performance (Howard, 2008) • design fixes should continue to be investigated (Schleicher et al., 2008; Howard, 2008; Arthur, Day & Woehr, 2008; Melchers & König, 2008) • Lance received support for candidates’ inconsistent performance....true variance • individuals use set of stable skills and can adjust and adapt KSA’s according to the situation • Some people perform better than others in a specific exercise

  13. ACs AT A CROSSROADS • Persistent EXERCISE EFFECTS • performance variability: more situation-specific (57%) than situation-consistent (43%) (Hoeft and Schuler’s (2001) • Lack of consensus on solutions and future directions • Walter Mischel (1968) • Consistency in behaviour ONLY when situational factors are acknowledged and taken into account • TRAIT-ACTIVATION THEORY (TAT)

  14. TRAIT-ACTIVATION THEORY (TAT) • person-situation interaction to explain behaviour on the basis of responses to trait-relevant cues found in situations. • SITUATION STRENGTH: strong vs weak situations • SITUATION RELEVENCE: A situation is considered relevant to a trait if it provides cues for the expression of trait-relevant behaviour

  15. APPROACHES TO IMPLEMENT TAT • Adapting the CONTENT of the exercise • Influencing the INSTRUCTIONS of each exercise to guide participants what type of behaviour to show • training ROLEPLAYERS on how to interact in order to elicit certain behaviours from participants • LARGE NUMBER OF SHORTER EXERCISES to obtains samples of performance on a number of independent tasks

  16.  Halaand and Christiansen (2002) found stronger convergence of AC ratings • GAP IN RESEARCH about effectiveness of TAT

  17. TO SUMMARISE SACKETT & DREHER (1982) CONSEQUENCES OF INVALID CONSTRUCTS DESIGN FIXES DON’T TAKE AWAY MY DIMENSIONS ABANDON DIMENSIONS DESIGN FIXES: TRIED & TESTED EPIC OF THE CV DEBATE CRITICS OF LANCE (2008) ACs AT A CROSSROADS

More Related