Effectiveness of Coaching on a Web-based In-basket Exercise

1. Effectiveness of Coaching on a Web-based In-basket Exercise Frederik Anseel & Filip Lievens Ghent University, Belgium [frederik.anseel@ugent.be]

3. Can performance on web-based tests be enhanced through online coaching ?

4. Web-based testing • Internet offers advantages for testing (Lievens & Harris, 2003) • Test administration and scoring efficiencies • Large savings in costs and turnaround time • Diverse locations and different times • Research on web-based testing (Lievens & Harris, 2003) • Measurement equivalence and psychometric properties • Applicant’s perceptions and reactions • No attention to coaching effects • Threat to predictive validity, fairness and utility of web-based tests • Concern to practitioners • Online tests might be copied or shared • People might cheat by using online reference materials or help form others

5. Coaching • Different types of coaching • Practice on sample items vs. intensive instruction (Messick & Jungleblut, 1981) • Practice, tutoring, tips, explanation of constructs, modeling, and feedback (Maurer, Salomon, & Troxtel, 1998) Coaching programs = blend of tactics • Cognitive measures (Sackett, Burris, & Ryan, 1989) • Mostly educational settings (SAT) • Small but significant increases in test performance • Noncognitive measures (Sackett, Burris, & Ryan, 1989) • Little research • Assessment centers, situational interviews, integrity testing • Small but significant increases in performance (mixed findings)

6. Current study • Effectiveness of • practice and coaching • on a web-basedin-basketexercise • in a proctored and an unproctored setting

7. Samples • Sample A • Proctored setting • 274 master students • Training program in basic managerial competencies • 37% male, 63% female • Mean age = 22.7 yrs (SD = 2.3) • Sample B • Unproctored setting • 206 participants • Career assessment tool available through governmental organisation • 61% male, 39% female • Mean age = 40.7 yrs (SD = 10.5) • Working experience = 13.6 yrs (SD = 10.7); 51 % university degree

8. In-basket Form 1 Control In-basket Form 2 Performance 2 Coaching Performance 1 Procedure • Pre-post control group design (random) • Pre-post differences in control group reflect a practice effect • Web-based in-basket • Adapted from Tett, Steele, and Beauregard (2003) • 4 managerial competencies • 10 emails with four response options • Parallel form: Item-cloning procedure (Clause et al., 1998) • Content, grammatical structure, option structure

9. Procedure • Coaching Manipulation • Feedback on 4 competencies • Specific test-taking strategies (Tett, Menard, Guterman, & Beauregard, 2001) • Manipulation check (Student Sample) • 5 items (α = .84) • “After the first part, I knew to which emails I had responded correctly.” • “I knew how to improve my scores in the second part of the in-basket.” • MCont = 3.58 (SD = 1.16) vs. MCoach= 4.49 (SD = 1.20), p < .001 • Coordinating • Indicates that you organize the activities of subordinates and the allocation of resources • Your Score : 16 / 20 • Tip: Experts schedule appointments and meetings to promote the productive use of time. They try to integrate multiple tasks and the people necessary to perform them. It is important to establish efficient work routines among subordinates as a group.

11. Results Sample A (student) Table 1: Mean and Standard Deviations for Control and Coaching condition F(1, 272)=81.10, p < 0.001, η² = .23 • No difference across forms in control condition • No practice effect • Significant increase in performance on form 2 in coaching condition • Coaching effect

12. Results Sample B Table 2: Mean and Standard Deviations for Control and Coaching condition F(1, 204)=32.25, p < 0.001, η² = .13 • Significant decrease in performance in control condition (p < .001) • No practice effect • Significant increase in performance in coaching condition (p < .001) • Coaching effect

13. Discussion • Practice on the in-basket does not improve performance • No change in performance in proctored setting • Decrease in performance in unproctored setting • Online coaching improves performance • In both proctored and unproctored setting • Consistent with previous coaching research • Educational literature • Noncognitive measures • In-basket (Brannick et al., 1989; Brostoff and Meyer, 1984; Gill, 1982)

14. Implications • Strategy 1 : Increased security awareness • Protect tests: supervision + new technologies • Use tests with high g-loadings • Construct new and parallel versions of tests • Future research: Does coaching for one web-based test enhance performance on a different web-based test ? • Strategy 2: Embrace coaching effects • Applicant’s point of view • New possibilities for career assessment purposes • Instant feedback and coaching is one of the main advantages of web-based testing Web-based testing is here to stay

15. Good or bad news ? • We don’t know (a) Coaching increases scores without increasing an applicant’s standing on the construct (b) Coaching increases scores by eliminating error variance due to lack of test experience or test anxiety (c) Coaching actually improves an applicant’s standing on the characteristic • Future research: How does coaching affect validity of web-based tests ?

17. Different competencies (student)

18. Different competencies (employee)