Training Evaluation- Design Issues

Training Evaluation- Design Issues • Basic Designs • Post-test only • Pre-rest/Post-test • Complex Designs • Post-test with control group • Pre-test/Post-test with control group • Time Series • Solomon 4 group

Basic Designs • Post-test Only • occurs when training is provided and then a post-test is given X T2 X – training T2 – post-test evaluation

Basic Designs - Post-test Only (cont’d) • Goals of evaluation: • to determine if change has taken place • to determine if a level of competence has been reached • Post-test only design is sufficient if the evaluation is for the second goal • e.g. a training which is developed to equip employees with certain legal requirements

Basic Designs - Post-test Only (cont’d) • Post-testing can be turned into a pre-test/post-test design by utilizing data from the following sources: • needs analysis • archival data, e.g. • performance appraisal • measures of quality • An equivalent group could also be chosen and given the same post-test and the evaluation becomes post-test only with control group

Basic Designs - Post-test Only (cont’d) • Post-test only design is problematic for assessing change because change could be caused by a number of other competing courses

Basic Designs- Pre-test/Post-test • The design can be represented as: T1 X T2 T1 – Pre-test X – Training T2 – Post-test • Main criticism is that it is without a control group – difficult to determine the impact of training on the changes that occur

Basic Designs- Pre-test/Post-test (cont’d) • A method of dealing with the lack of a control group is known as Internal Referencing Strategy (IRS) • IRS includes using relevant and non-relevant test questions in the pre- and post-test • relevant : questions that deal with training contents • non-relevant : questions with aspects not trained (these questions serve as control)

Complex Designs • Two factors to consider in developing a sound evaluation design: • control group: a group of similar employees who do not receive the training • random assignment: the assignment of employees to either the control group or training group by chance

Complex Designs – Post-test Only with Control Group • This design can be represented by the following Trainee Group X T2 Control Group T2 • Differences in test scores between the groups will provide evidence of the effectiveness of the training • Weakness: there is no pre-test score to indicate equivalence of the groups before training • Representative sampling: matching of employees on specific variables e.g. age, educational background, experience etc. may reduce the weakness

Complex Designs- Pre-test/Post-test with Control Group • It is represented by the following expression: Trainee Group T1 X T2 Control Group T1 T2 • Provides indication of equivalence between the two groups (through T1) • Post-test differences will provide evidence for training effectiveness

Complex Designs- Time Series Design • The expression for this design is: Trainee Group T1 T2 T3 T4 X T5 T6 T7 T8 • It uses a series of measurements before and after training • The design can be made more powerful by using a control group Trainee Group T1 T2 T3 T4 X T5 T6 T7 T8 Control Group T1 T2 T3 T4 T5 T6 T7 T8

Complex Designs- Multiple Baseline Design • Expression: Trainee Group A T1 T2 T3 X T4 T5 T6 T7 T8 T9 T10 .. Trainee Group B T1 T2 T3 T4 T5 X T6 T7 T8 T9 T10 .. Trainee Group C T1 T2 T3 T4 T5 T6 T7 X T8 T9 T10 .. Trainee Group D T1 T2 T3 T4 T5 T6 T7 T8 T9 X T10 .. • Multiple measures are taken (as in time series) but each group is trained at different times • If each group improves after receiving training, it can be said that the changes are due to training

Complex Designs- Solomon 4 Group Design • Expression: Group 1 T1 X T2 Group 2 T3 T4 Group 3 X T5 Group 4 T6 • Members of the groups should be based on representative sampling • If T2 > T1 and T4, and if T5 > T6, and if T5 > T3 then inference about the effects of training can be made with confidence • If T6 = T1 and T3 (history and maturation are not the cause of improvement) • If T2 and T5 are equal (both higher than T1) then the reactive effect of testing is not the cause

Evaluation– The Choice of Design • The true impacts of training depends on the validity of evaluation results • The more complex the design, the more valid the results • When multiple measures are not possible, the following design may be useful: Trainee Group A T1 X T2 Trainee Group B T1 X T2 Trainee Group C T1 X T2 Trainee Group D T1 X T2

Training Evaluation- Design Issues