2007 National Human Services Training Evaluation Symposium

1. 2007 National Human Services Training Evaluation Symposium Fasih Ahmed, Ph.D.

2. Reaction to Training: A Few Basic Questions about theNature of the Construct • Is reaction to training a multi-dimensional construct with distinct components or a generalized holistic perception of training experience? • If reaction to training is a multi-dimensional concept, how the various dimensions relate to each other and how they collectively formulate the overall reaction to training? • Are the various dimensions identifiable (and measurable) as separate ‘valid’ constructs with sufficient specificity for application in evaluative analysis of training programs?

3. Some Possible Approaches and Assumptions • Most trainee satisfaction surveys assume reaction to training as a multi-faceted concept and measure satisfaction/dissatisfaction with such aspects of training as the curriculum, the trainer, and training delivery arrangements/facilities, etc. • Statistical techniques can be used to delineate and examine specific dimension of reaction to training. These can range from such sophisticated techniques as factor and logistic analyses to simple cross tabulations, depending on the level of measurement and tenability of assumptions about the underlying distributions of the specific variables being measured. • The purpose of this analysis is to determine the nature of relationship between the various dimensions of reaction, and how they individually relate to the overall reaction to training experience. • Question: are individual dimensions independent of each other, or related in some causal model, or have interactive effects on each other? • Question: What is the nature of relationships between individual dimensions with the overall reaction to training?

4. Methodology & Data: The Case of North Carolina • NC uses the Participant Satisfaction Form (PSF) to survey reaction to training and its components: curriculum (five questions); trainer(s) (six questions); training delivery arrangements (three questions), in addition to a few attitudinal questions and a question on satisfaction with the overall training experience. • Almost all questions have four response categories (strongly agree, agree, disagree, strongly disagree) which. when combined, yield dichotomous variables with two outcomes: satisfied-dissatisfied. • Our focus of this analysis are only two components: reaction to curriculum and reaction to trainer. This analysis is based on data collected over three years: 2004,2005, and 2006. Data for each year is analyzed separately to discern any patterns. • In order to aggregate data, a mean score for each component is computed from the cluster of questions pertaining to that component, which yields an interval scale continuum of satisfaction-dissatisfaction. Distribution of mean scores for reaction to trainer and reaction to curriculum are shown below.

8. Limitations of Analysis • Given the extremely skewed distribution of mean scores, our statistical analysis will have to be limited to non-parametric approaches, which do not depend on assumptions of normal, or approximately normal, distributions. • Mindful of the fact that the responses were originally measured as categorical variables, we have transformed the mean scores into two categories: satisfied and dissatisfied, for each of the component. • Since the scores of 3 and 4 in the original measurement indicated satisfaction, all mean scores of less than 3 were categorized as indicative of dissatisfaction, which typically includes the bottom 10% to 27% of all mean scores. • Given the need to aggregate the data as categorical variables, cross tabulation is the statistical techniques we have used for the following analysis. • Simple distribution of satisfaction/dissatisfaction with curriculum and trainer for the three years under consideration are shown below.

10. Are Reactions to Curriculum and Trainer Independent Dimensions? 2004 2005 2006

11. Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2004 • Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV. • A. Condition of dissatisfaction with trainer: • 44.5% of those dissatisfied with curriculum are also dissatisfied with trainer. • 7.0 % of those satisfied with curriculum are dissatisfied with trainer. • Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with curriculum. % difference: 44.5-7.0 = 37.5. • B. Condition of satisfaction with trainer: • 55.5 % of those dissatisfied with curriculum are satisfied with trainer. • 93% of those satisfied with curriculum are also satisfied with trainer. • Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. % difference: 93.0-55.5 = 37.5. • Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV. • C. Condition of dissatisfaction with curriculum: • 70.4% of those dissatisfied with trainer are dissatisfied with curriculum. • 18.3% of those satisfied with trainer are dissatisfied with curriculum. • Indication: dissatisfaction with curriculum is more likely to be related to dissatisfaction with trainer. % difference: 70.4-18.3 = 52.10. • D. Condition of satisfaction with curriculum: • 29.6% of those dissatisfied with trainer are satisfied with curriculum. • 81.7 of those satisfied with trainer are satisfied with curriculum. • Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. % difference: 81.7-29.6 = 52.10. • Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more tenable.

12. Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2005 • Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV. • A. Condition of dissatisfaction with trainer: • 43.7% of those dissatisfied with curriculum are also dissatisfied with trainer. • 9.8 % of those satisfied with curriculum are dissatisfied with trainer. • Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with curriculum. % difference: 43.7-9.8 = 33..9. • B. Condition of satisfaction with trainer: • 56.3 % of those dissatisfied with curriculum are satisfied with trainer. • 90.2% of those satisfied with curriculum are also satisfied with trainer. • Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. % difference: 90.2-56.3 = 33.9. • Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV. • C. Condition of dissatisfaction with curriculum: • 33.7% of those dissatisfied with trainer are dissatisfied with curriculum. • 6.6% of those satisfied with trainer are dissatisfied with curriculum. • Indication: dissatisfaction with curriculum is not more likely to be related to dissatisfaction with trainer. % difference: 33.7-6.6 = 27.1. • D. Condition of satisfaction with curriculum: • 66.3% of those dissatisfied with trainer are satisfied with curriculum. • 93.4% of those satisfied with trainer are satisfied with curriculum. • Indication: satisfaction with curriculum is not more likely to be related to satisfaction with trainer. % difference: 93.4-66.3 = 27.1. • Tentative Conclusion: Comparing the % differences under the two assumptions, Assumption II seems more plausible, but only slightly so.

13. Nature of Relationship Between Reaction to Curriculum and Reaction to Trainer2006 Assumption I: Reaction to curriculum is the IV and reaction to trainer is the DV. • Condition of dissatisfaction with trainer: 42.0% of those dissatisfied with curriculum are also dissatisfied with trainer. 7.5 % of those satisfied with curriculum are dissatisfied with trainer. Indication: dissatisfaction with trainer is more likely to be related to dissatisfaction with curriculum. % difference: 42.0-7.5 = 34.5. B. Condition of satisfaction with trainer: 58.0 % of those dissatisfied with curriculum are satisfied with trainer. 92.5% of those satisfied with curriculum are also satisfied with trainer. Indication: satisfaction with trainer is more likely to be related to satisfaction with curriculum. % difference: 92.5-58.0 = 34.5. Assumption II: Reaction to trainer is the IV and reaction to curriculum is DV. C. Condition of dissatisfaction with curriculum: 40.3% of those dissatisfied with trainer are dissatisfied with curriculum. 7% of those satisfied with trainer are dissatisfied with curriculum. Indication: dissatisfaction with curriculum is more likely to be re lated to dissatisfaction with trainer. % difference: 40.3-7.0 = 33.3. D. Condition of satisfaction with curriculum: 59.8% of those dissatisfied with trainer are satisfied with curriculum. 93.0% of those satisfied with trainer are satisfied with curriculum. Indication: satisfaction with curriculum is more likely to be related to satisfaction with trainer. % difference: 93.0-59.8 = 33.2. Tentative Conclusion: Comparing the % differences under the two assumptions, both Assumption I and Assumption II seem equally tenable.

14. Interaction Between Overall Training Experience and Reaction to Trainer 2004 Interaction Between Overall Training Experience and Reaction to Curriculum 2004 • PSF Data for 2004 • There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum. • The effect of overall experience on reaction to trainer is a little stronger than that on reaction to curriculum, suggesting an interactive effect between overall experience, reaction to curriculum, and reaction to trainer. • One should, however, be mindful that perception of overall experience is based on only one question.

15. Interaction Between Overall Training Experience and Reaction to Trainer 2005 Interaction Between Overall Training Experience and Reaction to Curriculum 2005 • PSF Data for 2005 • There is positive association between overall experience of training and reaction to trainer as well as reaction to curriculum. • The effect of overall experience on satisfaction with trainer is slightly more than the other way round. • The effect of reaction to curriculum on perception of overall experience is more than effect of overall experience on curriculum. • These pattern suggest an interactive effect between overall experience, satisfaction with trainer, and reaction to curriculum.

16. Interaction Between Overall Training Experience and Reaction to Trainer 2006 Interaction Between Overall Training Experience and Reaction to Curriculum 2006 • PSF Data for 2006 • There is significant association between overall experience of training and reaction to trainer, the effect of overall experience on reaction to trainer being slightly more than the converse. • There is a significant positive effect of overall experience of training on reaction to curriculum , the effect of overall experience on reaction to curriculum being slightly more than the converse. • The pattern of associations suggests interactive relationships between the overall experience of training, reaction to curriculum and reaction to trainer.

17. Inconclusive Conclusions The three years data does not yield any consistent patterns to allow definitive answers to the questions raised initially. However, some of the interesting, though unexpected findings are: • Given the patterns of relationships between reaction to curriculum, reaction to trainer, and the overall experience with training, there is no clear evidence that specific dimensions of reaction to training are independent components of a theoretical construct. • When examining relationships between reaction to curriculum and reaction to trainers under different assumptions of a causal model, there only indications, not evidence, that satisfaction with trainer tends to play more important role in determining reaction to curriculum, relative to perception of overall experience of training. • Although analysis of reaction to training may serve some limited evaluative purpose, its utilization as a decision-making tool for management of training warrants due caution.