MONITOR AND EVALUATE THE EFFECTIVENESS OF TRAINING OUTCOMES

1. D1.HRD.CL9.06 D1.HHR.CL8.07 D2.TRD.CL8.09 MONITOR AND EVALUATE THE EFFECTIVENESS OF TRAINING OUTCOMES

2. 1. Determine the workplace training outcomes to be monitored and evaluated 1.5 Schedule the workplace monitoring and evaluation process Obtain relevant planning documentation relating to workplace training provision 1.1 Clarify expected workplace training outcomes with relevant personnel 1.2 Determine specific workplace training outcomes identified by the organisation 1.3 Determine the specific workplace training outcomes to be monitored and evaluated 1.4

3. 1.1 Obtain relevant planning documentation relating to workplace training provision Relevant documentation could include: • Business plans • Strategic plans • Specific training-related plans • Curriculum documentation • Competency standards and/or other relevant training criteria • Relevant organisation policies and procedures • Meeting minutes from planning meetings

4. 1.2 Clarify expected workplace training outcomes with relevant personnel What type of training is delivered at your workplace? What kind of people are involved with training at your organisation?

5. Relevant personnel connected to workplace training activities • Trainers and assessors • Administration staff and support staff • Specialist staff and subject/skills experts • Consultants • Supervisors and managers • Human resource staff • External training providers and representatives involved in the training delivery and/or assessment • Co-workers who have worked with staff who have received training • Current and past trainees • Other organisational stakeholders such as funding body representatives

6. Determine specific workplace training outcomes identified by the organisation In any training activity we are trying to bring about change in three main areas. Trainers refer to these as the: • Psychomotor • Cognitive • Affective areas

7. Determine specific workplace training outcomes identified by the organisation cont’d

8. 1.4 Determine the specific workplace training outcomes to be monitored and evaluated Determining training outcomes to be monitored and evaluated requires a decision between: • Monitoring and evaluating all training outcomes at one time or • Monitoring and evaluating different aspects of the process over time or as part of a cyclical process

9. 1.4 Determine the specific workplace training outcomes to be monitored and evaluated Determining which specific training outcomes to monitor and evaluate may also be dependent upon several factors such as: • Monitoring and evaluation plans • Stakeholder requests • Feedback relating to specific training outcomes • Availability of resources, time and budget • Access to people with required skills for conducting the review • Business performance issues or results • Complaints

10. 1.5 Schedule the workplace monitoring and evaluation process • Monitoring is the process through which the organisation ensures that actual training activities conform to the planned and intended ones • Evaluation is the systematic process of collecting and analysing information in order to determine whether and to what degree the objectives of a program or project have been, or are being achieved

11. 1.5 Schedule the workplace monitoring and evaluation process cont’d The following questions should be answered in planning for evaluation and monitoring scheduling: • Who will conduct monitoring and evaluation? • What are the criteria for monitoring and evaluation? • How will the information be collected? • How will the information be utilised for the effective implementation of programs? • How will the information be shared? • What effects will the monitoring and evaluation activities have on other operational activities? • When should the information and data collected be forwarded to appropriate people?

12. 2. Monitor the effectiveness of training outcomes 2.1 Review data captured relating to outcomes of training 2.2 Meet with relevant personnel to discuss captured training data Ensure training data being captured enables effective evaluation to occur 2.3 2.4 Generate independent data relating to training outcomes 2.5 Compare independent training data with captured training data 2.6 Revise the data capturing process and records, as appropriate

13. 2.1 Review data captured relating to outcomes of the training Data relating to training outcomes could be captured from many different areas within the organisation. It is important to review the data to: • ensure captured data reflects identified data needs • ensure correct documentation is used to record outcomes of training • ensure personnel responsible for capturing training data understand their role • ensure relevant documentation is being recorded as anticipated • check that records are stored correctly • make objective judgments relating to the validity of the documentation • undertake spot checks to determine the authenticity of the records being maintained

14. 2.2 Meet with relevant personnel to discuss captured training data • Meeting with relevant personnel to discuss the gathered data is an important part of the monitoring and evaluating process. • It provides an opportunity for people involved in the workplace training process an opportunity to clarify data, ask questions, offer their own feedback or interpretation, and it allows relevant personnel the chance to make suggestionsand recommendations for improvement.

15. 2.3 Ensure training being captured enables effective evaluation to occur

16. 2.3 Ensure training being captured enables effective evaluation to occur

17. 2.4 Generate independent data relating to training outcomes Sometimes the information that is being collected during training activities may not be sufficient or relevant to check if the training outcomes are being achieved. If, after checking draft evaluation data, there seems to be a gap in relevant data related to the evaluation, then supplementary data gathering materials may need to be developed and implemented.

18. 2.4 Generate independent data relating to training outcomes

19. 2.5 Compare independent training data with captured training data All training and assessment programs and activities collectdata before, during and after the training. The informationcould range from: • Participant personal details to results from assessment activities • Complaints, suggestions, and trainer feedbackinformation • Regular evaluations and scheduled audits of the training outcomes implemented at different times.

20. 2.5 Compare independent training data with captured training data cont’d There are many factors that will influence the validity and relevance of data that is collected and any monitoring and evaluation process must consider not only the data results but also: • How the data is documented • When it is captured • Who captures it • Does the data reflect evaluation and training outcome purposes • Does the independently captured data match the data captured by relevant personnel

21. 2.6 Revise the data capturing process and records, as appropriate Ensuring that captured data is relevant, sufficient, and useful for evaluation purposes, is a critical factor in deciding what to do next. Once the data has been analysed and checked against independent data, there may be a need to revise the data capturing process and records.

22. 2.6 Revise the data capturing process and records, as appropriate cont’d Some issues that may arise from the data capturing process could include: • Challenges in gathering the data: time, location, cost, compiling the data, not enough human resources, not enough data records available • Issues with the content: questionnaire questions, survey questions, too many questions or not enough, language challenges for understanding questions • Data doesn’t reflect evaluation questions or purpose

23. 3. Evaluate the effectiveness of training outcomes 3.1 Assess training outcomes against identified need 3.2 Determine the performance of the trainers and assessors Determine the effectiveness of the resources provided to support the implementation of training and assessment 3.3 Identify positives and negatives in relation to the attainment of identified specific workplace training outcomes 3.4 Identify positives and negatives in relation to issues arising as a result of the monitoring and evaluation process 3.5 Identify new topics and/or targets for future training and assessment requirements 3.6 Determine information that can be used to benefit future training and assessment decisions within the organisation 3.7

24. 3.1 Assess training outcomes against identified need It is important for the organisation to assess training outcomes against identified needs, in order to gauge whether the training activities are achieving the training outcomes that they were designed for.

25. 3.1 Assess training outcomes against identified need cont’d What is validation?

26. 3.1 Assess training outcomes against identified need cont’d

27. 3.2 Determine the performance of the trainers and assessors So what are we actually evaluating? Effectively this process looks at four main things:

28. 3.3 Determine the effectiveness of the resources provided to support the implementation of training and assessment Evaluating the effectiveness of the resources provided to support the implementation of training and assessment is a critical step in determining overall success of training outcomes.

29. 3.4 Identify positives and negatives in relation to the attainment of identified specific workplace training outcomes It is important to analyse the data and identify the positives and negatives in relation to the achievement of training goals as this information will provide the basis for identifying ways to improve the training and assessment activities within the organisation.

30. 3.5 Identify positives and negatives in relation to issues arising as a result of the monitoring and evaluation process Issues that may arise from the monitoring and evaluation process could include: • Issues with staff or participant communication • Costs - either positive or negative • Timing and flexibility issues • Discovering that trainers and assessors have more or less skills than required • Identifying changes in industry practice, technology requirements or equipment failure or obsolescence

31. 3.6 Identify new topics and/or targets for future training and assessment requirements

32. 3.7 Determine information that can be used to benefit future training and assessment decisions within the organisation With so much data collected and information analysed it is necessary to determine information that can be used to improve future training and assessment decisions within the organisation. Identifying the situations that enhance training and assessment activities and communicating this before future plans are made can assist in the development of a more appropriate and effective system.

33. 4. Prepare a report on the effectiveness of workplace training outcomes Develop a formal report on the effectiveness of workplace training outcomes 4.1 Make recommendations for change regarding workplace training and assessment provision 4.2 4.3 Distribute the report 4.4 Obtain feedback on the report

34. 4.1 Develop a formal report on the effectiveness of workplace training outcomes The basic elements of an evaluation plan include: • Title page • Executive summary • Question overview • Intended use and users • Program description • Evaluation focus • Methods • Analysis and interpretation plan • Use, distribution of results, and recommendations

35. 4.2 Make recommendations for change regarding workplace training and assessment provision Suggestions for change could include: • Whether to continue with the existing training and assessment system with minor alterations or to cease with the current system in favour of an external provider administering training and assessment • Making recommendations relating to staff resources, training, payment, costs, assessment content or administration procedures • Expanding the existing system or reducing the scope of the existing system • Recommending changes to location and timing of the training and assessment system • Changing course content and materials • Changes to support infrastructure • Changes to targeted trainees and participants

36. 4.3 Distribute the report Some questions to ask about the potential audience(s) are the following: • Who is a priority? • What do they already know about the topic? • What is critical for them to know? • Where do they prefer to receive their information? • What is their preferred format? • What language level is appropriate? • Within what time frame are evaluation updates and reports necessary?

37. 4.3 Distribute the report cont’d

38. 4.4 Obtain feedback on the report