Chapter 3

1. Chapter 3 TAEDES505A Evaluate a training program

2. Plan and prepare evaluation For an evaluation undertaken to be of any real value it must be properly planned: you need to be clear about the purpose of the evaluation and what criteria will be used to determine outcomes.

3. Aim and scope Aim • Examine the effectiveness of the training • Is there a demand for this qualification? • Will training ensure an employment outcome for the learner? • Return on investment • Increase in productivity Scope • Examine the whole of the organisation • Examine only specific parts of the organisation • Examine only specific qualifications • Examine operational issues

4. Information sources • Analysis of work performance/observation • Analysis of workplace productivity • Feedback from employers, students and trainers • Secondary research from government, ISBs or other industry bodies • Organisational documents: policies and procedures • Legislative documents

5. Who are the stakeholders? • Employers • Training graduates • Workplace supervisors • Trainers and assessors

6. The project plan • Roles and responsibilities of review team members • Constraints on the project • Project risk assessment • Ground rules for the project

7. Collect and collate data With the project plan prepared and task roles and responsibilities allocated, the review process can begin. In the initial stages this means gathering all relevant documents and information and collating the results for evaluation.

8. Evaluation instruments • Surveys • Questionnaires • Direct examination of outcomes • Diaries or logs • Workplace observation

9. Example of an observation/competency checklist

10. Analysing information Regardless of whether you are reviewing the RTO’s entire operation or only selected segments, the process will be the same: the actual documents and procedures will need to be crosschecked and compared to the legislative or organisational standards to determine if they are compliant and meet requirements. Examples of what to look for follow ...

11. Learning and assessment strategy • Have these strategies been developed in line with the relevant qualification’s packaging rules? • Has the latest version of the qualification summary been used so that all unit codes and benchmarks are correct? • Have all the core units of the qualification been included? • Have the elective units been chosen with workforce development outcomes in mind? • Have all necessary prerequisite units been observed and included? Are these units being delivered in the right order? • Is the strategy aimed at a specific target group and if so: • does the strategy detail all the necessary resources needed for these learners to complete the program successfully? • are the delivery methods relevant to this target group? • is the duration of the training program sufficient to allow the target group to complete successfully? • does the program address workforce development needs?

12. Assessment tools • The review of these tools should be conducted with the unit/s of competency close to hand. • The assessment tool should be compared to and should encompass the unit’s performance criteria as well as any other requirements as outlined in the individual units. • Other aspects of assessment review will include the level of moderation and validation the assessment tool has received: • how often is moderation and validation undertaken? • what recommendations were made and what action was taken? • Check that the assessment is appropriate to the AQF level at which it is aimed.

13. Program documentation • Have student details been entered correctly? • Is the record of their progress through the qualification accurate and up to date? • Have completed assessment tools been marked and stored appropriately? • Have all necessary forms and documents been completed accurately and in a timely manner? • Are reporting requirements (such as AVETMISS) being completed accurately and on time?

14. Compliance with ASQA Standards In all cases, regardless of what is being reviewed, the RTO must measure its performance against the ASQA Standards. A condition of registration is that these standards be upheld and met at all times. This can be achieved through adherence to a continuous improvement program and the development of checklists or forms that can be used during an evaluation process.

15. Documenting findings Table 3.4 Self-audit document with non-compliances shown Source: www.training.com.au/pages/menuitem5cbe14d51b49dd34b225261017a62dbc.aspx

16. Factors that affect performance • Job redesign • Change in supervision arrangements • Change in work support/mentoring • Communication and information campaigns • Job aids • Standard operating procedures

17. Reporting and recommendations The final part of the review process is the formal report on the findings of the evaluations undertaken. This report should then, ideally, form part of the organisation’s continuous improvement program. The report should include information on any breaches of compliance identified as well as suggestions for their rectification.

18. Making recommendations Any suggestions made must be discussed with relevant stakeholders prior to formalising the report. There is little point in making a prescribed recommendation if the organisation is unable to meet the suggested pathway or alternative. While compliance with the ASQA standards is not negotiable, there are generally a number of methods that can be used to ensure that these standards are indeed met and discussions with stakeholders at the end of the evaluation process can identify the best ways of rectifying any areas identified for improvement.

19. Evaluation report Table 3.6 Rectification report (extract)