Download
1 / 65
650 likes | 685 Views
Training Evaluation. By: Rainer Zachmann. Unit: M03U05. Content. Introduction Relevance of evaluation Monitoring course climate Pre- and post-tests Partial evaluation. Content. Continuous activity evaluation Evaluation of course coordination Evaluation of course materials
E N D
Training Evaluation By: Rainer Zachmann Unit: M03U05
Content • Introduction • Relevance of evaluation • Monitoring course climate • Pre- and post-tests • Partial evaluation
Content • Continuous activity evaluation • Evaluation of course coordination • Evaluation of course materials • Final course evaluation
Introduction • Evaluation is an essential part of the training curriculum. • Evaluation helps in many aspects of training. • Evaluation shows whether objectives have been accomplished.
Introduction • Evaluation tools may range from objective tests to subjective assessments. • None of the evaluation procedures described are intended to grade trainees or resource people.
Relevance of evaluation • Kirkpatrick’s levels of training evaluation: • Level 1 = reaction • Level 2 = learning • Level 3 = behaviour • Level 4 = results
Relevance of evaluation • Level 1 (= reaction) measures participants’ satisfaction • lowest level of measurement, nonetheless important • includes aspects before and during the course
Relevance of evaluation • Level 2 (= learning) measures: • acquisition of new knowledge; • improvement of skills; • change in attitude.
Relevance of evaluation • Level 3 (= behaviour) follows the application after training. • Level 4 (= results) assesses impact at trainees' organisations.
Relevance of evaluation • This Unit addresses Levels 1 and 2. • Levels 3 and 4 are follow-up activities.
Relevance of evaluation • Evaluation is an essential and integral part of the training curriculum, not just the end of training.
Relevance of evaluation Evaluation helps to: • determine training needs; • specify training objectives; • adjust training methods and materials; • delete unnecessary content.
Relevance of evaluation Evaluation helps to: • execute training; • verify accomplishments; • follow-up with trainees; • reduce training costs.
Relevance of evaluation • Evaluating whether objectives have been accomplished is as important as specifying training objectives.
Relevance of evaluation Two types of evaluation: • formative evaluation to monitor training • summative evaluation to measure accomplishments
Relevance of evaluation • Teachers and organisers often evaluate informally and unconsciously.
Relevance of evaluation • Objective tests are based on specific questions to be answered. • Response tests give freedom for subjective expression. • Practical skills testing is important for PGIS.
Relevance of evaluation Formal evaluations include: • monitoring course climate; • pre- and post-tests; • partial tests; • continued activity evaluation; • evaluation of course coordination and course materials; • final course evaluation.
Relevance of evaluation • Evaluations are not intended to grade people, but to monitor course progress.
Monitoring course climate Many methods exist: • “Democracy Wall” • “Evaluation Dartboard” is useful for monitoring participants’ feelings
Multilingual democracy wall deployed at the 2007 Web2forDev Conference in Rome, Italy Photo: Anja Barth, CTA (From Unit M10U01)
Taylor, B. and Beniest, J. 2003. Training in agroforestry - A toolkit for trainers Page 2.176; ISBN 92 9059 151 x The World Agroforestry Centre United Nations Avenue PO Box 30677, GPO 00100 Nairobi, Kenya www.worldagroforestrycentre.org
Pre- and post-tests • The pre-test assesses trainees' initial knowledge. • Multiple-choice questionnaires may be used. • Multiple-choice questions are difficult to design, but analysis is easy and objective.
Pre- and post-tests • Multiple-choice questions consist of a "stem" and several options for answers. • The stem should be a concise statement that leads directly into the options.
Pre- and post-tests • Include in the stem as much information as possible about the item. • Formulate the stem concisely. • Each option should be a grammatically correct completion of the stem.
Pre- and post-tests • Multiple-choice questions consist of a stem and: • a definitely wrong distracter; • one correct answer; • (an)other definitely wrong distracter(s).
Pre- and post-tests • Example: Multiple-choice questions are: a) easy to develop distracter b) easy to answer distracter c) relatively objective answer d) not widely applicable distracter
Pre- and post-tests • Only one option should be the correct answer. • Correct answers must not be self-evident. • All other options, i.e. distracters, should be plausible but definitely wrong. • The length of options should not provide a clue to the answer.
Pre- and post-tests • Coordinators should ask only relevant questions. • Avoid negatively formulated questions. • Arrange the answers in random order. • Coordinators should not use multiple-choice questions when other evaluations are more appropriate.
Pre- and post-tests • Coordinators should present the pre-test at the beginning of the course. • Ask trainees to be honest and not to guess. • Analyse the test immediately. • Discuss results with trainees and resource people. • Coordinators should not give the questionnaires back.
Pre- and post-tests • Use the same questionnaire for the post-test. • Present the post-test one day before the course ends. • Analyse the test overnight. • Compare pre- and post-test results and present them to trainees and resource people.
Partial tests • Partial tests help ensure that resource people cover subject matter appropriately. • Coordinators should not use partial tests to grade trainees.
Partial tests • Coordinators should use the first 15 to 20 minutes of each day for the partial tests. • Ask four or five questions from topics of the previous day. • Ask open essay questions.
Partial tests • Coordinators should design questions and expected answers (both!) in collaboration with resource people. • The partial tests should only cover relevant course content.
Partial tests • Allow trainees to use open books, training documents, etc. to place them in a real life situation where they may consult references.
Partial tests • Coordinators should analyse answers immediately with instructors. • Return tests to trainees and discuss results.
Continuous activity evaluation • Traineesevaluate the course. • At the beginning of the course, explain how to use the evaluation. • Ask trainees to be critical.
Continuous activity evaluation • Every morning, coordinators should distribute new evaluation forms. • Ask trainees to evaluate immediately. • At the end of each activity, write the title of the activity on a board to remind trainees to evaluate.
Continuous activity evaluation • At the end of the day, coordinators should collect the evaluation forms. • Summarise results on the summary forms.
Continuous activity evaluation • Coordinators should include trainees in processing the evaluations. • Ask trainees to draw conclusions and report them back to the class the next morning.
Continuous activity evaluation • Coordinators should give the summaries to the resource people and discuss the results. • Keep copies in the files for future planning.
