Exercise Evaluation

1. IAEA Training in Emergency Preparedness and Response Exercise Evaluation Module L-056 Lecture

2. Objectives • Understand the importance of evaluation for the success of the exercise • Understand performance-based evaluation • Know the role of evaluators and their procedures • Be able to develop evaluation criteria • Know how to grade and report evaluations L-056: Exercise Evaluation

3. Principles of Evaluation • Performance required by plans and procedures is being tested • Evaluations are results-oriented to enhance the plan and the supporting elements, such as proper organization, training, and equipment L-056: Exercise Evaluation

4. Aim of Evaluations • The aim of an evaluation is to identify strengths and weakness in the emergency response capability L-056: Exercise Evaluation

5. What is a Performance-based Evaluation? • Focus on results, not on process • Based on response objectives • It answers the question: was the result achieved? L-056: Exercise Evaluation

6. How to Evaluate Based on Performance (1) • Define the response or exercise objective • e.g. Protection of facility personnel • Define the criteria • e.g. The personnel are promptly alerted • e.g. The dose to personnel is kept within established limits L-056: Exercise Evaluation

7. How to Evaluate Based on Performance (2) • For each performance objective, define the evaluation criteria • e.g. The personnel are promptly alerted • Was a public address message broadcast immediately upon discovery of an emergency situation? • Did all personnel hear the message? • e.g. The dose to personnel is kept within established limits • Were personnel relocated to safe areas? • Was the dose rate monitored at personnel assembly areas? L-056: Exercise Evaluation

8. How to Evaluate Based on Performance (3) • After the exercise, based on the FACTS, answer the question: “was the performance objective met”? • It is not absolutely necessary that all criteria be satisfied • The actual response will depend on the situation L-056: Exercise Evaluation

9. How to Evaluate Based on Performance (4) • If the performance objective was met • Was the procedure followed? • Was the result achieved in good faith or by emergency? • If the performance objective was not met, why? • Was the procedure followed? • Was training lacking? • Was equipment lacking? • Was the exercise at fault? L-056: Exercise Evaluation

10. Evaluation Team • Organization • Chief evaluator • Evaluators for all key groups or locations L-056: Exercise Evaluation

11. Evaluator Role and Responsibilities • Chief evaluator • Coordinate the evaluation team • Instruct evaluators on the schedule, method and criteria • Ensure consistency between evaluators • Instruct the evaluators on the format and schedule of the evaluation reports • Produce the final evaluation report L-056: Exercise Evaluation

12. Evaluator Role and Responsibilities • Evaluators • Observe and record • Do not evaluate during the exercise • Stick to facts • Evaluate after the exercise • Provide a report to the chief evaluator L-056: Exercise Evaluation

13. Choice of Evaluators • Evaluators must be qualified for the function they evaluate • Evaluators must be trained • Evaluation techniques • Procedures of the groups being evaluated • Evaluators do not need to know the detailed scenario L-056: Exercise Evaluation

14. Preparation of the Evaluation Team • Brief the evaluation team • Provide clear written instructions • Organize a site orientation tour L-056: Exercise Evaluation

15. Evaluator Instructions • Before the exercise: • Review the scenario timeline and objectives • Review the procedures used by the participants • Review the applicable evaluation criteria • Understand the actions required to demonstrate a performance objective L-056: Exercise Evaluation

16. Evaluator Instructions (Cont’d) • During the exercise • Arrive before an event • Locate a telephone • Wear Evaluator’s identification • Do not interfere with the participants • Never criticize the participants • Minimize communications L-056: Exercise Evaluation

17. Evaluator Instructions • Following the exercise • Attend debriefing and critique • Observe the effectiveness of the debriefing and critique • Obtain exercise documents • Ensure exercise location is in a safe state • Finalize your chronology notes L-056: Exercise Evaluation

18. Observation Techniques • Record arrival times • Record names and functions • Record times of major events • Record repetitive interventions • Listen to commands and instructions • Listen and record Controller inputs • Record exercise observations L-056: Exercise Evaluation

19. Guide for Evaluators • After the exercise, based on the FACTS, answer the question: “was the performance objective met” • It is not absolutely necessary that all criteria be satisfied: • The actual response will depend on the situation L-056: Exercise Evaluation

20. Drillmanship • Drillmanship is the art of exercise delivery • Record facts on: • Simulation • The exercise scenario • Safety • Participant feedback L-056: Exercise Evaluation

21. On the Use of Evaluation Checklists... • Several organizations use checklists for each group being evaluated • This focuses the evaluation on each group in isolation rather than on the overall coordinated response performance • Focus on recording important facts rather than “checking” a list • Evaluate the overall performance on the basis of exercise objectives using all the facts collected by the evaluators L-056: Exercise Evaluation

22. Debriefings • Hot Wash-up • Exercise Team Debriefing • Executive Debriefing • Brief the management staff • Give credit to the players for identifying deficiencies and give them a grace period to fix the problems before forcing compliance or imposing penalties L-056: Exercise Evaluation

23. Safety Responsibilities • Evaluators are also responsible for monitoring exercise play to ensure a safe environment L-056: Exercise Evaluation

24. Grading Criteria • Excellent • Satisfactory • Unsatisfactory L-056: Exercise Evaluation

25. Grading Criteria (cont’d) • Grading must be supported by facts (if it was not observed, it didn’t happen) • Trivial mistakes remain trivial even when they are observed L-056: Exercise Evaluation

26. Classification of Unsatisfactory Grades • Critical Deficiency • Major Deficiency • Minor Deficiency L-056: Exercise Evaluation

27. Deficiency Classification • Planning • Training • Resources L-056: Exercise Evaluation

28. Exercise Report • A useful report is most difficult to write • Based on facts, not impressions • Focus on key areas • Use the exercise objectives as the basis for the report • Prepare and publish in a timely fashion • Not everybody will agree with the findings L-056: Exercise Evaluation

29. Post Exercise Actions • Report clarification • Ensure there are no inaccuracies in the report • Critical deficiencies • Take corrective actions as soon as possible, e.g. one month • Major deficiencies • Take corrective actions within a reasonable time frame, e.g. months • Minor deficiencies • Take corrective actions during the annual revision of the plan or procedures L-056: Exercise Evaluation

30. Summary • The evaluation should be performance-based • Evaluators and controllers are not the same persons • An evaluator does NOT evaluate DURING an exercise, but after • Base the evaluation report on the exercise objectives • Look for corrective actions L-056: Exercise Evaluation