[Insert Exercise Name] Evaluator Briefing and Guidance

1. [Insert Exercise Name] Evaluator Briefing and Guidance

2. Welcome & Introductions

3. Meeting Agenda • Exercise Overview • Exercise Agenda • Evaluation Goals • Exercise Evaluation Guide (EEG) Overview • Evaluation Process • Assessment, Review, and Analysis of the Exercise • Conclusion

4. Purpose [Insert exercise purpose]

5. Scope [Insert exercise scope]

6. Target Capabilities • [Insert Target Capabilities]

7. Exercise Objectives • [Insert objectives. Use additional slides if necessary.]

8. [Insert exercise agenda] Agenda

9. Evaluation Goals • Validate strengths and identify improvement opportunities • Validate plans, procedures, and protocols • Improve processes and plans • Determine level of capability in regard to the exercise’s target capabilities

10. Exercise Evaluation Guide (EEG) Overview

11. EEG Concept • The Homeland Security Exercise and Evaluation Program (HSEEP) Volume III provides evaluation and improvement planning policy • Evaluations should be objective-based to remain in line with: • National Preparedness Goal • Target Capabilities List (TCL) • Jurisdiction and agency plans • HSEEP EEG templates provide streamlined process for standardized evaluation concept, not necessarily content

12. Consistent across exercises from TCL Consistent across exercises from Universal Task List (UTL) EEG Concept (cont’d) • Capability-based approach relies on similar activities to achieve a capability • Activities are achieved through successful completion of tactical tasks • Task detail should come from actual plan being exercised Content should be based on UTL detail, but include actual text from applicable plan

13. EEG Function • Allows evaluators to review validation of plan elements • Plans used will depend on participating agencies • Includes direct references to plan sections, tables, and charts • Allows evaluators to better prepare for exercise • Provides detail for what will be expected of players • Remains focused on exercise objectives as they relate to the selected target capability

14. Use of EEGs • Evaluation team should use EEGs in conjunction with full plans to prepare • Use EEGs as guides for observation, but do not focus on completing them during the exercise • Use EEGs for data analysis after exercise and in preparation for the evaluator debriefing • Use measured results to make actionable improvement recommendations within the After Action Report (AAR)/ Improvement Plan (IP)

15. Data Collection Process Facilitator Notes Issue Writeup AAR/IP Hot Wash following exercises to resolve conflicts Corrective Action Plan Policy, Plan, Procedure Revisions Evaluator Notes and Data EEG Analysis Form EEG AAR Data Input Form

16. Evaluator Notes and Data • Evaluators should capture player discussions, decisions, actions, and activity during exercise play • The EEGs provided should serve as guides for documentation of exercise activity

17. EEG Analysis Form • Each evaluator will be provided with EEGs that will give specific guidance on what data to collect during the exercise, how to record it, and how to analyze it • To be completed at the conclusion of the exercise, using data gathered during the exercise

18. EEG AAR Data Input Form • Each EEG will include a second form to be completed during the Facilitator and Evaluator Debriefing • This form will serve as an expansion to the notes captured in the EEG Analysis Form, including an area to provide recommendations

19. After Action Report (AAR)/Improvement Plan (IP) • Data from the EEGs and debriefing will be compiled into the AAR/IP • The AAR/IP will include assessments for each capability observed as well as issues and recommendations • AAR/IP data will be entered into the Corrective Action Program (CAP) System AAR Data Input Form Snapshot CAP System

20. Evaluation Process

21. Evaluator Responsibilities • Understand the exercise concept and scenario • Collect information • Make observations on player deliberations, discussions, and decision processes • Analyze actions and develop a post-exercise AAR/IP

22. Preexercise Responsibilities • Review appropriate plans, procedures, and protocols • Review EEGs • Review and be familiar with scenario, objectives, and target capabilities • Participate in Facilitator and Evaluator Briefing

23. Exercise Responsibilities • Monitor and document player performance • Use EEGs to confirm objectives are being met • Focus on critical tasks, as specified in the EEGs and plans being exercised • Participate in the Hot Wash and Facilitator and Evaluator Debriefing

24. Postexercise Responsibilities • Assist Lead Evaluator in compiling and reporting evaluation data • Use data to determine whether the exercise’s target capabilities and plans were effectively implemented or demonstrated • Use data to identify strengths and areas for improvement • Complete AAR/IP Input Form for each identified strength and area for improvement

25. Evaluator Guidelines At the start of the exercise, evaluators should: • Position themselves within the appropriate group or area • Determine an ideal location to observe exercise play

26. Evaluator Guidelines (cont’d) Evaluators should not: • Interfere with exercise discussion

27. Observing the Exercise Evaluators must keep an accurate written record of important issues that take place during the exercise: • Who would perform the action or made the decision? • What was the discussion that occurred? • Why was the decision made? • How did they make the decision?

28. Recording Important Events • Evaluators should record events such as: • Initiating scenario events • Discussions of players • Key decisions • Deviations from plans and procedures • Equipment discussed • Directives

29. Assessment, Review, and Analysis of the Exercise

30. EEG Analysis Matrix

31. Player Hot Wash • Allows players to participate in a self-assessment of the exercise play • Allows evaluators to clarify additional points and collect missing information

32. Facilitator and Evaluator (F/E) Debriefing • Work to complete EEGs for submission • Discuss degree to which each objective was completed • Provide strengths, best practices, lessons learned, and possible IP recommendations • Begin outlining issues to be included in AAR/IP

33. Questions/Discussion