Thar’s Gold in Them Thar AAR/ IPs

1. Thar’s Gold in Them Thar AAR/IPs

2. Welcome and Agenda Purpose and Value of After Action Reports How to develop an Improvement Plan Esther Corwin, MEP Exercise Training Officer, Texas Division of Emergency Management Esther.Corwin@dps.texas.gov 512-424-2198 Contact Info is on handout

3. What is an AAR A consolidation of information gathered during the testing and evaluation of a community’s emergency operations plan through an exercise Provides feedback to executive leadership, participating entities and governing agencies in the achievement of the exercise objectives and overall capabilities of the community Information gleaned from this process, identifies and guides future improvement actions

4. Why Write an AAR Creates a multi-agency process to review an exercise or incident General exercise identification information, but also includes evaluation information Facilitates honest and critical appraisal of actions based on plans and policies

5. Where Do We Look Exercises to simulate a stressful realistic scene Exercises to evaluate a new or seldom used equipment or skill Incidents presenting unexpected challenges Incidents having “less than expected” results OR Incidents employing innovative actions Incidents with unexpected positive results

6. Exercise Evaluation is Based On • Community policies, Emergency Operations Plan, agency SOPs and SOGs, MOUs, etc. • Exercise Evaluation Methodology • S.MA.R.T. objectives • Exercise Evaluation Guides • Observational Data • Hotwash • Participant Feedback

7. Incident Evaluation is Based On Community policies, Emergency Operations Plan, agency SOPs and SOGs, MOUs, etc. Incident Action Plans Briefings, De-Briefings Observational Data Participant Feedback

8. Types of Observational Data • Descriptive reporting • Typically yields reliable data. • Inferential reporting • Includes terms like “adequate” or “timely” • Yields inconsistent data • Evaluative reporting • Includes terms like “efficient” or “successful” • Difficult to reliably to collect

9. Three Levels of Analysis

10. Three Levels of Analysis Capability Being Evaluated Onsite Incident Command Critical Resource Logistics & Distribution Level of Evaluation Direct/Control incident management activities. Capability-Level Analysis Resource Management – Identify, dispatch, mobilize, track, demobilize and pay for resources Develop all necessary components of an IAP and obtain approvals Identify the resources based on incident need Activity-Level Analysis Establish incident objectives and priorities and operational periods Inventory resources by type/ category that are available to support this incident Task-Level Analysis

11. Exercise Evaluation Guides • Identify the activities, tasks, and performance measures to be observed • Developed and customized before the exercise • Evaluators complete the EEG by: • Logging times and actions accurately • Decision making processes and participants • Player roles and responsibilities • Coordination and cooperation • How actions were performed, resources involved • Documenting issues and recommendations

12. Data Analysis The goal is to evaluate the ability of involved functions to perform target capabilities

13. Preliminary Analysis • Evaluators organize observations into key issues and a chronological narrative • At a minimum, should include: • Description of the assigned function or operation, analyzed by capability, activity, and task • Documented record of significant observed actions

14. Review Objectives • Review exercise objectives • What was the intent of the objective? • What would demonstrate the successful performance of the objective? • If the objective was not met, what factors contributed to this result? • Review incident operational objectives • Review exercise objectives • What was the intent of the objective? • What would demonstrate the successful performance of the objective? • If the objective was not met, what factors contributed to this result? • Review incident operational objectives

15. 4 Steps of Data Analysis Fool’s Gold • Identify issues • Compare performance data to standards • Identify differences • Determine consequences • Determine root cause • Discuss conditions leading to observed behavior • The “Why” staircase

16. Why Staircase Why did it happen? Each step should explain the step above … Why did that happen? Why was that? And why was that? And why was that? ROOT CAUSE … down to the underlying root cause.

17. 4 Steps of Data Analysis • Develop recommendations • Sustain and improve • Short and long term • What needs to be changed and how • Give references • Capture lessons learned • Innovative practice or a piece of knowledge gained from experience • Provides guidance for approaching a similar problem in the future • Save time, conserve money, and accelerate preparedness

18. Draft AAR Provides feedback to participating entities and governing agencies regarding the achievement of objectives and overall capabilities Records recommendations for improvement Establish consensus and buy-in on next steps

19. The AAR • Suggested AAR format includes: • Executive Summary • Overview • Exercise Goals and Objectives • Analysis of Capabilities Demonstrated • Conclusion

20. Observations and Recommendations • Observations • References • Analysis • For improvements, list consequences of action or inaction • Recommendations • Observations • References • Analysis • For improvements, list consequences of action or inaction • Recommendations • Observations • References • Analysis • For improvements, list consequences of action or inaction • Recommendations • Observations • References • Analysis • For improvements, list consequences of action or inaction • Recommendations

21. After Action Conference Planning Team, evaluators, and stakeholders meet to review and refine draft AAR Evaluation leads to a disciplined process for implementing improvement actions and continually strengthening preparedness

22. After Action Conference Address identified issues Develop specific improvement actions Prioritizes action items Assigns responsibility to track implementation

23. Developing Improvement Actions These questions aid development: • What changes need to be made to plans and procedures to improve performance? • What changes need to be made to organizational structures? • What changes need to be made to leadership and management processes? ?

24. Developing Improvement Actions More questions to aid development: • What training is needed? • What changes to, or additional equipment are needed? • What lessons can be learned that will direct how to approach a similar problem in the future?

25. Improvement Benchmarks • Must be clearly defined and attainable • Examples include: • Number of personnel trained in a task • Percentage of equipment that is up-to-date • Finalization of an interagency agreement within a given amount of time • Include concrete deadlines to track progress toward full implementation

26. Activity – Improvement Development As a large group, given a scenario and issue, discuss possible recommendations and develop a simple improvement action. Older elementary school in a rural area just before dismissal on a spring day, parents are arriving to take students home NWS issues tornado warning estimating less than 10 minutes before impact Principal activates “bell” for sheltering, but students/parents confuse it with dismissal Some parents enter school to take child and “run” from the storm

27. Activity – Develop Improvement Issue: • Student Safety Basic Recommendation: • Develop something to assist the principle in communicating emergency information • Changes to plans and procedures • Changes organizational structures • Changes to leadership/ management processes • What training is needed • Changes or additional equipment • Lessons learned for the future Jarrell Tornado May 27, 1997

28. Summary Objectives and Evaluation Methodology are the foundation of Improvement Planning Root Cause analysis helps find the key policy, procedure, training or equipment to which an improvement could be made Applies to Exercises and actual Incidents Improvements make things better than before

29. Questions ??? Any question can lead to a good idea. Synergy is the Energy of People. Esther Corwin, MEP Exercise Training Officer, TDEM Esther.Corwin@dps.texas.gov 512-424-2198