METOC Metrics in Operational Modeling

METOC Metrics in Operational Modeling Prepared for: ASW Metrics Symposium 4 January 2007

Outline • Operational Modeling & METOC Metrics • WIAT Description • Parametric Analysis Description & Results • Application to ASW Metrics Program

Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a strike mission model. Operational Modeling & METOC Metrics * or other products

Forecasting & Mission Outcomes Symmetry exists between forecasting and operational outcomes

Forecasting & Mission Metrics Accuracy of Recommended Mission Changes = D / B Accuracy of Forecasting METOC Events = D / B

better Actual Environment better worse Predicted Environment Impacts of Environmental Uncertainty on Operational Metrics Mission Planning Uncertainty Forecast Uncertainty Mission Performance Forecast Performance Predicted (better outcome) Actual (worse outcome) Actual (worse environment) Predicted (better environment) Accurate Predictions Mission Outcome Uncertainty • Opportunity Costs • Actual is better than predicted • Over-prepared for environmental risks • Spend too many resources, too much time • Resources better used for other locations/missions • Effectiveness Costs • Actual is worse than predicted • Under-prepared for environmental risks • Deploy too few resources, too little time • Increased risk of mission failure

3 Forecasting Situations Good Forecasting Probability of Accurate Forecast Avg Forecasting Average Single Shot Probability of Kill Bad Forecasting Missions Saved due to METOC Recommendation Cloud Coverage % Forecast Accuracy Forecast Accuracy Strike Sorties w/ No BDA Forecast Accuracy Operational Modeling & METOC Metrics Improving METOC support products allows for improvement in both system and operational performance

Weather Impact Assessment Tool (WIAT) Overview • WIAT is a set of modeling and simulation tools used for assessing the impacts of METOC forecasts and METOC phenomena on strike operations • WIAT • Simulates modern naval strike operations with a focus on close air support (CAS) and kill box interdiction (KI) • Inputs both actual weather conditions and forecasts of conditions with varying lead times and accuracies • Emulates actual mission planning processes days and hours ahead of mission execution • Models customer decisions and outcomes based on weather predictions and other factors • Assesses impacts of weather and weather forecasting accuracy in terms of operational metrics • WIAT is currently configured for strike warfare (STW), but is adaptable to other warfare areas (e.g. NSW, ASW)

Weather Impact Assessment Tool (WIAT)Value • WIAT provides a readily accessible and versatile web-based system for developing model based data and calculating model base operational impacts metrics • Major uses of WIAT • Simulation of impacts of METOC forecasts and phenomena on spectrum of warfighting situations • As a laboratory for the assessment of strike effectiveness employing a variety of METOC support situations • Can allow for the evaluation of pros and cons of new METOC support products

ATO DRIVERS TACAIR Missions METOC Impacts on Operational Metrics Threat Forces Forecasted Weather • Changes Made During Planning Process Due to Weather Forecasts • Changes Made During Mission Due to Forecasts and Actual Weather • TACAIR Mission Success • Opportunity Costs of Decisions Based on Forecasts and Actual Weather ATO Actual Weather Blue Ground Forces ISR Assets WIAT - System Schematic for STW

Flight Plan Generation and Modification Timeline Resource (Sortie) Allocation Flight Ops Initial Plan Generation 1 hr Out 48 hrs Out 24 hrs Out JPITL • Secondary modifications • 1 hour prior to mission • Primary modifications • 24 hours prior to mission • Initial flight plan generation • 48 hours prior to mission METOC Forecasting Products Strike Missions CAS Missions KI Missions Daily Target List ATO Mission (#AC, Weapons) ATO Mission (#AC, Weapons) ATO Mission (#AC, Weapons) ISR, FACs Simulation of Adaptive Strike Mission Planning Process

Includes forecasted and actual conditions over time from OIF weather briefing maps Spatial resolution: 25 x 25 nm Temporal resolution: varying, down to 2 hrs Represents forecasted and observed cloud density, cloud layers Includes inputs for other forecasted and observed conditions Simulates interactions of OA division and mission planners Allows forecasts of various accuracies to capture operational effects of forecast uncertainty Theater Display Weather Display WIAT - METOC Features

Force Composition Size: Carrier Air Wing Sorties modeled as TACAIR missions 2 AC per mission Objectives Provide CAS / KI to ground forces in transit and during engagements Engage fixed strike targets & targets of opportunity Behavior Fly missions according to the adaptive ATO Sample Metrics Mission modifications due to weather 24 hrs prior & 1 hr prior TACAIR mission success (targets destroyed / KI stations maintained / CAS successes) Opportunity costs (targets that were incorrectly avoided due to erroneous forecasts) CAS Mission KI Target of Opportunity Fixed Strike Mission WIAT - TACAIR Features

Real World Data Collection and Analysis Operational Engagement Model Real World Metrics Model Metrics • Synthesis • improved metrics • process improvements • improved warfighting operations Integration of Real World and Model Metrics

ATO DRIVERS TACAIR Missions METOC Impacts on Operational Metrics Threat Forces Forecasted Weather • Changes Made During Planning Process Due to Weather Forecasts • Changes Made During Mission Due to Forecasts and Actual Weather • TACAIR Mission Success • Opportunity Costs of Decisions Based on Forecasts and Actual Weather ATO Actual Weather Blue Ground Forces ISR Assets Integration of Real World and Model Metrics Updated planning process to include probabilities of impact mitigation recommendations being correct / incorrect and accepted / unaccepted Based on Output From Real World Metrics System Updated Forecast Accuracy Data For METOC Features

Parametric Forecasting Analysis • Purpose • To show that improved forecasting capability leads to improvement in both METOC and operational metrics in STW • Scope • Parametric look at decreased forecasting capability of cloud coverage % • Coverage % in each location rated on a scale of 1 to 8 • 1-4 = Green Forecast (Go conditions) • 5-6 = Yellow Forecast (Degraded conditions) • 7-8 = Red Forecast (No Go conditions) • Weather stream taken from one week in March during Operation Iraqi Freedom (OIF) • Forecasts randomly generated • Mean is set to actual weather value • Standard deviation increases from 0 to 2.5 (out of 8)

# Correct Forecasts # Correct Event Forecasts FAC = POD = Total Forecasts Total Events Parametric Forecasting Analysis • Measures of Effectiveness/Performance • METOC Metrics • Forecasting Accuracy (FAC) • Probability of Detecting Weather Events (POD) • METOC Impact Metrics • Missions Saved due to METOC information • Mission Change Accuracy • Operational Metrics • Number Sorties with No Bomb Damage Assessment (BDA) • Probability of Kill per Weapon Expenditure

WIAT – METOC Metrics Forecast Accuracy FAC & POD Based on Forecasts in Mission Locations Only Percentage of Missions Flown • No Red Wx events occur during simulation • Higher accuracy in forecasting Green Wx than Yellow or Red • Execution Forecast POD is higher than Planning Forecast Probability of Detection of Weather Events Percentage of Missions Flown Standard Deviation

Missions Change Accuracy Missions Saved vs Total Mission Changes Missions False Alarms (Incorrect Changes) Standard Deviation Standard Deviation WIAT – METOC Impact Metrics Percentage As Forecast Accuracy Decreases, the Number of Missions Changes Increases with Decreasing Accuracy

As METOC forecast accuracy decreases more missions are needed to achieve similar numbers of target kills Over 20 additional missions are needed to inflict similar damage Targets Killed Per Mission Standard Deviation WIAT – Operational Metrics Missions Flown vs Target Kills • 18% decrease in Targets Killed per Mission Flown

The methods, tools, and techniques developed during the Strike Metrics project can be adapted to fit the ASW Metrics project ? Application to ASW Metrics Project

ASW Metrics DevelopmentModeling and Simulation Approach • Develop realistic vignettes and scenarios for ASW, including mission-level metrics of ASW success • Identify the environmental factors that influence ASW platform and system performance • *Describe and model the ASW mission planning process, to include how and when METOC information is incorporated • Develop an engagement model for ASW • Perform sensitivity analysis on METOC product accuracy and timeliness * - Probably the least understood process of this approach

ASW Metrics DevelopmentM&S Role Within the Metrics Development Process • Basic analysis and description of ASW process: • Planning, execution, and post-op assessment process used by ASW customers • METOC contributions to that process • Initial selection of major metrics to be generated by system (focus on performance and operational impacts of reachback cell support?) • Determination of classification level for system • Coordination CNMOC N6 to ensure alignment with IT standards • Selection, prioritization, and setting of development timeline for components of METOC metrics system. Major components: • Real world data collection and analysis system for assessing product performance and operational impacts • Operational model for simulating METOC impacts on operations • Pilot development of: • Real world data collection and analysis system for assessing product performance and operational impacts • Operational model for simulating METOC impacts on operations • Transitioning of pilot system to operational use Inputs to M&S Process Product of M&S Process

ASW Engagement Model Application • A “laboratory” for experimentation, similar to the Strike / NSW application of WIAT • Investigate the effect of METOC products and services across a wide range of ASW scenarios • Effect of Increased Accuracy • Effect of Enhanced Timeliness • Develop METOC support benchmarks to establish goals and evaluate real world performance Scenario Multiple combinations of METOC products and services support ASW METOC Product

ASW Scenarios Within a campaign several force allocation and/or CONOPs decisions exist: • SSN • ASuW • TLAM Strike • ASW Barrier Patrol • ADS Monitor • Surface Ship • ASuW • TLAM Strike • Theater Missile Defense • ASW Sea Base Defense • ASW SLOC Escort • MPA • ASW Sea Base “Pouncer” Mission • ASW Fwd Area Search • ASW SLOC Escort • ADS • ASW Barrier Length / Location • SURTASS • ASW Cuing Location • HELO / VTUAV • ASW Sea Base / SLOC Defense • ASW Barrier Patrol Notional Illustration of the ASW Campaign TLAM Threat Country Threat SS TLAM Threat SAG ASuW Threat SS ADS Pouncer Areas TMD Sea Base SLOC Transit Path

Environmental Impact on ASW Multiple Combinations of Sensors, Weapons, and Environmental Features:

Threat Port SSN To Threat Op Areas ASW Mission PlanningSSN Barrier Example Mission Planning Steps • Determine length and position of barrier desired between threat ports and op areas • Determine the barrier performance objective (e.g., 0.90 P(detect) against quiet transitor) • Compute forces required based on the perceived environment, sensor performance, and threat/searcher speed • METOC Metrics Issues • What is the impact of accurate knowledge and modeling of the environment on mission planning? • How does the timeliness of the information affect mission planning and execution? • Ultimately, what level of METOC support is required to support this mission?

Single SSN Effectiveness SSN Necessary For Mission Database Version Database Version Data Source Data Source ASW Mission PlanningValue of Updated METOC Information Improved METOC support allows for improved ASW mission planning by identifying the required number of assets

Costs of Inaccurate Information Mission Outcome Uncertainty (e.g. detections, kills) • Opportunity Costs • Actual is better than predicted • Deploy more assets than necessary • Resources better used for other locations/missions Accurate Predictions (Optimal # of Searchers are Provided) • Effectiveness Costs • Actual is poorer than predicted • Deploy too few assets • At risk for losses Actual Predicted

Notional Example of ASW Engagement ModelSSN Barrier Application • Goal: Investigate the affect of different combinations of METOC Product accuracy and timeliness on SSN Barrier planning and execution • Mission planner has to identify the optimal number of SSNs to perform the barrier search to 90% effectiveness • Up to five SSNs are available for the mission • SSN Performance prediction accuracy can be from +/- 1 to +/- 5 nm, depending on the level of METOC support provided • The more advance notice the mission planner receives, the less impact there is on asset availability and mission performance:

Notional Metrics Development Output Output: Measure the % Likelihood that the mission planner will provide the optimal number of SSNs required Resulting METOC Metrics EX: SSN Barrier Patrol, Product B Scenario No ASW payoff from additional accuracy / timeliness +/- 1 ASW Performance is Improved +/- 3 METOC Product UNSAT - No Impact above Historical +/- 5 Never H-12 H-24 H-2 Optimal combinations of accuracy and timeliness lie on the blue line

Summary • M&S Role in 2 of 3 major steps of metrics development program: • What metrics to collect • How to collect them • What standards/benchmarks to measure against • ASW mission planning process / modification timeline development • Performance benchmarks: • ASW does not lend itself well to the establishment of “go/no-go” criteria, as in STW • Working defn. of SAT: Any increased accuracy and/or timeliness would not improve the scenario-level metrics • Working defn. of UNSAT: The accuracy and timeliness provided provides no meaningful benefit above the use of historical databases / existing uncertainty • Measure of timeliness implies an estimate of forecast accuracy, in addition to sensor accuracy

SPA Team Contact Info Paul Vodola pvodola@spa.com paul.vodola_contractor@spa.dtra.smil.mil 703-399-7255 Luke Piepkorn lpiepkorn@spa.com 703-399-7239 Matt McNamara mmcnamara@spa.com 703-399-7266 2001 North Beauregard Street Alexandria, Virginia 22311-1739 Fax: 703-399-7365 www.spa.com

METOC Metrics in Operational Modeling