1 / 47

Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats

Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats. Presented by: Bruce W. Ford bruce@clearscienceinc.com fordbw@tsc-jax.navy.smil.mil http://www.clearscienceinc.com 904-379-9704 (office) 904-536-7180 (cell). This Brief is UNCLASSIFIED. For official use only.

graceland
Download Presentation

Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats Presented by: Bruce W. Fordbruce@clearscienceinc.comfordbw@tsc-jax.navy.smil.milhttp://www.clearscienceinc.com904-379-9704 (office)904-536-7180 (cell) This Brief is UNCLASSIFIED For official use only http://www.ClearScienceInc.com

  2. Contributors • LCDR Bill Swick – NSW Group 3 • LTJG Megan Thomas – MSC • Luke Piepkorn - SPA Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  3. The Questions 1. Can a viable METOC metrics system be set up for NSW operations? 2. Can a viable METOC metrics system be installed to collect and compute metrics for SOCAL Small Boat Team (SBT) training events? Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  4. The Short Answer Yes! • A viable METOC metrics collection and calculation system for NSW TEs in SOCAL involving boats is an attainable goal. To institute such a system, some changes may be needed in the way forecasts are assembled, verified, and changes must be instituted to collect customer debrief information. The extent of those changes will be decided upon by the NSW METOC community leadership and this document strives to lay out options for metrics data collection. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  5. The Study Goals • Define study and customers - SBTs • Dissect customer process • First-guess metrics • METOC Performance Metrics • Customer Performance Metrics • Operational Impact Metrics • Proxy Operational Impacts Metrics • Propose collection process • Describe data analysis • Describe operational modeling • Map to 3-D metrics space Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  6. Process for Developing Metrics of METOC Impacts on NSW Operations Mission/TE Outcomes Mission/TE Plans EM Forecasts * Scientific/Anecdotal Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products From Murphree et al., METOC Metrics, Jan 07, murphree@nps.edu

  7. Study Definition • Training Event (TE) types • *Small Boat Ops • *Combat Swimmer Ops • *Visit, Board, Search & Seizure (VBSS) • Land Warfare • Jump Ops • Geographical Region – SOCAL area The mission types designated with the red asterisk (*) involve boats and thus are the subject of this study. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  8. The Study Goals • Define study and customers - SBTs • Dissect customer process • First-guess metrics • METOC Performance Metrics • Customer Performance Metrics • Operational Impact Metrics • Proxy Operational Impacts Metrics • Propose collection process • Describe data analysis • Describe operational modeling • Map to 3-D metrics space Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  9. Mission Execution Brief Mission Execution Brief Mission Launch Mission Launch ULT Schedule Available ULT Schedule Available Logistics Logistics EM starts forecast EM starts forecast M-6 Months M-6 Months M-1 Month M-1 Month M-72 Hours M-72 Hours M-24 Hours M-24 Hours SOCAL TE/Mission Process • Planning • Execution • 3-48 hours in duration • Similar TEs may be held for up to 7-10 consecutive days • EM may be in contact with team • Team will usually go despite red conditions – Wait and see • Debrief • Often an informal “huddle” within 24 hours after TE – no set time or location • EM do not attend • Feedback to EM typically informal by exception Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  10. The Study Goals • Define study and customers - SBTs • Dissect customer process • First-guess metrics • METOC Performance Metrics • Customer Performance Metrics • Operational Impact Metrics • Proxy Operational Impacts Metrics • Propose collection process • Describe data analysis • Describe operational modeling • Map to 3-D metrics space Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  11. Proposed METOC Performance Metrics What we forecast Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  12. Proposed METOC Performance Metrics What we can verify Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  13. Proposed METOC Performance Metrics Location of Mission-by-mission METOC Performance Metrics within 3-D Metrics Space. Purple circle denotes location of metrics. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  14. Proposed METOC Performance Metrics Location of Multiple-mission METOC Performance Metrics within 3-D Metrics Space. Purple circle denotes location of metrics. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  15. Proposed Customer Performance Metrics • What SBTs use for measure of TE/Mission success is under investigation • Proposed single-TE/mission metrics: • Mission success – Were the pre-mission objectives met? • Small boat effectiveness – We small boats used to reach their planned destination? (Boat ops, VBSS) • Target vessel – Was the target boat reached and boarded? (VBSS) • Swimmer success – Were combat swimmer able to reach their destination? (Combat swimmer) • Others? Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  16. Proposed Customer Performance Metrics • Proposed multi-TE/mission metrics: • Mission success percentage – percentage of missions that achieved their declared objective • Small boat effectiveness percentage – percentage of missions involving small boats that reached their destination? (Boat ops, VBSS) • Target vessel percentage– percentage of VBSS missions in which the target vessel was successfully boarded • Swimmer success percentage– percentage of missions involving swimmer in which the swimmer reached their destination Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  17. Proposed Operational Performance Metrics Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  18. Proposed Operational Performance Metrics Location of Operational Impacts Metrics within 3-D Metrics Space. Purple circle denotes location of metrics. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  19. Proposed Proxy Operational Performance Metrics • Significant wave height – Strongly impacts nearly all types of boating operations • Breaker height – Strongly impacts all boating operations where near-shore disembarkation is involved. • Sea Surface Temperature – Strongly impacts missions involving personnel in the water. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  20. The Study Goals • Define study and customers - SBTs • Dissect customer process • First-guess metrics • METOC Performance Metrics • Customer Performance Metrics • Operational Impact Metrics • Proxy Operational Impacts Metrics • Propose collection process • Describe data analysis • Describe operational modeling • Map to 3-D metrics space Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  21. Process for Developing Metrics of METOC Impacts on NSW Operations Mission/TE Outcomes Mission/TE Plans EM Forecasts * Scientific/Anecdotal Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products From Murphree et al., METOC Metrics, Jan 07, murphree@nps.edu

  22. Data Collection for SOCAL TEs Involving BoatsWho inputs the data? EM Forecasts (pre-mission) Anecdotal Obs (debrief) Scientific Obs (post-mission) Mission Plans(pre-mission) Mission/TE Outcomes (debrief) Operational Performance Metrics METOC Performance Metrics Legend Metrics of METOC Impacts on Operational Performance EM Inputs Non-EM Inputs Automated Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  23. Data Collection for SOCAL TEs Involving BoatsData flow Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  24. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs System inputs Automated User Pulled Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  25. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Pre-mission web form Post-mission web form Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  26. Data Collection for SOCAL TEs Involving BoatsProposed forecast creation tool • Currently little standardization of EM forecasts • Without automating the collection of forecast data, EM will have to re-enter forecast elements into metrics collection system • Forecast creation tool • Consists of a web interface where EM enters forecast elements into intuitive web form • Outputs PPT file or image to be used in PPT file to serve as the start to TE/Mission forecast • Standardizes the look of common boat-related METOC forecast elements • Allows for entry of numerical/narrative entry of METOC elements and automated threshold interpretation (e.g., EM inputs seas as 5-7 feet and the interface determines the R-Y-G threshold on the output) • Provides helpful links, thresholds and error checking functions to improve overall forecast quality • Collecting forecast elements for use in metrics calculations Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  27. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Collected by Forecast creation tool decreasing imposition on the EM Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  28. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Development/transition serverat FNMOC a likely candidate tohost NSW metrics engine Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  29. The Study Goals • Define study and customers - SBTs • Dissect customer process • First-guess metrics • METOC Performance Metrics • Customer Performance Metrics • Operational Impact Metrics • Proxy Operational Impacts Metrics • Propose collection process • Describe data analysis • Describe operational modeling • Map to 3-D metrics space Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  30. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  31. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Products served by SBT, time span or geographical region or a combination for a single METOC activity Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  32. Forecasts Plans Scientific Obs Anecdotal Obs Outcomes Unique mission records on metrics server via SIPR web forms Multi-level metrics computed Level 1: Single Mission Metrics Level 2: Multi Mission Metrics Level 3: Multi Mission Metrics over time Data Collection for SOCAL TEs Involving BoatsData flow Legend EM Inputs EM Inputs Non-EM Inputs Automated User Pulled Products served by SBT, time span or geographical region or a combination. Emphasis on long time spans and multipleMETOC activities Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  33. Conclusions • A METOC metric collection and reporting system can be instituted for SOCAL NSW TEs involving boats. • Some process changes will be necessary to facilitate metrics data collection (e.g., attend TE debriefs) • Questions remain in some details, but nothing that should pose a serious roadblock to the ultimate goal of collecting quality metrics. Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  34. Discussion Questions • Will a SIPR based collection system give access to those that need it (EMs, MSC, leadership)? • Is there interest in creating a forecast building tool, which would require some forecast standardization? • How big of a paradigm shift will it be for EMs to attend TE debriefs? • How will SBTs accept the EM attending debriefs? • How intrusive will it be for EMs to place debrief data into the system by returning to the interface following each debrief and entering information? • How intrusive will it be for the MSC or another organization to enter observation information if this process cannot be automated? • Would such an interface be of use to EM who are deployed? • Are there customer performance metrics not mentioned in this document? • When do we expand this to beyond the boats? • What options exist for entering data when the EM is not “plugged in”? Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com For official use only

  35. Questions Bruce W. Fordbruce@clearscienceinc.comfordbw@tsc-jax.navy.smil.milhttp://www.clearscienceinc.com904-379-9704 (office)904-536-7180 (cell) Ford B.W. et al, Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats, 22 Mar 07, bruce@clearscienceinc.com

  36. Proposed METOC Metrics System for SOCAL NSW Training Events Involving Small Boats Presented by: Bruce W. Fordbruce@clearscienceinc.comfordbw@tsc-jax.navy.smil.milhttp://www.clearscienceinc.com904-379-9704 (office)904-536-7180 (cell) This Brief is UNCLASSIFIED For official use only http://www.ClearScienceInc.com

  37. Back-up slides Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  38. Online Real World Metrics System – System Schematic Carrier Strike Group (CSG) Data obtained by CSG personnel via interaction with air wing NPS Data written to database Data analyzed per NPS metrics procedures Output reports delivered via web site. Reports available to CSG on demand via automated metrics calculation process. Reports delivered by OA division to air wing personnel Data entered remotely via web site CSG From NPS thesis research of LT Jeremy Callahan, USN, 2006 Murphree et al., METOC Metrics, Jan 07, murphree@nps.edu

  39. Online Real World Metrics System – Data Collection Mission Information Debriefing Information Mission Data Forecast Negative Weather Events Observed Negative Weather Events Weather Events Impacts of Weather on Mission Mission Impact Changes Made to Original Mission by Pilot Target Changes And Cancellations Mitigating Actions From NPS thesis research of LT Jeremy Callahan, USN, 2006 Murphree et al., METOC Metrics, Jan 07, murphree@nps.edu

  40. Select 1-3 mission types that are most likely candidates for the establishment of a data collection system • Recommend the most predicable mission type for initial data collection effort • Allows process challenges to be solved with “low hanging fruit” before attempting more difficult mission types. • Established planning/execution/debrief cycle • Established, stable METOC products • Recommend stable mission area • Predictable tactics • METOC products not in flux • METOC verification possible • Debriefing is routine • METOC inputs are part of planning process • METOC personnel are welcomed participants at debriefs For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  41. 2. Diagram the mission flow process including each METOC and customer personnel step including planning, execution, and debrief. • Recommend timeline format where possible • Annotate METOC inputs (routine/non-routine/formal/informal) to the cycle. For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  42. Select 1-3 mission types that are most likely candidates for the establishment of a data collection system • Recommend the most predicable mission type for initial data collection effort • Allows process challenges to be solved with “low hanging fruit” before attempting more difficult mission types. • Established planning/execution/debrief cycle • Established, stable METOC products • Recommend stable mission area • Predictable tactics • METOC products not in flux • METOC verification possible • Debriefing is routine • METOC inputs are part of planning process • METOC personnel are welcomed participants at debriefs For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  43. 3. Identify discrete and non-discrete data elements provided by METOC personnel. Determine what elements are routinely verified. • Use current METOC product examples • Discrete data element: • One forecast/one value • Ex: High Temp - 74 • Ex: Sig wave ht (bottom) – 5 ft • Ex: Sig wave ht (top) – 8 ft • Non-discrete data element: • One forecast/many potential values • Ex: Satellite image • Ex: Coastal current image • List discrete elements and identify verification sources • Anecdotal verification from customer • Scientific verification • Trained observers • Collection equipment For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  44. 4. Determine how the customer determines mission success. • Determine currently used customer measures of success • Binary (e.g., success/failure) • Non-binary (e.g., T1-T4) • Determine our ability to collect customer measures of success. Is automation possible? • Pre-mission message • Post mission message • Determine the extent of customer self-analysis • What mission information is archived by the customer? For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  45. 5. Propose metrics to be collected (potentially multiple types) Working definitions: • METOC performance metrics – Metrics of the quality of the METOC organization, including its operations, products, and services (e.g., accuracy of METOC forecasts of breaker height) • Customer performance metrics – Metrics of customer success (e.g., number of successful beach infiltrations) • Operational impacts metrics – Metrics of the impacts of METOC products on customer operations (correlation of breaker height forecast accuracy to beach infiltrations) • Proxy operational impacts metrics - Metrics of product performance that are acknowledged by METOC and customer personnel to be closely related to customer performance, so that the proxy operational impact metrics can be assumed to be good indicators of operational impacts (e.g., accuracy of breaker height forecasts which is assumed to be closely related to customer success) • Phenomena metrics – Metrics that relate product performance, customer performance, or operational impacts to specific environmental phenomena (e.g., breaker height forecast accuracy in the presence of high winds and seas; beach infiltrations in the presence of high winds and seas; correlation of breaker height forecast accuracy and beach infiltrations in the presence of high winds and seas) • Other examples of METOC metrics: • METOC Quality Metrics – Metrics of the overall quality of a product or service (e.g., forecast grade). These are a type of METOC performance metric. • METOC Capacity/Readiness Metrics – Metrics of organization’s operations (e.g., number of forecasters available, number of trained watchstanders). These are a type of METOC performance metric. • METOC Efficiency/ROI Metrics – Metrics of how well an organization uses its resources (e.g., number of forecasters required to publish forecast). These are a type of METOC performance metric. For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  46. 6. Propose system for metrics collection, calculation and display • Recommend SIPR web-based system • Minimize negative impact to METOC personnel • Strive for intuitive interface • Forecast collection • Verification input • Debrief input • Automate steps where possible • Message parsing • Forecast creator/collector • Verification collection • Strive to give METOC personnel a value-added tool • Streamline forecast process • Replace/enhance any current data collection efforts • Real time metrics calculation and display For official use only Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

  47. Questions Ford B.W., NSW METOC Metrics - Next Steps, Feb 07, bruce@clearscienceinc.com

More Related