1 / 37

Metrics of METOC Support of NSW Operations

Metrics of METOC Support of NSW Operations. Tom Murphree and David Meyer Naval Postgraduate School (NPS) murphree@nps.edu Bruce Ford and Manuel Avila Clear Science, Inc. (CSI) bruce@clearscienceinc.com Paul Vodola, Matt McNamara, Luke Piepkorn, and Ed Weitzner

marsden-daniel
Download Presentation

Metrics of METOC Support of NSW Operations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metrics of METOC Support of NSW Operations Tom Murphree and David Meyer Naval Postgraduate School (NPS) murphree@nps.edu Bruce Ford and Manuel Avila Clear Science, Inc. (CSI) bruce@clearscienceinc.com Paul Vodola, Matt McNamara, Luke Piepkorn, and Ed Weitzner Systems Planning and Analysis (SPA) pvodola@spa.com NOSWC Conference NAB Coronado, Coronado, CA 26 June 2008 A Research, Development, and Transitioning Project Sponsored by SPAWAR / PMW120

  2. NSW METOC Metrics • Outline • Why METOC Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status and Way Ahead

  3. NSW METOC Metrics: Capabilities Summary • The NSW METOC Metrics System will enable the capability to: • Conduct semi-automated online collection of data from METOC units and customers • Assess performance of METOC forecasts and other products • Determine actual and potential impacts of METOC products on mission planning and outcomes • Estimate uncertainty and confidence in METOC products. • Identify gaps in, and methods for improving quality and efficiency of, mission planning and execution with respect to METOC factors • Develop recommendations for improving training, mission planning & execution • Helps answer the questions: • How good is METOC support for NSW Ops? • How good is good enough? NSW METOC Metrics, lpiepkorn@spa.com, 3

  4. NSW METOC Metrics: Benefits to Fleet • Improved operational effectiveness via improved METOC support • More accurate METOC products • Improved timeliness of METOC products • Identification of, and focus on improving, METOC products with large impacts on operational effectiveness • Quantitative and objective assessments of METOC support and products, for example: • Accuracy of forecasts • Uncertainty and confidence in forecasts • Quantitative and objective assessments ofthe impacts of METOC support on NSW operations • Provides mechanism for the warfighter to efficiently contribute to METOC process improvement

  5. NSW METOC Metrics • Outline • Why Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status & Way Ahead

  6. NSW METOC Metrics: Metrics Development Process Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance Apply this process to both real world data and output from military mission models. * or other products

  7. NSW METOC Metrics: Metrics Collection & Analysis Methodology Mission Plan METOC Forecast METOC Recommendation Revised Mission Plan Mission Execution Mission Debrief, Post-Mission Analysis, Reconstruction & Analysis, Forecast Verification Mission Planning, Execution, and Analysis Process In full METOC metrics effort, need to collect and analyze data for the entire process.

  8. NSW METOC Metrics: Quantifying Performance & Operational Impacts Current Speed: Actual vs. Forecasted 5 4 3 2 1 0 Actual 0 1 2 3 4 5 Forecasted Quantitative analyses of model forecast errors lead to S/T metrics (e.g., model forecast performance metrics). Which of these forecasts are good enough and add value to end user operations? For example, are forecasts with small percentage errors (inside purple cone) good enough?

  9. NSW METOC Metrics: Quantifying Performance & Operational Impacts Current Speed: Actual vs. Forecasted 5 4 3 2 1 0 Actual 0 1 2 3 4 5 Forecasted 2 No Go Threshold • Which forecast errors lead to significant negative impacts on the warfighter? • Dependent on several factors including: • Operational thresholds • Whether forecasts are followed by mission planners

  10. NSW METOC Metrics: Quantifying Performance & Operational Impacts Current Speed: Actual vs. Forecasted 5 4 3 2 1 0 Not Good Enough Good Enough Actual Good Enough Not Good Enough 0 1 2 3 4 5 Forecasted 2 No Go Threshold • Assuming thresholds and forecasts are generally followed by mission planners, we can assess: • Whether forecasts are good enough • Effectiveness of METOC support

  11. NSW METOC Metrics: Quantifying Performance & Operational Impacts Current Speed: Actual vs. Forecasted 5 4 3 2 1 0 Effectiveness Costs Minimal Costs Actual Minimal Costs Opportunity Costs 0 1 2 3 4 5 Forecasted 2 No Go Threshold • Need to assess the costs imposed by forecast errors: • Opportunity costs • Predicted conditions worse than actual • Effectiveness costs • Actual conditions worse than predicted

  12. NSW METOC Metrics: Project Components • Developing a Metrics System • Three Main Components of the NSW METOC Metrics System 1. Data Collection & Analysis System 2. Operational Modeling & Analysis 3. Coordination & Collaboration

  13. NSW METOC Metrics • Outline • Why Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status & Way Ahead

  14. NSW METOC Metrics: Data Collection, Analysis, & Reporting

  15. NSW METOC Metrics: Data Collection, Analysis, & Reporting • Component 1 System Modules • Online Data Collection and Metrics Analysis System • Module 1: Collects forecast data • Module 2:Collects observed METOC conditions data • Via Sensors • Module 3:Collects observed METOC conditions data • Via the Warfighter • Module 4:Collects mission plans and outcomes data • Via the Warfighter • Module 5:Analyzes data, calculates and displays metrics • Module 6:Administers metrics system • Module 7: Sets mission impacts via thresholds NSW METOC Metrics, lpiepkorn@spa.com, 15

  16. Forecaster Selects Mission Components NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 1: Collects forecast data – Ready for data collection • Forecast Manager • View and manage current forecasts • Allows forecast editing • Allows recording of observations • Forecast Builder • Forecasts built online based on user-selected mission components and governing METOC thresholds Access at: http://www.clearscienceinc.com/metrics/nsw_metrics/nsw_forecast_builder/Username: metrics; Password: 915metrics* NSW METOC Metrics, lpiepkorn@spa.com, 16

  17. NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 1: Collects forecast data - Ready for data collection • Forecast Builder Input • METOC elements are chosen based on the mission components selected (e.g., MOUT) • Allows forecaster to only focus on pertinent METOC elements, not colors/thresholds • Forecast Builder Output • Colors are computed based on forecast values and governing thresholds for mission components • Automatically records forecast data for metrics calculations • Forecasts can be inserted into PPT or other briefing media • No Additional Work for the Forecaster! NSW METOC Metrics, lpiepkorn@spa.com, 17

  18. NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 2:Collects data on observed METOC conditions (sensor data) -Ready for data collection • Freeform entry of observational data • Allows multi-source entry (good for SOCAL) • Eventual goal is to automate ingestion of observed data • Only requests data on METOC elements used in forecast

  19. NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 3 & 4:Collect data on mission outcomes, METOC impacts, mitigating actions - Ready for data collection • Collects data on: • Mission plans • Role of METOC forecasts and mitigation recommendations • Records on-scene observations of METOC conditions, METOC impacts on mission, and mitigating actions taken • Standardizes customer feedback process • Designed for quick, intuitive entry • Less than 1 minute for most missions

  20. False Alarm Rate, Jan-Apr 2009 60 50 40 30 20 10 0 FAR (%) Operational Impacts, ST-X (SOCAL Training), 2009 Number of Missions NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 5:Analyzes data, calculates and displays metrics results • Metrics shown on-demand by mission type, unit, time period, METOC element, etc. • Metrics displayed based on user permissions (e.g., forecasters, directorate) • Metrics displayed graphically or in tabular form • Metrics displays customized to meet users requirements NOTIONAL DATA NOTIONAL DATA NSW METOC Metrics, lpiepkorn@spa.com, 20

  21. Mission Components METOC Elements Established Thresholds NSW METOC Metrics: Component 1 Data Collection, Analysis, and Reporting • Module 7: Sets mission impacts based on established thresholds - Completed • Sets impacts for forecasted and actual environment • Impacts set for each mission component • Based on established, governing thresholds • Allows top level adjustments to thresholds • Viewable as a quick reference • Improves forecaster awareness of thresholds • Individual threshold settings has been requested • By user login

  22. NSW METOC Metrics • Outline • Why Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status & Way Ahead

  23. NSW METOC Metrics: Component 2 Operational Modeling & Analysis • Objectives • Model NSW operations, especially METOC inputs to planning process and impacts on operational outcomes • Assess operational implications of METOC performance • Develop quantitative operational benchmarks and thresholds for the performance of METOC products through sensitivity testing • Integrate data collected through Component 1 to evaluate support performance • Produce overall METOC support assessment Operations Analysis Links METOC Conditions and Forecast Performance to Customer Outcomes

  24. NOSWC & Operator Input Model as Laboratory METOC Data Collection Performance Thresholds Model Development Model Design • METOC Support Process • Products • Timelines • NSW Decision Process • Inputs • Timelines • Mission Planning Process • Etc… New Support Products & Concepts Support Evaluation Performance Benchmarks Cyclical Metrics-Based Analysis of METOC Performance Updated benchmarks NSW METOC Metrics: Component 2 Operational Modeling & Analysis

  25. NSW METOC Metrics: Component 2 Operational Modeling & Analysis Step 1. Model Design Step 2. Model Development

  26. Doctrinal Swimmer Current Threshold: 1 knot Incorrectly Predict Go Conditions Correctly Predict NoGo Conditions Effectiveness Costs Incorrectly Predict NoGo Conditions Correctly Predict Go Conditions Opportunity Costs NSW METOC Metrics: Component 2 Operational Modeling & Analysis Step 3. Performance Thresholds: Regions of Acceptable Performance Current Speed (kt)

  27. Doctrinal Swimmer Current Threshold: 1 knot Incorrectly Predict Go Conditions Correctly Predict NoGo Conditions Effectiveness Costs Incorrectly Predict NoGo Conditions Correctly Predict Go Conditions Opportunity Costs NSW METOC Metrics: Component 2 Operational Modeling & Analysis Step 3. Performance Thresholds: Regions of Acceptable Performance Current Speed (kt) Ops Analysis Models will Allow us to Define These Regions of Acceptable, Less-Than-Ideal Performance Through Sensitivity Testing of the Boundary Cases

  28. Step 4. Support Evaluation Mission Outcome Probability • Step 4. Support Evaluation • Data Collected by the Metrics System is Integrated • Operational Impacts Calculated • Informs Leadership of the Current Support Performance for Mission-Product Pairs NSW METOC Metrics: Component 2 Operational Modeling & Analysis Collected Data Ops Modeling

  29. Performance Benchmarks NSW METOC Metrics: Component 2 Operational Modeling & Analysis Step 5. Support Evaluation Collected Data Ops Modeling • Step 5. Performance Benchmarks • Current Support Performance Measured Against Benchmarks • Process is Repeated Across the Range of NSW Mission Types and METOC Products UAV Ops

  30. NOSWC & Operator Input Model as Laboratory METOC Data Collection Model Design • METOC Support Process • Products • Timelines • NSW Decision Process • Inputs • Timelines • Mission Planning Process • Etc… NSW METOC Metrics: Component 2 Operational Modeling & Analysis Performance Thresholds Model Development As New Data Becomes Available, the Benchmark-Evaluation Process is Repeated New Support Products & Concepts Support Evaluation Performance Benchmarks Cyclical Metrics-Based Analysis of METOC Performance Updated benchmarks

  31. NSW METOC Metrics • Outline • Why Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status & Way Ahead

  32. NSW METOC Metrics: Component 3 Coordination and Collaboration • Collaborate with related projects, including: • - Other METOC metrics efforts • - NSW Characterization of the Battlespace • - Other NOSWC/NSW analysis and assessments efforts • Deliverables: • - Collaborative databases • - Coordinated analyses of processes, products, and impacts • - Coordinated evaluations of METOC support • - Recommendations for improving support • - Recommendations for improving NSW operations NSW METOC metrics efforts need to be consistent, and coordinated, with all related efforts (e.g., end-to-end NSW assessment).

  33. NSW METOC Metrics: Component 3 • What It Will Take To Win at NSW Metrics, in General • Unwavering top level support. All levels must know that this program is supported • Strong leadership at subordinate that supports upper level goals • Involvement of rank and file support providers in improvement of metrics process • Promotion of an NSW metrics culture --- a culture of objective, quantitative, data driven, self assessment and improvement • Coordination and collaboration with related efforts, especially regarding: • Development of common methods for data collection, analysis, and modeling • Coordination of data collection and modeling efforts • Data sharing • Sharing of lessons learned • Development of coordinated long range plans Success in NSW METOC metrics will increase effectiveness of NSW operations.

  34. NSW METOC Metrics • Outline • Why Metrics? • METOC Metrics Methodology • Component 1: Data Collection, Analysis, and Reporting • Component 2: Operational Modeling and Analysis • Component 3: Coordination and Collaboration • Current Status & Way Ahead

  35. NSW METOC Metrics: The Way Ahead • Data Collection, Analysis, and Reporting • Demo release 2-08 (Q3 FY08) • Complete modules 3 and 4 (warfighter obs & outcomes) • Awaiting MSC input for module completion • Deploy data collection to METOC Metrics Web - MMW (ASAP) • Awaiting final approval/implementation • ATOS2 environment (FNMOC) • Common landing pad for multiple METOC metrics projects • Deploy metrics display and admin modules (5 and 6) to MMW • Q1 FY09 • Complete online training for system users • Q1 FY09 • Ops Analysis and Modeling • Finalize, develop, and test operational mission model for SBU operations (insertion/extraction) – Scoping model available: Awaiting NOSWC Input • Operational timelines & METOC support inputs • Link to Component 1 modules • Fully understand the current mission planning systems in use (SOMPE) • Complete a full analysis cycle for SBU operations across a range of METOC features by end FY08 • Explore sensitivities to determine ranges of acceptable METOC support performance • Determine detailed benchmarks for METOC support to SBU operations

  36. NSW METOC Metrics: Metrics Websites Questions? NIPR: http://met.nps.edu/metrics/navy/nsw_metrics/ SIPR: http://web.ntsstl.nps.navy.smil.mil/metrics/nsw_metrics/

  37. Naval Postgraduate School Department of Meteorology 254 Root Hall, 589 Dyer Road Monterey, CA 93943-5114 Fax: 831-656-3061 Tom Murphree, Ph.D. (project lead) Email: murphree@nps.edu murphrjt@nps.navy.smil.mil Phone: 831-656-2723 Clear Science, Inc. 7801 Lonestar Rd Suite #17 Jacksonville, FL 32211 Fax: 904-379-9704 Bruce W. Ford Email: bruce@clearscienceinc.com fordbw@tsc-jax.navy.smil.mil Phone: 904-379-9704 • Systems Planning and Analysis, Inc. • 2001 N. Beauregard Street • Alexandria, VA 22311 • Fax: 703-399-7365 • Paul Vodola, Ph.D. • Email: pvodola@spa.com • Paul.Vodola_Contractor@spa.dtra.smil.mil • Phone: 703-399-7225 • Luke Piepkorn • Email: lpiepkorn@spa.com • Phone: 703-399-7239 • Naval Research Laboratory • Oceanography Division • NRL Code 7322 • Stennis Space Center, MS 39529-5004 • Fax: 228-688-4759 • Richard Allard • Email: allard@nrlssc.navy.mil • Phone: 228-688-4894 • Kacey Edwards • Email: kedwards@nrlssc.navy.mil • Phone: 228-688-5077 NSW METOC Metrics: Contact Information NSW METOC Metrics, lpiepkorn@spa.com, 37

More Related