1 / 14

ASW METOC Metrics: VS07 Data Collection

ASW METOC Metrics: VS07 Data Collection. Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu Paul Vodola, Matt McNamara, and Luke Piepkorn Systems Planning and Analysis (SPA) pvodola@spa.com. Brief for VS07 Training

elewa
Download Presentation

ASW METOC Metrics: VS07 Data Collection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ASW METOC Metrics: VS07 Data Collection Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu Paul Vodola, Matt McNamara, and Luke Piepkorn Systems Planning and Analysis (SPA) pvodola@spa.com Brief for VS07 Training 18-22 June, 2007

  2. Why Bother With Metrics? • 1. Develop and transition to operational use systems for: • a. collecting data from METOC units and their customers • b. quantifying METOC performance and impacts on customer • operations • c. modeling and predicting impacts of METOC support on • war fighting operations • 2. Identify methods for improving quality and efficiency of METOC support. • 3. Recommend: • a. focus directions for METOC resources • b. methods to incorporate METOC into OPNAV assessments Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  3. Definitions • Metrics: Objective, quantitative, data based measures of products and services. Examples: • Metrics of product quality • Metrics of effects of products on customers • METOC Metrics: Metrics of METOC organization’s products and impacts. Two main types: • Performance metrics: metrics of capacity, readiness, quality, efficiency / return on Investment • Impacts metrics: impacts on warfare customer operations (planning, execution, post-assessment) • Methods for Generating METOC Metrics •  Collect / analyze real world data on METOC and customer ops •  Model METOC and customer ops Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  4. Process for Developing Metrics of METOC Impacts on Military Operations Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  5. Hierarchy of Metrics Higher Level (Navy-wide SLD accuracy) Metric Symposium Focus Space CNMOC/FleetMetrics Larger Spatial and/or Temporal Scale (Exercise forecast location) Directorate Metrics NOAC Metrics Performance(Temperature and salinity accuracy) Impacts (Number of positively identified submarines) Exercise Metrics NOAT Metrics Ind. Forecast Metrics Smaller Spatial and/or Temporal Scale (Point forecast location) Lower Level (MOATs SLD accuracy) Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  6. The Eventual Collection System Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  7. VS07 Opportunity • Offers the opportunity to simulate a larger metrics system by collecting and analyzing data as an established metrics program would • Offers an excellent opportunity to develop, test, and improve data collection and analysis process from multiple perspectives: • NOAT • RBC • MPRA • Exercise • Types of data to collect • Performance data (METOC and customer) • Impacts data • Process data • Metrics collection process • METOC support process Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  8. VS07 Data Collection Sites • Stationing of Data Collectors in Priority Order • Aboard each CV/ASWC NOAT • At RBC • Embedded with CTF-74 • Embedded with deployed MPRA TSC (MOCC) • Embedded with 3rd Fleet staff Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  9. General Data Collection Plan • Aboard each CV • Mimic NOAT data collection role by collecting: • Discrete forecast data • Verifying in situ data • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. • At RBC • Monitor watch officer actions and log for potential data collection • Monitor NOAT interactions • Observe product generation process • Embedded with deployed MPRA TSC (MOCC) • Mimic MPRA data collection by collecting: • Data from each GREEN and PURPLE • Discrete forecast elements • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  10. General Data Collection Plan (continued) • Embedded with each major staff • Mimic NOAT data collection role by collecting all: • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  11. General Data Collection Methods • Each data collector will: • Be thoroughly trained on data collection objectives prior to VS07 (e.g., this training!) • Record data on specially-prepared paper forms and spreadsheets • Maintain contact with other data collectors via twice-daily chats (hosted by the RBC) • Submit daily spreadsheets to RBC for archival • Document the METOC support process used • Document collection process used • Archive chats • Stay vigilant regarding the overall goals and methods • Record observations and suggestions • Recommend improvements to collection forms/process • Assess how the metrics interface and process will impact future deployed NOATs • Learn more about the METOC support process • Strive to understand the ASW warfighter and how success is measured • Ask questions! Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  12. Post VS07 Activities • Gather data collectors to discuss findings • Issue data collection report of key findings to ASW metrics executive committee • Propose changes to ASW metrics project data collection plan • Collate data and compute metrics (NPS student?) • Investigate potential of sanitizing data for ease of analysis • Assemble exercise level data • Evaluate exercise as a whole • Incorporate data collection and process information in operational modeling effort • Document our process, successes, and failures -– capture lessons learned Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  13. VS07 Metrics Data Collection and You? • Go about the support process as usual • Indulge the data collectors when questions are asked • They may have to learn about the process • Look for impacts that METOC information has on your customer • Positive and negative! • Formal (presented at brief or in a product) • Informal (in the p-way on the way to chow, phonecon, etc.) • We want to track the effect of all tactical recommendations • Followed or not? • If not, why not? • If so, what was the outcome? • If not, what was the outcome? • Report all impacts to your metrics collector! • VS07 metrics data is not: • Grading forecasts • Grading NOATs • Spying on you Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

  14. Questions? Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

More Related