ASW METOC Metrics: VS07 Data Collection

1. ASW METOC Metrics: VS07 Data Collection Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu Paul Vodola, Matt McNamara, and Luke Piepkorn Systems Planning and Analysis (SPA) pvodola@spa.com Brief for VS07 Training 18-22 June, 2007

2. Why Bother With Metrics? • 1. Develop and transition to operational use systems for: • a. collecting data from METOC units and their customers • b. quantifying METOC performance and impacts on customer • operations • c. modeling and predicting impacts of METOC support on • war fighting operations • 2. Identify methods for improving quality and efficiency of METOC support. • 3. Recommend: • a. focus directions for METOC resources • b. methods to incorporate METOC into OPNAV assessments Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

3. Definitions • Metrics: Objective, quantitative, data based measures of products and services. Examples: • Metrics of product quality • Metrics of effects of products on customers • METOC Metrics: Metrics of METOC organization’s products and impacts. Two main types: • Performance metrics: metrics of capacity, readiness, quality, efficiency / return on Investment • Impacts metrics: impacts on warfare customer operations (planning, execution, post-assessment) • Methods for Generating METOC Metrics •  Collect / analyze real world data on METOC and customer ops •  Model METOC and customer ops Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

4. Process for Developing Metrics of METOC Impacts on Military Operations Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

5. Hierarchy of Metrics Higher Level (Navy-wide SLD accuracy) Metric Symposium Focus Space CNMOC/FleetMetrics Larger Spatial and/or Temporal Scale (Exercise forecast location) Directorate Metrics NOAC Metrics Performance(Temperature and salinity accuracy) Impacts (Number of positively identified submarines) Exercise Metrics NOAT Metrics Ind. Forecast Metrics Smaller Spatial and/or Temporal Scale (Point forecast location) Lower Level (MOATs SLD accuracy) Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

6. The Eventual Collection System Operational Outcomes Operational Plans METOC Forecasts * METOC Observations Operational Performance Metrics METOC Performance Metrics Metrics of METOC Impacts on Operational Performance We apply this process to both real world data and output from a military mission model. * or other products Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

7. VS07 Opportunity • Offers the opportunity to simulate a larger metrics system by collecting and analyzing data as an established metrics program would • Offers an excellent opportunity to develop, test, and improve data collection and analysis process from multiple perspectives: • NOAT • RBC • MPRA • Exercise • Types of data to collect • Performance data (METOC and customer) • Impacts data • Process data • Metrics collection process • METOC support process Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

8. VS07 Data Collection Sites • Stationing of Data Collectors in Priority Order • Aboard each CV/ASWC NOAT • At RBC • Embedded with CTF-74 • Embedded with deployed MPRA TSC (MOCC) • Embedded with 3rd Fleet staff Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

9. General Data Collection Plan • Aboard each CV • Mimic NOAT data collection role by collecting: • Discrete forecast data • Verifying in situ data • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. • At RBC • Monitor watch officer actions and log for potential data collection • Monitor NOAT interactions • Observe product generation process • Embedded with deployed MPRA TSC (MOCC) • Mimic MPRA data collection by collecting: • Data from each GREEN and PURPLE • Discrete forecast elements • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

10. General Data Collection Plan (continued) • Embedded with each major staff • Mimic NOAT data collection role by collecting all: • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

11. General Data Collection Methods • Each data collector will: • Be thoroughly trained on data collection objectives prior to VS07 (e.g., this training!) • Record data on specially-prepared paper forms and spreadsheets • Maintain contact with other data collectors via twice-daily chats (hosted by the RBC) • Submit daily spreadsheets to RBC for archival • Document the METOC support process used • Document collection process used • Archive chats • Stay vigilant regarding the overall goals and methods • Record observations and suggestions • Recommend improvements to collection forms/process • Assess how the metrics interface and process will impact future deployed NOATs • Learn more about the METOC support process • Strive to understand the ASW warfighter and how success is measured • Ask questions! Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

12. Post VS07 Activities • Gather data collectors to discuss findings • Issue data collection report of key findings to ASW metrics executive committee • Propose changes to ASW metrics project data collection plan • Collate data and compute metrics (NPS student?) • Investigate potential of sanitizing data for ease of analysis • Assemble exercise level data • Evaluate exercise as a whole • Incorporate data collection and process information in operational modeling effort • Document our process, successes, and failures -– capture lessons learned Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

13. VS07 Metrics Data Collection and You? • Go about the support process as usual • Indulge the data collectors when questions are asked • They may have to learn about the process • Look for impacts that METOC information has on your customer • Positive and negative! • Formal (presented at brief or in a product) • Informal (in the p-way on the way to chow, phonecon, etc.) • We want to track the effect of all tactical recommendations • Followed or not? • If not, why not? • If so, what was the outcome? • If not, what was the outcome? • Report all impacts to your metrics collector! • VS07 metrics data is not: • Grading forecasts • Grading NOATs • Spying on you Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

14. Questions? Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com