METOC Metrics for ASW: VS07 Preliminary Findings

1. METOC Metrics for ASW: VS07 Preliminary Findings Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu murphrjt@nps.navy.smil.mil Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com fordbw@tsc-jax.navy.smil.mil A Research, Development, and Transition Project Funded by PMW180 Presented to CAPT Jim Berdeguez CNMOC 14 August 2007 Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

2. Goals • Conduct real-world test of prototype data collection process • Collect and analyze exercise level data • Refine verification and impact metrics • Refine plans for overall ASW METOC metrics project Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

3. Methods • Collect data at CFT74, KHK, JCS, NIM, MOCC, RBC collected data on: • Forecasts/analyses for all three BonD tiers • Verifying observations • Product usage • Recommendations • Customer performance/impressions • METOC impacts on planning, execution, and outcomes • Collect and analyze additional data on ocean and acoustic models, performance surfaces, and customer performance (R&A) • Develop revised data collection and analysis process for overall ASW METOC metrics project Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

4. METOC Data and Performance *Daily commodore’s brief nc – Not computed due to low number of verifying observations # of sensor performance predictions provided to at-sea staffs and CTF74 = 160 Source: VS07 NOAT data collection forms Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

5. NOAT Recommendations Source: VS07 NOAT data collection forms Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

6. NOAT Recommendations *Commonly reported by CTF74 due to CTF74 not being in a position to mandate recommendations Source: VS07 NOAT data collection forms Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

7. Post-Ex NOAT Survey • Extensive post-VS07 survey sent to NOAT metrics collectors and forecasters on 12Aug07 • Half of expected responses received as of 13Aug07 • Sample responses • Most accurate forecasts: SLD, SST • Least accurate forecasts: COF, passive ranges • Most valued forecast: sensor performance (100% agreement) • Customer satisfaction with sensor performance predictions: Very satisfied • Response to NOAT recommendations: Very positive • Most positive recommendation outcomes: High rate of contacts resulting from sensor placement recommendations • Most negative recommendation outcomes: LFA recommendations made without prior coordination with LFA platform • Use and impacts of products in commodore briefs: • TOFA: high • Performance Surfaces: high • Decision Layer Products: very low Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

8. Post-Ex NOAT Survey i. Shining moments: • “[NOAT] fully integrated into planning process” • “every slide [in brief] had force impacts” • “convincing DESRON to…adapt screen to the environment” • “we helped with the success of the exercise” j. Biggest busts: • “very poor start” • “slow at catching dropping SLD” • “too much faith in the model” • “[overly] optimistic screen formation” Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

9. Preliminary Findings on Data Collection • Data collection was labor intensive • Collectors’ skill in gathering data increased as exercise progressed • Identified opportunities for automation of some data collection • Challenges in verifying a number of METOC predictions (e.g., sensor performance ranges) • Identified challenges in collecting customer performance data • Customer may not collect complete performance data • Conflicting sources for some data • Refined list of key data to be collected and metrics to be calculated Note: These findings consistent with those from our other METOC metrics projects. Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com

10. Next Steps • Complete analysis of data collected from NOATs and RBC • Coordinate with related efforts: R&A analysis, model skill analysis • Prepare METOC metrics data collection and analysis report, including lessons learned on metrics process • Participate in VS07 review • Incorporate lessons learned into overall ASW METOC metrics project Murphree and Ford., VS07 Preliminary Findings, Aug, 2007, murphree@nps.edu, bruce@clearscienceinc.com