METOC Metrics for ASW: VS07 Preliminary Findings

1. METOC Metrics for ASW: VS07 Preliminary Findings Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu murphrjt@nps.navy.smil.mil Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com fordbw@tsc-jax.navy.smil.mil A Research, Development, and Transition Project Funded by PMW180 and the ASW Directorate Brief Presented to CAPT Jim Berdeguez, CNMOC 14 August 2007 Updated 07 September 2007 Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

2. Acknowledgements ASW RBC team VS07 METOC metrics data collectors: • CDR(s) Tony Cox • LCDR(s) Tara Lambert • LT Eric MacDonald • LT Scott Parker • Bob Miyamoto Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

3. Goals • Conduct real-world test of prototype data collection process • Collect and analyze exercise level data • Refine verification and impact metrics • Refine plans for overall ASW METOC metrics project Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

4. Methods • Collect data at CFT74, KHK, JCS, NIM, MOCC, RBC collected data on: • Forecasts/analyses for all three BonD tiers • Verifying observations • Product usage • Recommendations • Customer performance/impressions • METOC impacts on planning, execution, and outcomes • Collect and analyze additional data on ocean and acoustic models, performance surfaces, and customer performance (R&A) • Develop revised data collection and analysis process for overall ASW METOC metrics project Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

5. METOC Data and Performance *Daily commodore’s brief nc – Not computed due to low number of verifying observations # of sensor performance predictions provided to at-sea staffs and CTF74 = 160 Source: VS07 NOAT data collection forms Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

6. METOC Performance Actual Predicted SLD: Actual vs. Predicted MPRA NOAT predictions and AXBT measurements of sonic layer depth (SLD) from Valiant Shield 2007 (VS07; preliminary data analysis).Note tendency to under-predict SLD by about 50 ft. Source: VS07 MPRA data collection forms Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

7. NOAT Recommendations Source: VS07 NOAT data collection forms Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

8. NOAT Recommendations *Commonly reported by CTF74 due to CTF74 not being in a position to mandate recommendations Source: VS07 NOAT data collection forms Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

9. Post-Exercise NOAT Survey • Extensive post-VS07 survey sent to NOAT metrics collectors and forecasters on 12Aug07 • Half of expected responses received as of 13Aug07 • Sample responses • Most accurate forecasts: SLD, SST • Least accurate forecasts: COF, passive ranges • Most valued forecast: sensor performance (100% agreement) • Customer satisfaction with sensor performance predictions: Very satisfied • Response to NOAT recommendations: Very positive • Most positive recommendation outcomes: High rate of contacts resulting from sensor placement recommendations • Most negative recommendation outcomes: LFA recommendations made without prior coordination with LFA platform • Use and impacts of products in commodore briefs: • TOFA: high • Performance Surfaces: high • Decision Layer Products: very low Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

10. Post-Exercise NOAT Survey i. Shining moments: • “[NOAT] fully integrated into planning process” • “every slide [in brief] had force impacts” • “convincing DESRON to…adapt screen to the environment” • “we helped with the success of the exercise” j. Biggest busts: • “very poor start” • “slow at catching dropping SLD” • “too much faith in the model” • “[overly] optimistic screen formation” Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

11. Preliminary Findings on Data Collection • Data collection was labor intensive • Collectors’ skill in gathering data increased as exercise progressed • Identified opportunities for automation of some data collection • Challenges in verifying a number of METOC predictions (e.g., sensor performance ranges) • Identified challenges in collecting customer performance data • Customer may not collect complete performance data • Conflicting sources for some data • Refined list of key data to be collected and metrics to be calculated Note: These findings consistent with those from our other METOC metrics projects. Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

12. Next Steps • Complete analysis of data collected via NOAT, MPRA, and RBC data collection forms; post-exercise NOAT survey forms; and anecdote collection. • Coordinate with related efforts: R&A analysis, model skill analysis. • Prepare METOC metrics data collection and analysis report, including lessons learned on metrics process. • Participate in VS07 review. • Incorporate lessons learned into overall ASW METOC metrics project. Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com

13. ASW METOC Metrics: Additional Information For additional information on the VS07 METOC metrics, and the overall ASW METOC metrics effort, see our ASW METOC Metrics NIPRnet and SIPRnet sites. NIPR: http://wx.met.nps.navy.mil/metrics/navy/asw_metrics/asw_symposium/index.php SIPR: http://web.ntsstl.nps.navy.smil.mil/metrics/asw_metrics Murphree and Ford, VS07 Preliminary Findings, 14Aug07, murphree@nps.edu, bruce@clearscienceinc.com