Coastal Surface Piercing Profiler Issues

1. OOI Construction Completion Review: Coastal Surface Piercing Profiler UpdateRutgers University, New Brunswick, NJ

2. Coastal Surface Piercing Profiler Issues • Uncabled CSPP • Firmware and hardware failures occurred on deployments at Endurance and Pioneer. • At time of critical design review and production readiness review, it was noted that CSPP could not survive wave orbital velocities in winter near the bottom at 25 m depth off Oregon and Washington and that alternatives needed to be identified and explored

3. Coastal Surface Piercing Profilerinstances on OOI • cabled Coastal Surface Piercing Profiler (CSPP) • Endurance Oregon shelf site, 80 m depth • Power and data connection from RSN medium power (MP) Jbox • Winch overwrap caused stoppage in November 2014 • To be replaced with plan B (additional uCSPP if uCSPP design is proven out) • Uncabled CSPP • Endurance Oregon inshore site, 25 m depth; Endurance Washington inshore and shelf sites, 29 and 80 m depths • Pioneer Inshore and Central sites 91.5, 133 m depths • Testing at Oregon shelf site, deployments in spring 2015 underway

4. Endurance Array – Newport, OR, line • uses fixed and mobile assets to observe cross-shelf • and along-shelf variability in the coastal upwelling region off Oregon and Washington • provides an extended spatial footprint that encompasses a • prototypical eastern boundary current regime • joins with the RSN cabled infrastructure. uCSPP cCSPP

5. Endurance Array – Grays Harbor, WA, line • uses fixed and mobile assets to observe cross-shelf • and along-shelf variability in the coastal upwelling region off Oregon and Washington • provides an extended spatial footprint that encompasses a • prototypical eastern boundary current regime uCSPP uCSPP

6. Pioneer Array • 2 Locations • Inshore (91.5 m depth) • Central (133 m depth) • Paired with Surface Moorings • Requirements • Small footprint anchor • CSPP at ~100 m depth at Central site • Common design/components with Pioneer moorings • Deploy/recovery from a Global class ship • Deploy/recover from small ship

7. Panel recommendations from March 2015 review Un-cabled Surface Piercing Profiler. The Panel recommends OOI continue working with the manufacturer to conduct the 30-day field test of the system. If the tests indicate the piercing profiler system is not ready for acceptance, then the project must institute a plan B. While the 30-day test is underway, the project must provide NSF with a full contingency plan (Plan B) including all of the science impacts. Concurrent testing and planning will insure that if an alternative plan is needed an Engineering Change Request can be immediately executed. Cabled Surface Piercing Profiler. For this functional element the Panel recommends OOI go through the accepted descoping process, including the creation of an ECR and submission to the CCB for approval, prior to any further action. OSU has not yet provided NSF with a full contingency plan and plan to provide this at the time of the uCSPP test deployment review.

8. Outline: Uncabled Coastal Surface Piercing Profiler (uCSPP) Test Results • Planned Test • Deploy 2 profilers for 1 month • Actual Test and Deployments • All 5 profilers deployed, 4/5 successfully deployed, some stoppages due to environmental conditions (high waves) • Test Results & Analysis • Reliability Status • Capability Status

9. uCSPP temperature and salinity profiles from test deployment

10. Results Summary (as of 18 May 2015)

11. Assessment of uCSPP performance • Mechanical: • OOI’s main concern when starting this project and WET Labs’ main design focus for the past decade has been preventing the winch rope from breaking or over-wrapping. Neither has happened in OOI’s 14 ocean deployments, except the one time a CSPP stopped at the surface in rough seas for a week. • A winch quality error stopped the March Oregon Shelf deployment. A winch design error stopped the December Pioneer deployment. Both problems were identified and addressed immediately. • Summary: hard problems solved; easy-to-solve problems have bit us • Electrical: • No known problems except for the battery interface, which have been addressed • Design problems with the battery interface were corrected in early 2015. • Quality problems with battery enable connectors were corrected in March 2015. • WET Labs continues to work with Bluefin (battery vendor) to improve the interface • CE07SHSP has either a mechanical or electrical problem • Software: • Found bugs have been addressed promptly by WET Labs. • The system is deployable, but potential failure points remain such as premature profiling during deployment, unexplained resets, miscommunications with their Bluefin batteries, and reduction/handling of Iridium drop outs. WET Labs is addressing these issues. • Software testing has improved, but inadequate testing has in the past caused deployment failures and a lot of last-minute scrambling before deployments • The interface should prevent user error better. • CGSN Mooring side • Acoustic modem needs a better power supply • Interface could be easier to use

12. Implementation of wave avoidance algorithms • In the meantime, operators should look at the marine forecast daily and then set the profiler to delay surfacing as needed. This can be done via Iridium through the GUI or via acoustic modem through the mooring interface. In both pathways, one can either queue commands to be sent at a scheduled time, or one can manually communicate in real-time. • Avoiding rough seas • Requirement L4-CG-PR-RQ-198: A surface-piercing profiler shall assess the surface conditions and shall avoid surfacing of the sensor package in hazardous conditions. • From L3-CG-RQ-987: CGSN shall include design features that support prevention of damage to system components • WHE (wave height estimator) function exists, but it is turned off until base profiler functionality is better tested. Measures swell waves but the profiler cannot measure not wind, wind waves, nor can it remotely sense currents, so it would be better to use surface mooring data to evaluate conditions. Surface mooring data are not yet online. When they are available, a script can be put on the OMC to read those data, and then update profiler settings via acoustic modem.

13. Present status of uCSPP decision making • OSU SE Presented test results on 15 May to OSU PM, COL COTR, WHOI SE, WHOI operator, COL quality • Generated several actions from this meeting • OSU to distribute procedures for powering and using the coastal surface mooring acoustic modem to communicate with the uCSPP • OSU to add acoustic modem powering of 24 V instruments (including acoustic modem) to a mooring punch list • WHOI to develop punch list of uCSPP issues encountered on Pioneer • Remaining uCSPP issues to be tracked using Redminetroubleticketing • Update requirements verification compliance matrix (RVCM) based on test results • Did not have quorum to make go/no-go decision, will reschedule meeting with NSF, COL attendance • Will finish uCSPP Plan B as per NSF Panel Review instruction prior to this meeting

14. Inshore Winter plan B status • Winter Plan B is distinct from the uCSPP plan B. Winter plan B is the substitution of fixed and glider sensors for the uCSPP during the winter at the Oregon and Washington inshore (25m depth) sites • Sensors tested on October 2014 inshore mooring deployments – logged through DCL and transmitted to shore. • Initial plan B implementation was to purchase 2 additional shallow gliders with thrusters for use at Oregon and Washington inshore sites in winter • Alternatively could retrofit existing gliders with thruster, recommend not adding sensors etc. • Summer 2014 testing on this was inconclusive. We slowly ramped up thruster power to 3W in 25 m of water. This was not enough to generate significant additional speed. • Manufacturer subsequently recommended ramping to 9W and ramping more rapidly. • Additional testing has not been possible due to breakage of CTD on recovery and delays in its replacement.

15. Plan B Fixed Instruments • Tested CTD and FLORT at the bottom of the ISSM buoys during Oct 2014 deployment • Frames built with brackets for these additional instruments VELPT CTD ACOMM FLORT Instrument cage

16. Winter Plan B Science, Budget, & Schedule Impacts • Plan A will not work • At the uCSPP CDR, modeling shown that uCSPP will be thrashed against the seabed and its winch line will break during extreme winter storms • WET Labs has not designed the profiler for EA Inshore winter conditions • Plan B is a TWR hybrid glider plus fixed instruments on the bottom of the 25m ISSM buoys. • The coastal glider has CTD, FLORT, PAR, DOSTA, and ADCP. The uCSPP has all of these sensors plus NUTNR, and SPKIR and OPTAA. The propeller may enable the glider to profile closer to shore, where the 25m depth sites are located. The glider would also be powered by a standard buoyancy engine. • The fixed sensors, CTD and FLORT, would enable measurement of winter runoff plumes, which are too shallow to be picked up by the other fixed ISSM sensors on the near surface instrument frames (~6m depth). • Schedule: • Decision on gliders and fixed instruments sufficient for delivery for fall 2015 deployment • Added port for CTD & FLORT added to the ISSM end cap and ISSM buoy instrument cage includes room for these instruments. • Science Impact: • All instruments would not be co-located. • Profiles would not be vertical or co-located. • No profiles of NUTNR, SPKIR and OPTAA in winter at inshore site (which is often unstratified) • More profiles, but these would occur in bunches every few days instead of regularly every 6 hours. • Build Budget: Plan B units would be O&M 2nd builds, not MREFC • O&M 2nd build funded equipment budget: $401,037 for each of the two CSPP units • Plan B equipment: about $30k per glider retrofitted with thruster + $26,203 fixed instruments, mounts, connectors • O&M Steady State Budget: This Plan B will be equivalent to (or less than) annual O&M budget to Plan A. • The $55M budget constrains this Plan B to winter glider operations during 3 months of the October through April fall deployment. Anticipate two deployments with thrusters in the vicinity of inshore moorings. The first would go from the fall uCSPP recovery until after the fall transition of stratified to unstratified seas (~6 weeks). The second would go from 1-Mar to the spring uCSPP deployment, so that there is sampling during the spring transition. Following their plan B sampling, they would be retasked as inshore gliders on the Oregon and Washington shelves. That is, the Plan B gliders would not be deployed for all four months of winter near the inshore moorings, unlike the fixed instruments.

17. Backup slides: CSPP

18. Uncabled coastal surface piercing profiler (uCSPP) Endurance anchor and uCSPP

19. Uncabled CSPP timeline/overview • CSPP Award issued May 2013 • CSPP Critical Design Review September 2013 • First Article Test Nov – Dec 2013 (test unit upgraded to production configuration) • Production Readiness Review February 2014 • OSU received second unit prior to April 2014 • OSU deployed second unit 17 April 2014 at Oregon inshore site with communications and profiling failure on 15 May. Recovered 28 May. JIRA ticket submitted • FAR report delivered and by WET Labs July 2014 and failures addressed • WHOI received first unit 28 July, second shipped 10 Sep 2014 • WHOI encountered integration problems in Nov and Dec 2014 • WHOI deployed 1 Pioneer CSPP mooring 14 Dec 2014. It ceased communicating via Iridium on 21 Dec 2014, became adrift on 6 Jan and was recovered on 11 Jan 2015. • Nonconformance report submitted to Les Anderson • Second FAR reviewed 23 Feb 2015 with WHOI, COL, NSF, OSU • NSF Review and Panel Recommendations 9-10 March 2015 • Testing of 2 CSPP’s deployed 17 March 2015 and recovered 2 and 29 April 2015. • Preliminary test results presented to COL COTR and WHOI on 15 May 2015.

20. CSPP April 2014 Endurance deployment failure detailed in Failure Analysis Report 3111-00023 • Failure was due to a combination of user error and two firmware problems. • While attempting to offload decimated data files, the profiler was inadvertently set to make an Iridium call home only once every 34 times it reaches the surface. • Battery communications errors caused more than one profiler reset per profile. Eventually, the profiler did not recover after a reset. • When the profiler entered the breakaway state, it should have gone to the surface, and called home until its batteries ran down. It came up to the surface, but it did not call home due to the user error noted above. Instead of opening an Iridium session, it opened up its serial bulkhead connector port. It surfaced during a very low tide (1 ft. below MLLW), so it did surface properly. • WET Labs addressed failures noted above • profiler’s firmware updated to reduce resets and to recover from resets • WET Labs has and will continue to meet with the profiler’s battery vendor, Bluefin, to coordinate battery and battery charger firmware upgrades. • WET Labs made the telemetry mast longer and stronger to improve Iridium connections. • Updated firmware was tested either at sea or in a quarry for over 200 profiles each for three profilers, serial numbers WLP-002, WLP-003, and WLP-005 • All profilers updated with software and hardware updates prior to Dec 2014 WHOI Pioneer deployment

21. CSPP Dec 2014 Pioneer deployment failure detailed in Failure Analysis Report 3111-00024 • The failure was due to a component breaking off the winch control board. • This part, a DC-DC converter, had been soldered to the board via its three leads. • Without this part, the winch controller received no power. Following the failure, the main profiler controller turned on the winch system, and then stalled in its operation when it did not receive any serial replies from the winch controller. • Within this stuck state, the main controller left the winch system power ON while waiting for serial communications from the winch controller. As a result, the batteries ran down until the profiler shut down due to lack of power. Running down the batteries can damage them. • WET Labs addressed failure noted above • On all existing profilers and on any future production, the DC-DC converter will be epoxied to the circuit board. • The CSPP control can will be inspected and similar mitigation steps applied on other components as needed. • the controller firmware will be updated to detect winch communications failures and shut the profiler down without running down the batteries. A low power detection circuit will be added to the controller as a further mitigation against battery depletion. • WET Labs will inspect and make any required repairs to the winch system on the failed unit. • A four week test of two profilers with the updates will be performed at the Oregon shelf site (80 m depth) • All profilers will be updated with software and hardware updates prior to WHOI Pioneer and OSU Endurance deployments

22. Commanding & Querying via Acoustic Modem Intended commands to break out of breakaway state and then start profiling at an assigned time Automatic query from mooring to profiler of range, modem voltage, date, profiler state, future intent, last profile summary (size, time, end depth, number, voltage…) From:slerner@whoi.eduDate: May 9, 2015 at 17:15:13 EDTTo: Diana Wickman <dwickman@whoi.edu>Subject:Re: $PWETC,,,,,PST,2,23:00:00,05/09/2015*1F Diana, Here are the commands that are queued-up. We will replace thelast five with your updated ones for 00:00:00,05/10/2015. Theiridium schedule has been updated to every hour.$PWETC,,,BC,,DATE*48$PWETC,,,BC,,PFS*19$PWETC,,,BC,,PST*0B$PWETC,,,BC,,SUM*17$PWETC,,,,,BREAK*02$PWETC,,,,,BREAK*02$PWETC,,,,,BREAK*02$PWETC,,,,,BREAK*02$PWETC,,,,,BREAK*02$PWETC,,,,,PST,2,23:00:00,05/09/2015*1$PWETC,,,,,PST,2,23:00:00,05/09/2015*1$PWETC,,,,,PST,2,23:00:00,05/09/2015*1$PWETC,,,,,PST,2,23:00:00,05/09/2015*1$PWETC,,,,,PST,2,23:00:00,05/09/2015*1 These commands were never sent from the mooring to the profiler. They do not appear in the mooring’s acoustic modem log on the OMC.

23. Oregon Shelf Test Deployment I changed the profiling interval, but not the time of the next profile because I didn’t know when conditions would be OK again. With acoustic comms, I could have parked it a long time ahead, and then restarted profiling when conditions improved. WAVES o complete Iridium session x incomplete Iridium session problem evident ship lost comms to OMC surfacing requirement no Iridium comms WIND • Profiled 104 times over 29 days, 3/18-4/16. 4 times per day unless commanded to profile less often due to rough seas.Stopped in rougher than specified seas • L4-CG-PR-RQ-355: The CSPP shall have the capability to reach the surface in conditions of winds of 10 m/s and maximum significant wave heights of up to 3 meters • L4-CG-PR-RQ-78: The CSPP shall have the capability to reach the surface in the presence of a uniform current of 40 cm/sec • Not deployed next to a surface mooring, so commands to park could not be sent over acoustic modem

24. Oregon Shelf Test Deployment Stoppage Screw-down style = bad • One of the winch gears was the wrong kind of gear. • The gear was replaced with the right part; one that cannot slip because it attaches to the axle with a pin. • Our other 4 profilers have the right gear. Winch slipped with respect to level-wind causing the drum to be harder to turn Pin style = good

25. Oregon  Washington Shelf Test Deployment Oregon ORWA The Washington Inshore CSPP is still operating. We have had good acoustic comms with it from the start. Successful completion of Iridium sessions is not clearly a function of wind and waves. Need currents too. Deployed at the Oregon Shelf site 3/18. Recovered while operating 4/2. Redeployed in Washington 4/10. Stopped profiling 4/15. Status established via acoustic modem 5/5. 93 profiles spanning 27 days

26. Pioneer Central Attempted Deployment • Deployment attempted May 7 • OOI-built anchor failed right after deployment. Recovered. • Redeployment attempted May 7 • Prematurely started profiling during deployment following Iridium reconnection due to a firmware bug, which caused it to think it had broken away. • Operators tried but failed to send the profiler commands via acoustic modem to exit breakaway state and then start profiling (i.e. an OOI problem, not a WET Labs problem). • Recovered May 11. • Redeployment attempted May 13 • Profiler thought it had broken away because it was commanded to go to a Home Depth of 1 meter below the surface instead of 1 meter from the anchor. • Recovered immediately instead of sending restart commands via acoustic modem. • Upon recovery the winch rope was found to be scuffed while on land, ending operations.