Hernan G. Arango, Rutgers University (arango@imcs.rutgers)

1. A Community Terrain-following Ocean Modeling System Hernan G. Arango, Rutgers University (arango@imcs.rutgers.edu) Tal Ezer, Pricenton University (ezer@splash.princeton.edu) FTP File: TOMS.tar

2. COLLABORATORS • Bennett et al. (FNMOC; OSU) • Chassignet / Iskandarani et al. (RSMAS) • Cornuelle / Miller (SIO) • Geyer (WHOI) • Hetland (TAMU) • Lermusiaux (Harvard) • Mellor (Pricenton) • Moore (U. Colorado) • Shchepetkin (UCLA) • Signell (SACLANT; USGS)

3. OTHER COLLABORATORS • Chao / Song (JPL) • Preller / Martin (NRL) • Naval Operational Community • POM Ocean Modeling Community • ROMS / SCRUM Ocean Modeling Community

4. OBJECTIVES • To design, develop and test an expert ocean modeling system for scientific and operational applications • To support advanced data assimilation strategies • To provide a platform for coupling with operational atmospheric models (like COAMPS) • To support massive parallel computations • To provide a common set of options for all coastal developers with a goal of defining an optimum coastal/relocatable model for the navy

5. APPROACH • Use state-of-the-art advances in numerical techniques, subgrid-scale parameterizations, data assimilation, nesting, computational performance and parallelization • Modular design with ROMS as a prototype • Test and evaluate the computational kernel and various algorithms and parameterizations • Build a suite of test cases and application databases • Provide a web-based support to the user community and a linkage to primary developers

6. CHALLENGE “The complexity of physics, numerics, data assimilation, and hardware technologyshould betransparent to the expert and non-expert USER”

7. TOMS KERNEL ATTRIBUTES • Free-surface, hydrostatic, primitive equation model • Generalized, terrain-following vertical coordinates • Boundary-fitted, orthogonal curvilinear, horizontal coordinates on an Arakawa C-grid • Non-homogeneous time-stepping algorithm • Accurate discretization of the baroclinic pressure gradient term • High-order advection schemes • Continuous, monotonic reconstruction of vertical gradients to maintain high-order accuracy

8. Dispersive Properties of Advection 5/2 Parabolic Splines 2 10 Vs Finite Centered Differences 6 3/2 8 K(k) • x 4 1 2 1/2 /4 3/4 /2 kx

9. TOMS SUBGRID-SCALE PARAMETERIZATION • Horizontal mixing of tracers along level, geopotential, isopycnic surfaces • Transverse, isotropic stress tensor for momentum • Local, Mellor-Yamada, level 2.5, closure scheme • Non-local, K-profile, surface and bottom closure scheme

10. TOMS BOUNDARY LAYERS • Air-Sea interaction boundary layer from COARE (Fairall et al., 1996) • Oceanic surface boundary layer (KPP; Large et al., 1994) • Oceanic bottom boundary layer (inverted KPP; Durski et al., 2001)

11. Boundary Layer Schematic 1. ABL 2. SBL 3. BBL 4. WCBL L o n g w a v e Shortwave O E v a p H H O H H

12. TOMS BOUNDARY LAYERS • Air-Sea interaction boundary layer from COARE (Fairall et al., 1996) • Oceanic surface boundary layer (KPP; Large et al., 1994) • Oceanic bottom boundary layer (inverted KPP; Durski et al., 2001) • Wave / Current / Sediment bed boundary layer (Styles and Glenn, 2000) • Sediment transport

13. TOMS MODULES • Lagrangian Drifters (Klinck, Hadfield) • Tidal Forcing (Hetland, Signell)

14. Gulf of Maine M2 Tides Surface Elevation (m)

15. TOMS MODULES • Lagrangian Drifters (Klinck, Hadfield) • Tidal Forcing (Hetland, Signell) • River Runoff (Hetland, Signell, Geyer)

16. Hudson River Estuary 30 -5 25 -10 20 Salinity (PSS) Depth (m) -15 15 -20 10 -25 5 25 5 15 20 10 Distance (km)

17. TOMS MODULES • Lagrangian Drifters (Klinck, Hadfield) • Tidal Forcing (Hetland, Signell) • River Runoff (Hetland, Signell, Geyer) • Biology Fasham-type Model (Moisan, Shchepetkin) • EcoSim Bio-Optical Model (Bissett)

18. TOMS TESTING • Systematic evaluation of numerical algorithms via robust test problems • Data/Model comparisons • Study optimal combination of algorithmic options for various coastal applications • Documentation of testing procedures

19. TOMS CODE DESIGN • Modular, efficient, and portable Fortran code (F77+, F90) • C-preprocessing managing • Multiple levels of nesting • Lateral boundary conditions options for closed, periodic, and radiation • Arbitrary number of tracers (active and passive) • Input and output NetCDF data structure • Support for parallel execution on both shared- and distributed -memory architectures

20. TOMS PARALLEL DESIGN • Coarse-grained parallelization

21. PARALLEL TILE PARTITIONS 8 x 8 Ny } } Nx

22. TOMS PARALLEL DESIGN • Coarse-grained parallelization • Shared-memory, compiler depend directives MAIN (OpenMP standard) • Distributed-memory (MPI; SMS) • Optimized for cache-bound computers • ZIG-ZAG cycling sequence of tile partitions • Few synchronization points (around 6) • Serial and Parallel I/O (via NetCDF) • Efficiency 4-64 threads

23. TOMS DATA ASSIMILATION • Nudging • Optimal Interpolation (OI) • Tangent linear and Adjoint algorithms • 4D VARiational data assimilation (4DVAR) and Physical Statistical Analysis System (PSAS) algorithms • Inverse Ocean Modeling System (IOMS) • Ensemble prediction platform based on singular value decomposition • Error Subspace Statistical Estimation (ESSE)

24. + ESSE Flow Diagram ESSE Smoothing via Statistical Approximation ^ DY0/N Field Initialization Central Forecast ^ ^ Y0 Ycf(-) Ymp(-) Shooting Sample Probability Density Measurement Model OA via ESSE Measurement Model Select Best Forecast Options/ Assumptions Mean SVDp Performance/ Analysis Modules Perturbations Minimum Error Variance Within Error Subspace (Sequential processing of Observations) Adaptive Error Subspace Learning + Scalable Parallel Ensemble Forecast Error Subspace Initialization Normalization Peripherals Analysis Modules Key Convergence Criterion Continue/Stop Iteration Breeding Field Operation Assumption DE0/N + DP0/N - - + Most Probable Forecast + Synoptic Obs A Posteriori Residules dr (+) Historical, Synoptic, Future in Situ/Remote Field/Error Observations d0R0 + - - Data Residuals Measurement Error Covariance ^ d-CY(-) Ensemble Mean + + ^ eq{Yj(-)} Gridded Residules ^ Y(-) + - ^ ^ j=1 Y(+) Y(+) Y1 Yj Yq ^ - Y1 Yj Yq + 0 + - E(-) P(-) ^ - + 0 + + - +/- ^ E0 P0 j=q 0 uj(o,Ip) with physical constraints Continuous Time Model Errors Q(t) Ea(+) Pa(+) E(+) P(+)

25. PRESSURE GRADIENT FORCE • Density Jacobian Class (Blumberg and Mellor, 1987; Song 1998; Song and Wright 1998) • More Accurate • Error vanishes with linear density profiles • Pressure Jacobian Class (Lin 1998; Shchepetkin and McWilliams, 2001) • JEBAR consistent • Conserve Energy

26. Seamount Test Case (64 x 64 x 20) dx = dy = 8 km

27. Second Order Advection Scheme Models with 2nd order advection scheme POM ROMS Surface Elevation Anomaly Stream Function Anomaly

28. Advection Schemes in ROMS (Seamount Case) V Second Order Centered Third Order Upstream Bias Fourth Order Centered

29. Pressure Gradient Errors POM POM (6th order) V (cm/s) U (cm/s) ROMS X (km)

30. Relative CPU per time step Percentage

31. RESULTS (YEAR 1) • Build TOMS from ROMS prototype • Mellor-Yamada, level 2.5 • Passive and active open boundary conditions • Tidal forcing • River runoff • Lagrangian drifters • Data assimilation • Inter-comparison between POM and ROMS • Evaluation of time-stepping, advection, and pressure gradient algorithms • Initial development of TOMS web site

32. Initial web page: www.aos.princeton.edu/WWWPUBLIC/ezer/TOMS

33. TRANSITION PATHS • To Be Determined !!! • Potential Users: • NAVO • FNMOC • NOAA • USCG

34. PUBLICATIONS • Chassignet et al., 2000: Damee modeling review • Ezer, 2000: Mixed-layer evaluation • Ezer and Mellor, 2000: POM Damee application • Haidvogel et al., 2000: ROMS Damee application • Malanotte-Rizzoli et al., 2000: ROMS Damee • Mellor, 2001: Improved turbulence scheme • Mellor et al., 2001: Generalized vertical coordinate