Coupler Design Issues from the Modular Ocean Data Assimilation Project

1. Coupler Design Issues from the Modular Ocean Data Assimilation Project Dr. Richard Loft Computational Science Section Scientific Computing Division National Center for Atmospheric Research Boulder, CO USA

2. Outline • Project Introduction • goals / pariticipants / scope • http//:iom.asu.edu • IOM: weak variational data assimilation • Inputs/outputs • Data components • Functional components • IOM Coupling design • Strategy • Details • Role of Domain Specific Languages (DSL)

3. Modular Ocean Data Assimilation • NSF ITR/AP • Focus on the OSU Inverse Ocean Model(IOM) system for ocean data assimilation. • Variational data assimilation system • Weak assimilation • Iterative algorithm for solving nonlinear assimilation problems • Suite of diagnostics (posterior error statistics) • 20 years in development by Bennett, et al.

4. Objectives of MODA Project • Enhance the IOM System with modern information technology. • Modular software design • Hybrid parallel implementation • Coupled model strategy • Automated code generation • Distribute it to the ocean modeling community. • 5 ocean modeling partners • Facilitate application of the system to coastal oceans, ocean basins and the global ocean.

5. NSF ITR MODA Collaboration Institution Researcher Arizona State U. Muccino U. Colorado Moore NCAR Loft NCSA Baker U. North Carolina Leuttich Oregon State U Bennett, Egbert, Erwig Rutgers U. Arango, Haidvogel UCSD ( Scripps) Cornuelle, Miller

6. IOM Team Members • IOM-PEZ Parallelism • PI: Rich Loft, NCAR • Component: parallel super and Infra-structure (hybrid F90 module framework) • Domain Specific Languages • PIs: Martin Erwig & Zhu Fu, OSU • Component: automated code generator (Haskell) • Visualization • Pis: Polly Baker, NCSA/IU • Component: VisBenchToolVisualization Software

7. Participating Model Teams • PEZ (Primitive Eq. Z-coordinate)model • Description: 3D, free-surface, z-coordinate • Bryan-Cox • Grid: spherical polar, “B” grid • Language: F90 • Parallelism: SPMD, MPI • ROMS (Regional Ocean Model System) • Description: 3D, free-surface, S-coordinate • Grid: Horizontal orthogonal curvilinear coordinate, “C” grid • Language: F90 • Parallelism: SPMD, OpenMP, MPI, SMS

8. Participating Model Teams • SEOM (Spectral Element Ocean Model) • PI: Dale Haidvogel, Rutgers • Description: 3D, free-surface, S-coordinate • Grid: h-p finite element, quadrilateral element • Language: F90 • Parallelism: SPMD, MPI • ADCIRC (Advanced Circulation) model • PIs: Julia Muccino, ASU; Rich Luettich, UNC • Description: 3D, free-surface, sigma-coordinate • Grid: finite element, linear triangular element Language: F90 • Parallelism: SPMD, MPI

9. Participating Model Teams • OTIS (Internal Tides) • PI: Gary Egbert, OSU • Description: Laplace tidal equations plus 10 years of TOPEX/Poseidon • Description: Solid Earth mageto-tellurics (magnetic field of earth’s crust)

10. Data Assimilation Checklist: Inputs • observing system • e.g. measured sea surface temperatures, isotherm depths and surface winds • dynamics • e.g. the hydrostatic primitive equations • hypothesis concerning the error • covariances of errors in the • initial conditions • boundary conditions • and forcing • estimator • space-time integrated weighted sum of squared errors • optimization algorithm • iterative indirect representer algorithm

11. Data Assimilation Checklist: Outputs • state estimate • hindcast over the period in the data collection • space-time fields for state variables • data residuals, dynamic residuals • Space-time minimum residuals • posterior error statistics • space-time covariances • test statistics • chi square internal consistency tests • model improvements • e.g. suggested from the distribution of errors • observing system assessment

12. Inverse system data components… • Vector of data to be assimilated • Trajectories • multi-variate physical space-time fields • generated by tangent linear/adjoint model • user prescribed • Parameters • covariance matrices • user prescribed

13. Inverse system functional components… • Pure data space components • iterative solver • Physical space components • tangent linear/ adjoint models • Components that map between the two spaces. • measurement operator • impulse operator • Looks like a coupled system

14. Coupler macro design: IOM + coupler + ocean models Fwd & adjoint Ocean model IOM grids State Data Data space Physical space • Ocean Models: • PEZ • ROMS • SEOM • ADCIRC • OTIS • Data Assimilations: • IOM core Coupler • Both spaces • Impulses / measurements: • DSL autogeneration • Control info

15. Data / Physical Space Interactions Interpolations IOM PEs model PEs IOM Iterative Solve (data space) Ocean Model Time Integration (physical space)

16. IOM System Pseudo-code d = data uF = meas. first guess h = misfit  = representer coefficients Key: IOM(){ read d first( ; uF) h = d - uF solve(h ; ) final(  ; ) } ^ ^ ^ Physical Space / user supplied Both / Auto-generated Data Space/IOM supplied Subroutine Notation: foo( in ; out ) Pure Data Space…

17. IOM First Guess Code first(;uF){ read F tanlin( F(x,t) ; UF(x,t) ) measure(UF(x,t) ; uF) } F(x,t) = Initial / boundary / interior forcing UF(x,t) = response to F uF = measured first guess

18. IOM Inner Solver Code ^ (x,t) = adjoint (x,t) = forward D(x,t) = Dirac comb  = representer coefs  = measured   = scratch  = scratch solve(h ;  ){ = h while (  ≠ ){ comb(  ; D(x,t)) adjoint( D(x,t) ; (x,t) ) convolve( (x,t) ; (x,t)) tanlin( (x,t) ; (x,t) ) measure((x,t) ; ) stabilize( ,  ; ) call precongrad(  ; ) }  =  } ^ ^

19. IOM Final Sweep ^ final(  ; ){ comb(  ; D(x,t)) adjoint( D(x,t) ; (x,t) ) convolve((x,t) ; (x,t)) tanlin((x,t) + F(x,t); U(x,t) ) write U(x,t) } D(x,t) = Dirac comb U(x,t) = optimal estimate (x,t) = optimal adjoint ^ ^ ^ ^

20. IOM System Architecture IOM Coupling Layer IOM Solver DSL Generated Code Tangent Linear Model Adjoint Model Parallel Infrastructure External libraries: MPI, NetCDF, …

21. IOM Parallel Module Support F90 Module Heirarchy: PEZ 9pt stencil Solver global sums stencils parallel IOM broadcast buffers process thread virtual topology

22. IOM Coupling Design Details • Key object: the observation • Simplest case: a point observation • is a distribution:  (-', -', t-t' ) • In general smeared out over space-time. • The observation (self describing) • F90 derived type • type of observation • units • Associated methods must be supplied for mapping observations onto gridded state variables.

23. IOM Coupling Design Details • Fortran/Unix specific (applications / code generator) • Preferred mode: separate executables. • IOM + coupler + ocean models (constant functions) • IOM-PEZ components currently merged into one executable • IOM supports parallel components (MPI/OMP/hybrid) • Fixed number of processors • MPI_SPAWN and MPI_SPAWN_MULTIPLE not supported on many major platforms (e.g. IBM) ). • Coupled component execution does not overlap. • Strategy adopted to enable easy interfacing with diverse ocean codes, with different internal forms of parallelism. • Solver checkpoint/restarts IOM system, no internal state saved on TLM/adjoint model side.

24. Automatic Generation of Model-Specific IOM Tools Martin Erwin and Zhe FuOregon State University Idea: • Specify ocean modeling tools once • Identify model-dependent parameters • Generate Fortran programs from specification and values for parameters • Ocean modelers: • select simulation tools • provide parameters for their models • obtain a customized variational system

25. p1 = ......pk = ... p1 = ......pk = ... module tool_PEZ ... module tool_ROMS ... IOM DSL System Tool Specification ModelConfigurations p1 = ......pk = ... tool(p1,...,pk):obj[n] = ...... PEZ ROMS Compiler Fortran Code

26. IOM Developers Computer Scientists Ocean Modelers IOM-DSL-Modeler Interactions Configuration DSL p1 = ......pk = ... tool(p1,...,pk):obj[n] = ...... DSL-Compiler Fortran

27. Questions?