Distributed Data Assimilation: A Case Study

1. Distributed Data Assimilation - A case studyAad J. van der SteenHigh Performance Computing GroupUtrecht University 1. The application 2. Parallel implementation 3. Model and experiments 4. Perspectives for distributed implementation

2. Theapplication - 1 • The application, Ensflow, assimilates ocean flow data into a stochasic ocean flow model. • Many realisations of the model with randomly distributed parameters forming an ensemble are run. • Perodically these runs are integrated with satellite data and an optimal ensemble average is computed. • The sequence of ensemble averages over time describes the development of the ocean's currents best fitting the observations.

3. Theapplication - 2 The region of interest in the southern tip of Africa: Data from the TOPEX/Nimbus satellite are used for the assimilation. Purpose is to understand the evolution of streams and eddies in this region.

4. The application -3 Because of the stochastic nature of the model many realisations of the model with slightly different parameter values are to be evolved. The observations of the top layer values are interpolated to a 251x151 grid. The ensemble members are allowed to develop independently for some time and combined to find the ensemble mean the best estimate for the model evolution without observations. With = matrix of field measurement covariances. = matrix of representer coefficients.

5. The application - 4 The model performs two computational intensive tasks: 1. Generation of the ensemble members. 2. The computational flow part that describes the evolution of the stream function . Every 240 hourly timesteps an analysis of the ensemble is done to obtain the optimal estimate for the past period.

6. Parallel implementation -1 1. Ensemble members are distributed evenly over the processors. 2. Data of ensemble members are independent and are local to the processors. 3. Only in the analysis phase to determine the globally optimal data have to be exchanged (using MPI). 4. The optimal global field is distributed and a new cycle is started.

7. Parallel implementation -2 The program contains 2 irreducible scalar parts: 1. Initialisation, linearly dependent on the number of ensemble members, and depends on , the number of gridpoints by . Init time = . 2. The analysis part for which holds that the analysis time . On the DAS-2 systems and (for ).

8. Parallel implementation -3 The time per ensemble member per 24h time step This amounts to 20x60x30 = 36,000s = 10h single processor time for the complete 20 day cycle considered. After the init phase a distribute operation and per analysis step a collect and a distribute operation are required. The total amount of data moved is . The bandwidth at with this occurs is 120-140 MB/s (using Myrinet on one cluster). So, the total communication time is about 0.12s per transfer. Total communication time within one run .

9. Model and experiments -1 The timing model has the following form:

10. Model and experiments -2 Remarks: 1. There is a mistery with respect to the computation phases: for p = 1 , for p > 1 consistently. 2. For p < 6, using 1 CPU/node is somewhat faster, from p = 6 on, 2 CPUs/node is marginally faster due to decreasing competition for memory and faster intra-node communication.

11. Model and experiments: Simulation results Shown is a simulation of 180 dayly periods, note the blue eddies that form counterclocwise in the Atlantic.

12. Perspectives for distributed implementation -1 The timing model has the following form: In the single-cluster implementation is quite small (ca. 15 s) and virtually independent of . For the distributed version this might not be the case: 1. Presently Globus cannot be used yet in conjunction with Myrinet's MPI, communication must be done via IP. 2. The geographical distance between the DAS clusters introduces non-negligable latencies.

13. Perspectives for distributed implementation -2 As can be seen from the figure the communication time is still insignificant when distributing the model over two locations (UU and VU):

14. Perspectives for distributed implementation -3 is quite erratic, more determined by synchronisation than communication time:

15. Perspectives for distributed implementation -4 The results show that this application is excellently suited for distributed processes. Still, both communication and the analysis phase may be made more efficient: 1. When is known which process id.s are located where, first intra-cluster communication can be done, then the assembled messages can be exchanged. 2. The analysis could be done on the local ensemble members (remember ) and synchronised less frequently.

16. Perspectives for distributed implementation -5 Using more sites has a notable effect on the communication. Again, synchronisation effects are more important than the communication time proper : Sites Exec.time (s) Comm.time(s) 1 3310 45.1 2 3274 62.9 3 3339 208.5 4 3299 151.9 12-proc. run: UU, UU+VU, UU+VU+Leiden, UU+VU+Leiden+Delft

17. Perspectives for distributed implementation -6 This case study was a particular well suited candidate for distributed processing. Apart from improving this implementation we will proceed with three other projects that are promising: 1. Running two coupled oceanographic model within the Cactus framework. 2. Inexact sequence matching of genetic material. 3. Pattern recognition on proteomic micro arrays. Acknowledgements: Fons van Hees for the single-system parallelisation