Hurricane Storm Surge Simulation on Petascale Computers

1. Hurricane Storm Surge Simulation on Petascale Computers Clint Dawson and Ethan Kubatko, UT Austin; Joannes Westerink, Notre Dame; Anna Spagnuolo, Oakland U. Funded through: NSF OCI-0749015/0749017/0746232 OBJECTIVE In this project, we will explore the use of petascale computing for highly detailed and accurate modeling of storm surges due to hurricanes and tropical storms. Specifically, we will develop physical and mathematical models, and the numerical and computational infrastructure necessary, within a petascale environment, to advance the following goals: 3) Numerical resolution of all pertinent flow scales A highly resolved mesh is necessary to define and capture local topography and bathymetry, local roughness, critical hydraulic conveyances, structures and raised features that impede or enhance flow, and wind/wave transformation scales. Currently, the typical grid resolution is on the order of tens of kilometers in the deep ocean to 50-100 meters in parts of the Louisiana/Mississippi coastal shelf, near-shore and overland regions. With advanced optical remote sensing technology such as LiDAR (Light Detection and Ranging) and improved land-cover databases, we will have the capability to reduce the scale of resolution in our model to 1-10 meters in the most vulnerable and complex areas of the coast. • Compute storm surge forecasts of unprecedented resolution and accuracy, possibly following multiple storm tracks, within a short enough time frame to provide crucial information to emergency managers and first responders. • Study in great detail the vulnerability of coastal regions to storm surge, and study the effectiveness of various proposed hurricane protection systems and their impact on the coastal environment. Figure 5: Domain decomposition of the finite element mesh 4) Robust and accurate solution of the resulting system of PDEs. In addition to improving the mathematical models, and the level of detail and mesh resolution incorporated into the computational domain, we are investigating more robust and accurate solution methods for the resulting system of PDEs, all focused towards the development of the “next-generation” ADCIRC simulator. This algorithmic development is funded under an NSF Computational Mathematics in the Geosciences (CMG) project The primary focus of this project is the development, analysis and implementation of novel hp-adaptive discontinuous Galerkin (DG) finite element methods for shallow water circulation coupled with winds, wave and bed morphology. PRELIMINARY PARALLEL SCALING Parallelism is achieved through the use of domain decomposition (see Figure 5) and MPI. Preliminary studies indicate that the DG methodology offers significant improvements in accuracy and robustness over the current continuous Galerkin (CG) finite element strategy used in ADCIRC. Furthermore, the DG method, while involving more “work” per element than the current CG method, is highly parallelizable. Thus, the DG method is a prime example of an algorithm which may be deemed computationally inefficient on serial or small-scale parallel machines, but may be significantly more scalable to petascale architectures than current algorithms. COMPONENTS OF A HIGH-RESOLUTION STORM SURGE MODEL PIs Westerink, Dawson and Kubatko have been at the forefront in the development of the Advanced Circulation (ADCIRC) storm surge model (see www.adcirc.org). The basic components of a high fidelity storm surge model include: 1) An accurate definition of the physical system One of the basic philosophies underpinning the ADCIRC model has been the application of suitably large physical domains. These domains have boundaries placed in the deep ocean, and extend through entire basins to inland waters. Current storm surge studies focused on the Louisiana and Mississippi coasts are computed on a domain that includes the western North Atlantic Ocean, the eastern seaboard of the U.S., the Caribbean Sea and the Gulf of Mexico, see Figure 1. RESULTS FOR HURRICANES KATRINA AND RITA Recent events in the Gulf of Mexico have demonstrated the vulnerability of coastal populations and infrastructure to storm surges. In the aftermath of Hurricane Katrina, the ADCIRC model was a central piece of the Department of the Army led Interagency Performance Evaluation Task Force (IPET) study and an instrumental part of on-going studies related to the long-term design of future protection systems for Louisiana, under the auspices of the congressionally-mandated Louisiana Coastal Protection and Restoration project (LACPR) and the USACE Hurricane Protection Office, and the development of updated FEMA coastal inundation maps for the Louisiana, Mississippi and Texas coasts. Figure 1: Western North Atlantic Ocean with finite element mesh. As storms approach land, the incorporation of relevant coastal features must be included in the definition of the physical domain. Therefore, as seen in Figure 2, there is significant detail in the description of southern Louisiana and Mississippi included in the domain. During the course of this project, through improved high-resolution data collection and petascale computing resources, we will focus on incorporating an even finer level of detail into our models. Figure 6: Parallel speedup (left) and efficiency (right) of the ADCIRC-CG and ADCIRC-DG codes on up to 1024 processors • PROPOSED WORK • Our experiences to date with simulating complex storm events in the Gulf of Mexico indicate that improvements in modeling and coupling of relevant physical processes, in domain definition and mesh resolution, and in numerical algorithms, significantly enhances the accuracy of storm surge models. Petascale computing opens the window to simulating storm events at unprecedented fidelity, which gives rise to a number of modeling, algorithmic and computer science challenges which we will investigate during the course of the project. The project will be divided into the following major tasks, with supporting subtasks: • 1) Development of a highly refined storm surge model based first on ADCIRC-CG, and later tested on ADCIRC-DG; • 2) Development of an overland flow component with rainfall/runoff at the watershed scale; • Coupling of overland flow into the surge model • 4.) Testing of the full model on simulated storms. • In support of these major tasks, we will also investigate underlying algorithmic implementation and computer science issues, and the possibility of using emerging hybrid architectures as a path to petascale applications. Figure 3: ADCIRC simulation results of storm surge for hurricanes Katrina (left) and Rita (right). (a) (b) Figure 2: Zoom into New Orleans/ Lake Pontchartrain area: (a) bathymetry; (b) finite element mesh 2) Incorporation of all relevant physical processes With increasing fidelity in the description of the flow domain, improvements in the model physics are also necessary to accurately capture multiscale features of the flow. Storm surges are driven by winds and waves; furthermore, a significant amount of sediment can be transported during the storm, resulting in dramatic changes in the seabed. Improved wind modeling and more accurate wind/wave/circulation coupling is one of the important objectives of this project. Figure 4: Hydrographs from Hurricane Katrina at various locations throughout Louisiana comparing ADCIRC results with measurements (left) and a comparison of ADICRC maximum surge levels with observed high water marks (above).