330 likes | 439 Views
ARSC Initiatives in Weather, Climate and Ocean Modeling. Dr. Gregory Newby, Chief Scientist Arctic Region Supercomputing Center Presentation to the HPC User Form September 9 2008. Common Themes with Other User Forum Presenters. Lots of work with WRF Work with CCSM
E N D
ARSC Initiatives in Weather, Climate and Ocean Modeling Dr. Gregory Newby, Chief Scientist Arctic Region Supercomputing Center Presentation to the HPC User Form September 9 2008
Common Themes with Other User Forum Presenters • Lots of work with WRF • Work with CCSM • Mature community-developed applications that scale reasonably well and are hungry for more CPU power • Desire for higher resolution to better approximate reality • Desire to add increasingly sophisticated processes to the phenomena under investigation • Therefore, these items will not be major themes of today’s talk
ARSC Themes of potential interest to the HPC User Forum • Performance analysis of current (multicore) and forthcoming (Cell, GPU, FPGA, T2+) processors for WRF • Quasi-operational WRF with NWS/NOAA users • Wildfire smoke prediction based on WRF/Chem • Model coupling • High resolution ocean forecasts • Arctic ice tide impacts • WRF sensitivity analysis • These will be the major themes of today’s talk
Multicore Performance Analysis on ARSC’s Supercomputers • Mostly on Midnight (dual core) • Sun x2200m2 & x4600 cluster nodes • 2312 Opteron cores, IB, 4GB/core, Lustre • 12.88 TFLOP theoretical peak • Benchmarks • Real-world applications • Forthcoming: Pingo (quad core) • Cray XT5 • 3456 cores, SeaStar, 4GB/core, Lustre • About 30 TFLOP theoretical peak
2-socket AMD64 topology CPU-2,3 CPU-0,1 One 1 GHz 16x16 HyperTransport link per supported processor with 8GB/second bandwidth
8 socket AMD64 topology CPU2 CPU4 CPU6 CPU0 CPU7 CPU1 CPU3 CPU5 * All HT links are operating at 1GHz and 8GB/s
WRF Performance on Midnight Time required to compute 1 forecast hour on 3-nest domain
Next-Generation Processor Performance • GPU, Cell, quad-core Xeon & Opteron, FPGA, CMT • Many benchmarks completed, some applications • Forthcoming: WRF test cases for AK domains (submitted for AGU 2008) • GPU (nVidia 9800 GTX), possibly Firestream • FPGA: WRF doesn’t have enough “hot spots,” porting BLAS too much work (but RapidMind might help) • Cell: QS22 cluster being configured; should work well • UltraSPARC T2+: scaled well to 8 threads per socket; WRF is too CPU-bound to benefit much from T2+ CMT
Quasi-Operational and Research Weather Modeling • Work has been evolving since 2005 • Initially, efforts were directed at quasi-operational, scheduled WRF forecasts for the Fairbanks Weather Forecast Office (WFO). The goal was to leverage lots of CPUs to provide high-resolution forecasts, and continually assess & improve • The scheduled runs continue to dominate our efforts, but rigorous analysis and study has yielded important results
2-way Nested Domains for ARSCwrf 200x200x75 cells 421x328x75 cells 151x151x75 cells
Operational WRF – AWIPS This is the end product of our twice-daily WRF runs, as seen on the NWS Advanced Weather Interactive Prediction System (FAI WFO)
Value to WFO FAI • Grid resolution is approximately 5km. The ability to populate with a reasonably accurate, high-resolution weather model saves the forecasters a lot of time. In the best case, the model output is the NDFD product.
Verification Products • Four days after a forecast, we automatically retrieve all available observations and tabulate a comparison of the WRF forecast for each location vs. the observations
Data Assimilation • Motivation – better set of initial conditions • Current initial conditions are interpolated from coarser resolution model runs – clearly much room for error • With data assimilation, we perturb our original input grid by carefully assimilating available observations • We are currently in the process of adding this capability to our operational runs, and once implemented will perform assimilated and non-assimilated runs side-by-side for comparison
Data Assimilation • Test Case – 48-hour forecast starting at 00Z on 01 May 2007 Vertical soundings from raobs and satellite – temperature, dewpoint, pressure, winds Surface obs – temperature, dewpoint, pressure, winds
Alaska Air Quality Impacted by Wildfires South Fairbanks, June 28, 2004. Air quality particulate level at approximately 900 micrograms / cubic meter. South Fairbanks, July 6, 2004. Air quality particulate level at approximately 10 micrograms/cubic meter. Photos courtesy of Dr. James Conner, FNSB http://www.dec.state.ak.us/air/am/2004_wf_sum.htm
WRF-CHEM PARTICULATE, CHEMICAL & METEOROLOGICAL FORECAST MODIS FIRE DETECTION & BURN AREA WRF-MET METEOROLOGY FORECAST FUEL MOISTURE FIRE EMISSIONS FIRE SPREAD PLUME DYNAMICS Initial architecture: WRF/Chem Smoke Dispersion System DEM, Static Fuel Static Fuel Data Emission Factors FOFEM Identical WRF domain initialization (SI/WPS) WRF/Chem Gridded Hourly Emissions POSTPROCESSING WRF-Chem netCDF Adapted from a scheme used by our partners from the US Forest Service Fire Science Lab in Missoula.
Example Product for Monday, 8th September 2008, 9:00 UTC A weak smoke concentration due to North-Easterly winds and extended fires North/East of the Yukon has been clearly confirmed in the Fairbanks area. Source: smoke.uaf.edu
Coupling Work: Regional Arctic Climate Model & More • Practical ongoing work, DoE funded. Wieslaw Maslowski, PI • ARSC is also looking at some more general approaches to coupled climate models, including adding permafrost, ice sheets, hydrology, ecosystems, human infrastructure, and coastal erosion. Major partner: International Arctic Research Center, UAF. • RACM is a regional Arctic climate system model including WRF, VIC (land surface model), NAPC (ocean and sea ice model). Current emphasis is coupling WRF, VIC and NAPC via cpl7. • Cpl7 is still in development. Our work will keep pace with cpl7 progress. CCSM4 project groups have made early-release codes available to RACM partners • Since cpl7 is designed for CCSM4, where the framework is different with WRF and VIC, some changes of cpl7 and WRF are necessary.
MPI_COMM_WORLD Partitioning of RACM Processing CPLICE CPLATM ICE ATM CPL CPLLND CPLATM LND OCN 5 COMPONENTS 10 COMMUNICATION GROUPS
ROMS and Beyond • ROMS is an ocean model that can operate at both very large and very small scales, and takes shorelines into account • At ARSC: Ongoing research for northwest Pacific, Gulf of Alaska, Beaufort Sea & Bering Sea • Also, partnering to couple ROMS with other best-of-breed models for ocean / ice prediction
ROMS Driving Ecosystem Model Physical Forcing (Wind, temperature Sunlight, mixing) Nutrients NO3, NH4… Primary Producers (Phytoplankton) Secondary Producers (Zooplankton) Fish Nutrient Phytoplankton Zooplankton models
Beaufort Sea WRF Sensitivity Analysis • Funded by MMS • Emphasis includes near-coast oil spills in Beaufort Sea (north of AK & Yukon Territory) • Many scenarios, many WRF runs • Reanalysis study • Storm case studies • Impact of winds on waves
MMS WRF Report: Higher Resolution not needed for Effective Near-Shore Wind Fields
From Field to Supercomputer: Designing Next-Generation Sea Ice Modules for Very High Resolution Ice-Ocean Models Data + models
Tidal and wind forcing of the ice-ocean boundary layer verses simple wind forcing (14km resolution) These animations demonstrate the difference between simulations being purely wind driven (right), and those that are both tidally and wind driven (left). Both animations span the same period, and have the same timestep, which is roughly 50 minutes) Arctic System Modeling
Conclusions • Climate, weather and related phenomena (sea ice, land surface, physical oceanography) models tend to be community developed and supported • Scale fairly well • Ported & maintained for new hardware • Mostly Fortran + MPI, but very complex codes • Scientists are hungry for higher resolution, longer runs, and additional runs to parameterize phenomena • The Arctic requires several adjustments to models for optimal verisimilitude, versus mid-latitudes
Huge THANKS • Don Morton (ARSC & U. Montana): Weather • Abdullah Kayi (GWU): HyperTransport • Martin Stuefer (UAF), Georg Grell (NOAA) & Saulo Freitas (CPTEC/INPE): Wildfire smoke • Kate Hedstrom (ARSC): ROMS • Georgina Gibson (ARSC): NPZ • Wieslaw Maslowski (NPS): POP, RACM • Andrew Roberts (ARSC), Jennifer Hutchings (UAF): Tidal ice • Juanxiong He (ARSC): Coupling • Jing Zhang, Jeremy Krieger (UAF), Don Morton: MMS WRF sensitivity • Many other partners & collaborators