D. Scott McRae Aerospace Engineering North Carolina State University

1. Prediction of Optical Scale Turbulence with a Typical NWP Code: Lessons Learned (?) Projected to Petascale Computing D. Scott McRae Aerospace Engineering North Carolina State University NCAR Theme of the Year Workshop May 6, 2008

2. Acknowledgments • Prior funding: • The US Army Research Laboratory, Battlefield Environments Division, WSMR and the HELJTO monitored by Dr. David Tofsted, ARLWSMR • Current funding: • The US Air Force Research Laboratory, Space Vehicles Directorate, Hanscom AFB, MA through contract FA8718-04-C-0019; monitored by Dr. George Jumper, AFRL/VSBYA • NorthWest Research Associates, CORA division, monitored by Dr. Joe Werne • Contributors – Xudong Xiao, H. A. Hassan, and Yih-Pin Liew, NCSU; Talat Odman, GIT; and Frank Ruggiero,AFRL

3. Outline • Current goal • Prediction of Optical and Clear Air Turbulence by modifying existing NWP codes to increase prediction accuracy • Approach • Examples of current work • Projection to Petascale • Analysis of solution • Numerical Issues • Concluding Remarks

4. What is optical turbulence? • The term “Atmospheric Optical Turbulence” refers to fluctuations in the refractive index of air due to turbulence in the atmosphere. • Affects optical propagation by random refraction • Reduces the effective power of optical signals • Quantitative measure of the intensity of atmospheric optical turbulence: structure parameter of refractive index, Cn2 , • Integration of an accurate prediction of Cn2 along the beam/viewing path is the primary need for many optical systems • Accurate prediction requires well resolved dynamics and physics, whether from observation or from atmospheric models. In the latter case, physically accurate turbulence models are required for the scales unresolved by the atmospheric model

5. Observation versus Clear 1 Model Jumper, Beland, 2000

6. Approach • Prediction usually required for sub-mesoscale domains • Radiosonde soundings • Numerical weather prediction codes with parameterizations for • DNS simulation • Statistical techniques using many sources • Present approach- modify existing NWP codes to increase accuracy of prediction • LES scale Prediction Using Dynamic 3-D Adaptive Grid • Hybrid LES/RANS Turbulence Model with direct Output-described by Hassan in a later talk • Resolution of shear requires adaptation in all three coordinate directions • Accuracy of prediction can be increased by including more physics of turbulence in the model

7. Model Modifications • NCSU r-refinement Dynamic Solution Adaptive Grid Algorithm (DSAGA) • Resolve selected features/characteristics/properties dynamically • Criteria selected initially for resolution • Code determines location and resolution automatically • Adapts in all three dimensions • The NCSU k- hybrid turbulence model • Four equations based on exact equations derived from the Navier-Stokes and modeled term by term- described by Hassan in a following talk

8. Results – 2D case Mesh size: 221X126 Same setup as in Ref. 14 (by Doyle etc.) Geometry of the computational domain

9. Results – 2D case Inflow wind speed profile from the Grand Junction, CO, sounding for 1200 UTC 11 January 1972

10. Results – 2D case(MILES) Adaptive mesh at t=3h

11. Results – 2D(MILES) Potential temperature contours

12. Numerical Lidar (x= -2km)

13. Velocity vectors- 3hours

14. Detail- Velocity vectors- 3 hours

15. “Isolated “ vortices

16. Velocity vector animation

17. Cn2 contours and balloon trajectory

18. Comparison of zonal wind profile

19. Comparison of zonal wind profile

20. Comparison of meridional wind profile

21. Comparison of meridional wind profile

22. Comparison of

23. Comparison of vertical spacing

24. Weight Function Along Trajectory

25. Snapshot of the Adaptive Grid: Tennessee Valley Ozone Simulation Grid adapting to density of NOx plumes From Dabberdt W. F. et al., “Meteorological Research Needs for Improved Air Quality Forecasting” Bulletin of the American Meteorological Society, vol. 85, no.4, pp. 563-586, April 2004.

26. Superior O3 Predictions with Adaptive Grid Sumner Co., TN Graves Co., KY

27. Scaling Adaptive MM5 to Franklin

28. Adaptation plus Petascale • Adaptive meshing will move specific resolutions upwards on the previous chart • Dynamic adaptation may provide a more efficient alternative to standard nesting • Targeted solution dependent weight function determines automatically where resolution is needed rather than a predetermined nest structure • Initial budget mapped uniformly unto processors • No mesh boundary errors (however, adaptation is not without error) • Unfortunately, present NWP codes are unlikely to scale to full use of a petascale resource due to a combination of factors

29. Evaluation of MM5 Results • Filtering/dissipation in code tends to reduce or eliminate structure/frequencies needed for prediction • Solution does not converge as mesh is refined- S. Koch, NOAA • Approximately 3-1 reduction in vertical spacing improves structure but still appears over damped. Benefit due to local mesh refinement difficult to assess • LES resolution of turbulence scales not yet achieved (Terra Incognita – Wyngaard) • WRF-ARW shares basic MM5 approach with updated algorithms- has 8 identifiable dissipation sources

30. Numerics- MM5 • Horizontal integration scheme- time centered explicit (leapfrog) • Neutrally stable for all , for central space • MM5 filtering (Asselin, 1972) for any variable α: Where for all conditions • Staggered grid with averaging

31. Numerics • Vertical coordinate • Where • The turbulence model provides an alternative Eddy viscosity

32. Numerics • Vertical integration scheme- semi-implicit (Klemp and Wilhelmson, 1978) • horizontal results held constant, With divergence damping added

33. Assessment of Damping • prediction derived from local state and spatial variation- spatial filtering is then the issue • Asselin filter used to stabilize Leapfrog in MM5 Where for all conditions This can be expressed as Assuming linear advection

34. Assessment of Damping Which becomes This implies that spatial damping due to the Asselin filter remains constant relative to the mesh if and CFL remain constant as the mesh spacing is reduced This filter contributes to mathematical non-convergence

35. Assessment of Damping Divergence Damping (Skamarock and Klemp, 1992) • Essentially a modification of the normal stress term in the diagonal of the stress tensor (“Normal Stress Damping”, McRae, 1976) • Simplified analysis as in Durran – x momentum where Discretizing the first term gives

36. Assessment of Damping This is a spatial damping that remains constant relative to the mesh as the mesh spacing is reduced This damping contributes to mathematical non-convergence Furthermore, divergence damping- • Inserts a modified normal stress into the Euler equations with an unscaled coefficient of the order of an eddy viscosity • Interacts with turbulence models/parameterizations resulting in non-physical shear layers • Biases the pseudo- incompressibility formulation

37. Evaluation of MM5 Results Assessing sensitivity of convergence to individual dissipations time consuming and problematical Needed- A technique for assessing the net effect of all of the dissipations applied in the course of the integration A posteriori ( forensic ) analysis of the solution The mesh is a band pass filter and determines the number of Fourier terms available for constructing the solution

38. Total Assessment of Dampinfg Use Fourier analysis of the solution to ascertain, approximately, how much of the spectrum theoretically resolved by the mesh remains in the solution (ignoring aliasing) • Sample the solution in a direction in which resolution is important • Use discrete FFT to obtain approximate frequency versus amplitude distribution • Infer overall filtration

39. Sampling

40. Sampled Function

41. FFT Output

42. Concluding Remarks • Three coordinate dynamic grid adaptation in conjunction with physically based turbulence modeling results in demonstrated improvement in optical turbulence prediction • However, damping and filtering in present NWP codes limit the improvement • Some damping types, as used, contribute directly to mathematical non-convergence and may lead to non-physical shear layers, thereby damaging further the optical turbulence prediction

43. Achieving full advantage from use of Petascale computing resources will likely require major changes in the current NWP code equations, algorithms and their implementation Concluding Remarks

44. Concluding Remarks • We are grateful for the many helpful conversations with the people of AFRL/VSBYA, ARL/WSMR, NWRA, DRI, NCSU MEAS, NCAR and others