Vladimir Krasnopolsky NOAA/NCEP/SAIC University of Maryland/ESSIC

1. CTB Seminar Series Development of Neural Network Emulations of Model Radiation for Improving the Computational Performance of the NCEP Climate Simulations and Seasonal Forecasts Vladimir Krasnopolsky NOAA/NCEP/SAIC University of Maryland/ESSIC In collaboration with: M. Fox-Rabinovitz, Y-T. Hou, S. Lord, and A. Belochitski Acknowledgements: H. Pan, S. Saha, S. Moorthi, M. Iredell

2. Outline • Background • CFS; Motivation for Development of NN Radiation • Neural Networks • NN Radiation: • Accurate and Fast NN Emulations of LWR and SWR Parameterizations • Validation • Approximation Accuracy • Parallel Runs • 17 year climate simulation • Seasonal predictions • Conclusions V. Krasnopolsky, Neural Network Emulations of Model Radiation

3. CFS Background and Motivations for Development of NN Radiation V. Krasnopolsky, Neural Network Emulations of Model Radiation

4. NCEP Climate Forecast System (CFS) (1)The set of conservation laws (mass, energy, momentum, water vapor, ozone, etc.) • Deterministic First Principles Models, 3-D Partial Differential Equations on the Sphere: •  - a 3-D prognostic/dependent variable, e.g., temperature • x - a 3-D independent variable: x, y, z & t • D - dynamics (spectral) • P - physics or parameterization of physical processes (1-D vertical r.h.s. forcing) • Continuity Equation • Thermodynamic Equation • Momentum Equations V. Krasnopolsky, Neural Network Emulations of Model Radiation

5. NCEP CFS (2)Physics – P, currently represented by 1-D (vertical) parameterizations • Major components ofP = {R, W, C, T, S, CH}: • R - radiation (long & short wave processes): AER Inc. rrtm, ncep0, and ncep1 • W – convection, and large scale precipitation processes • C - clouds • T– turbulence • S – land (noah), ocean (MOM3/4), ice – air interaction • CH – chemistry (aerosols) • Components of P are 1-D parameterization of complicated set of multi-scale theoretical and empirical physical process models simplified for computational reasons • P is the most time consuming part of climate/weather models! V. Krasnopolsky, Neural Network Emulations of Model Radiation

6. NCEP CFS T126L64 Distribution of NCEP CFS Calculation Time ~20% ~60% ~20% V. Krasnopolsky, Neural Network Emulations of Model Radiation

7. Motivations • Calculation of model radiation takes usually a very significant part (> 50%) of the total model computations. • Calculation of model radiation is always a trade-off between the accuracy and computational efficiency: • NCEP and UKMO reduce the frequency of calculations • ECMWF: • reduces horizontal resolution of radiation calculations in climate and NWP models • uses neural network long wave radiation in DAS • Canadian Meteorological Service reduces vertical resolution of radiation calculations V. Krasnopolsky, Neural Network Emulations of Model Radiation

8. Developed Accurate and Fast NN Radiation: • Allows sufficiently frequent calculations of radiation • Allows radiation calculations at each grid point of high resolution 3D grid • NN developed for both long and short wave radiations V. Krasnopolsky, Neural Network Emulations of Model Radiation

9. NN Background V. Krasnopolsky, Neural Network Emulations of Model Radiation

10. Mapping and NNs • MAPPING (continuous or almost continuous) is a relationship between two vectors: a vector of input parameters, X, and a vector of output parameters, Z, • NN is a generic approximation for any continuous or almost continuous mapping given by a set of its input/output records: SET = {Xi, Zi}i= 1, …,N V. Krasnopolsky, Neural Network Emulations of Model Radiation

11. Multilayer Perceptron: Feed Forward, Fully Connected NN - Continuous Input to Output Mapping Neuron tj Output Layer Hidden Layer Input Layer Y = FNN(X) Jacobian ! V. Krasnopolsky, Neural Network Emulations of Model Radiation

12. NN as a Universal Tool for Approximation of Continuous & Almost Continuous MappingsSome Basic Theorems: • Any function or mappingZ = F (X), continuous on a compact subset, can be approximately represented by a p (p  3) layer NN in the sense of uniform convergence (e.g., Chen & Chen, 1995; Blum and Li, 1991, Hornik, 1991; Funahashi, 1989, etc.) • The error bounds for the uniform approximation on compact sets (Attali & Pagès, 1997): ||Z -Y|| = ||F (X) - FNN (X)|| ~ O(1/k)k -number of neurons in the hidden layer V. Krasnopolsky, Neural Network Emulations of Model Radiation

13. One Training Iteration NN Training E ≤  W V. Krasnopolsky, Neural Network Emulations of Model Radiation

14. Major Advantages of NNs: • NNs are generic, veryaccurate and convenient mathematical (statistical) models which are able to emulate complicated nonlinear input/output relationships (continuous or almost continuous mappings ). • NNs are robust with respect to random noise and fault- tolerant. • NNs are analytically differentiable (training, error and sensitivity analyses): almost free Jacobian! • NNs emulations are accurate and fast but NO FREE LUNCH! • Training is complicated and time consuming nonlinear optimization task; however, training should be done only once for a particular application! • NNs are well-suited for parallel and vector processing V. Krasnopolsky, Neural Network Emulations of Model Radiation

15. Basis for Accurate and Fast NN Emulations of Model Physics • Any parameterization of model physics is a continuous or almost continuous mapping • NN is a generic tool for emulating such mappings V. Krasnopolsky, Neural Network Emulations of Model Radiation

16. NN Emulations of Model Physics ParameterizationsLearning from Data Original Parameterization GCM NN Emulation F FNN Y X Training Set …, {Xi, Yi}, … Xi Dphys NN Emulation FNN Y X V. Krasnopolsky, Neural Network Emulations of Model Radiation

17. NN for RadiationLong Wave Radiation • Long Wave Radiative Transfer: • Absorptivity & Emissivity (optical properties): V. Krasnopolsky, Neural Network Emulations of Model Radiation

18. The Magic of NN Performance Original Parameterization NN Emulation Xi Yi Xi Yi NN Emulation of Input/Output Dependency: Input/Output Dependency: Y = F(X) YNN = FNN(X) Input/Output Dependency: {Xi,Yi}I = 1,..N Numerical Scheme for Solving Equations Mathematical Representation of Physical Processes V. Krasnopolsky, Neural Network Emulations of Model Radiation

19. NCEP LW Radiationand NN Characteristics • 612 Inputs: • 10 Profiles: temperature, humidity, ozone, pressure, cloudiness, CO2, etc • Relevant surface and scalar characteristics • 69 Outputs: • Profile of heating rates (64) • 5 LW radiation fluxes • Hidden Layer:One layer with 50 to 300 neurons • Training: nonlinear optimization in the space with dimensionality of 15,000 to 100,000 • Training Data Set: Subset of about 200,000 instantaneous profiles simulated by CFS for 17 years • Training time: about 1 to several days • Training iterations: 1,500 to 8,000 • Validation on Independent Data: • Validation Data Set (independent data): about 200,000 instantaneous profiles simulated by CFS V. Krasnopolsky, Neural Network Emulations of Model Radiation

20. NCEP SW Radiationand NN Characteristics • 650 Inputs: • 10 Profiles: pressure, temperature, water vapor, ozone concentration, cloudiness, CO2, etc • Relevant surface and scalar characteristics • 73 Outputs: • Profile of heating rates (64) • 9 LW radiation fluxes • Hidden Layer:One layer with 50 to 200 neurons • Training: nonlinear optimization in the space with dimensionality of 25,000 to 130,000 • Training Data Set: Subset of about 200,000 instantaneous profiles simulated by CFS for 17 year • Training time: about 1 to several days • Training iterations: 1,500 to 8,000 • Validation on Independent Data: • Validation Data Set (independent data): about 200,000 instantaneous profiles simulated by CFS V. Krasnopolsky, Neural Network Emulations of Model Radiation

21. NN Approximation Accuracy and Performance vs. Original Parameterization(on independent data set) V. Krasnopolsky, Neural Network Emulations of Model Radiation

22. Error Vertical Variability Profiles RMSE profiles in K/day LWR – solid line; SWR – dashed line V. Krasnopolsky, Neural Network Emulations of Model Radiation

23. Individual Profiles (NCEP CFS) V. Krasnopolsky, Neural Network Emulations of Model Radiation

24. Validation of Full NN Radiation in CFS • The Control CFS run with the original LWR and SWR parameterizations is run for 17 years. • The NN Full Radiation run:CFS with LWR and SWR NN emulations is run for 17 years. • Another Control CFS Run after updates of FORTRAN compiler and libraries • Validation of the NN Full Radiation run is done against the Control run. The differences/biases are less than/within observation errors and uncertainties of reanalysis • The differences between two controls (“butterfly”/”round off” differences) have been also calculated and shown for comparison. V. Krasnopolsky, Neural Network Emulations of Model Radiation

25. Climate Simulation17 years:1990 – 2006 V. Krasnopolsky, Neural Network Emulations of Model Radiation

26. Zonal and time mean Top of Atmosphere Upward Fluxes (Winter) LWR SWR The solid line – the difference (the full radiation NN run – the control (CTL)), the dash line – the background differences (the differences between two control runs). All in W/m2. V. Krasnopolsky, Neural Network Emulations of Model Radiation

27. Zonal and time annual mean Downward and Upward Surface Long Wave Fluxes Downward Upward The solid line – the difference (the full radiation NN run – the control (CTL)), the dash line – the background differences (the differences between two control runs). All in W/m2. V. Krasnopolsky, Neural Network Emulations of Model Radiation

28. The time mean (1990-2006) SST statistics for summer & winter NN Full Radiation Run Fields Control Run Control1 – Control2 Differences NN - Control The contour intervals for the SST fields are 5º K and for the SST differences are 0.5º K. V. Krasnopolsky, Neural Network Emulations of Model Radiation

29. NN FR CTL SST CTL_O – CTL_N NN - CTL CTL1 – CTL2 V. Krasnopolsky, Neural Network Emulations of Model Radiation

30. NN FR CTL SST CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

31. The time mean (1990-2006) total precipitation rate (PRATE) statistics for summer & winter NN Full Radiation Run Fields Control Run Differences Control1 – Control2 NN - Control The contour intervals for the PRATE fields are 1 mm/day for the 0 – 6 mm/day range and 2 mm/day for the 6 mm/day and higher; for the PRATE differences the contour intervals are 1 mm/day V. Krasnopolsky, Neural Network Emulations of Model Radiation

32. CTL NN FR PRATE CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

33. CTL NN FR PRATE CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

34. The time mean (1990-2006) total) total clouds statistics for summer & winter NN Full Radiation Run Fields Control Run Differences Control1 – Control2 NN - Control The contour intervals for the total clouds fields the cloud fields are 10% and for the differences – 5%. V. Krasnopolsky, Neural Network Emulations of Model Radiation

35. CTL NN FR JJA CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

36. CTL NN FR DJF CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

37. The time mean (1990-2006) convective precipitation clouds statistics for summer & winter NN Full Radiation Run Fields Control Run Differences Control1 – Control2 NN - Control The contour intervals for the ) total clouds fields the cloud fields are 10% and for the differences – 5%. V. Krasnopolsky, Neural Network Emulations of Model Radiation

38. CTL NN FR JJA CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

39. CTL NN FR DJF NN - CTL CTL1 – CTL2 V. Krasnopolsky, Neural Network Emulations of Model Radiation

40. The time mean (1990-2006) boundary layer clouds statistics for summer & winter NN Full Radiation Run Fields Control Run Differences Control1 – Control2 NN - Control The contour intervals for the boundary clouds fields the cloud fields are 10% and for the differences – 5%. V. Krasnopolsky, Neural Network Emulations of Model Radiation

41. CTL NN FR JJA CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

42. CTL NN FR DJF CTL1 – CTL2 NN - CTL V. Krasnopolsky, Neural Network Emulations of Model Radiation

43. Some Time Series V. Krasnopolsky, Neural Network Emulations of Model Radiation

44. V. Krasnopolsky, Neural Network Emulations of Model Radiation

45. Temperature at 850 hPa, K Solid – NN run Dashed – Control Runs V. Krasnopolsky, Neural Network Emulations of Model Radiation

46. Seasonal Predictions V. Krasnopolsky, Neural Network Emulations of Model Radiation

47. SST seasonal differences for winter NN Full Radiation Run Fields Control Run Control1 – Control2 Differences NN - Control The contour intervals for the SST fields are 5º K and for the SST differences are 0.5º K. V. Krasnopolsky, Neural Network Emulations of Model Radiation

48. V. Krasnopolsky, Neural Network Emulations of Model Radiation

49. Total precipitation rate (PRATE) seasonal differences for summer NN Full Radiation Run Fields Control Run Differences Control1 – Control2 NN - Control The contour intervals for the PRATE fields are 1 mm/day for the 0 – 6 mm/day range and 2 mm/day for the 6 mm/day and higher; for the PRATE differences the contour intervals are 1 mm/day V. Krasnopolsky, Neural Network Emulations of Model Radiation

50. V. Krasnopolsky, Neural Network Emulations of Model Radiation