Smart Initialization

1. Smart Initialization Mike Romberg

2. Lecture • Smart Init: what, why, how? • Configurability • Structure of Smart Init Modules • Override Procedures • Examples of Algorithms • Enhancements • Laboratory Smart Initialization

3. What is it? • Derives sensible weather elements from model data. Creates the IFP databases from the D2D data. • Works on models, not MOS Smart Initialization

4. Why is it needed? • Models do not provide sufficient spatial resolution. • Models do not provide set of needed sensible weather elements. Smart Initialization

5. Topography adjustments made Sampled to 5km resolution Raw model Data at 80km 2 meter Temperature Surface Temperature Derived Surface T Smart Initialization

6. Characteristics of Smart Init • Field tailorable algorithms • Can add new weather elements • Can add local models, if the netCDF files are in the correct D2D-style format • Dependencies automatic • Grids are generated as D2D data arrives Smart Initialization

7. 2 Starts appropriate module 1 ifpServer ifpserver monitorsfor data arrival Smart init module Stores generated grids 4 D2D netCDF Model Databases 3 Determines what is needed to generate, gets/stores data IFP Grid Databases Destination Database Source Database How it works Smart Initialization

8. Two sources Different names Configurability (serverConfig/localConfig) • INITMODULES • Maps algorithms modules to models INITMODULES = { “MesoEta” : [“MESOETAU”, “MESOETAS”], “Eta” : [“ETA”], “LAPS” : [“LAPS”] } INITMODULES = { “smart init module” : [‘D2D Source Databases’], } Smart Initialization

9. Configurability (serverConfig/localConfig) • INITSKIPS • Can skip certain model runs (INITSKIPS) INITSKIPS = {“RUC” : [1,2,4,5,7,8,10,11,13,14,16,17,19,20,22,23]} INITSKIPS = {“D2D ModelName” : [hour1,hour2,hour3 to skip] Smart Initialization

10. Configurability (serverConfig/localConfig) • Number of D2D Versions • Not directly associated with smart init • Does NOT override D2D purging. • Default is 2 versions. D2DDBVERSIONS = { “ETA”: 3, “NGM”: 1 } D2DDBVERSIONS = { “D2D model name” : numOfVersions, } Smart Initialization

11. Configurability (serverConfig/localConfig) • Smart init modules • Can override and provide own • Coded in Numerical Python • One file per new model or changed model Smart Initialization

12. serverConfig/localConfig INITMODULES Forecaster (etc/BASE/Init.py) EtaForecaster (/etc/BASE/Eta.py) MyEtaForecaster (/etc/SITE/MyEta.py) import EtaForecaster Smart Init structure • File locations etc/BASE, etc/SITE • Override files through etc/SITE • Uses Python inheritance tells ifpServer what to run import Forecaster Smart Initialization

13. AWIPS D2D ifpServer Destination IFP Database (ifpServer) D2D Database /data/fxa/… Smart Init Source Concept of Source/Destination Database – (needed in defining new smart init modules) • Source database – D2D netCDF file • Destination database – IFP Smart Initialization

14. Format of Smart Init File • Class Statement • Levels (function) • Calc Functions • def calcT(arguments): • Helper Functions (if needed) • Main Declaration Smart Initialization

15. Forecaster (base class) Init module EtaForecaster Eta module MRFForecaster MRF module NGMForecaster NGM module AVNForecaster AVN module Smart Init Inheritance Smart Initialization

16. Import from next class Constructor for parent ETA is D2D source, Eta is IFP destination Available levels Format of smart init file Class Statement and Levels from Init import *class EtaForecaster(Forecaster): def __init__(self): Forecaster.__init__(self, "ETA", "Eta") def levels(self): return ["MB1000", "MB950", “MB900", "MB850", "MB800", "MB750", "MB700","MB650","MB600“] Smart Initialization

17. Arguments Returns the calculated grid Format of smart init file Calc Functions def calcSnowAmt(self, T, FzLevel, QPF, topo): m1 = less(T, 9) m2 = greater_equal(T, 30) snowr = T * -0.5 + 22.5 snowr = where(m1, 20, snowr) snowr = where(m2, 0, snowr) snowamt = where(less_equal(FzLevel - 1000, topo * 3.048), snowr * QPF, 0) return snowamt Smart Initialization

18. Arguments Returns the answer Format of smart init file Example of a helper function def linear(self, xmin, xmax, ymin, ymax, we): m = (ymax – ymin) / (xmax – xmin + 0.0000001) b = ymin – m * xmin return m * we + b Smart Initialization

19. Needs to match class name Format of smart init file Main declaration def main(): EtaForecaster().run() if __name__ == "__main__": main() Smart Initialization

20. parmName_level t_FHAG2 Single grid for ParmName/level – source database parmName_c rh_c Cube of data, for the levels(), from source database topo topo High-res topo in meters, from destination database stopo stopo Model topo in meters, from source database parmName T Weather element in destination database calc Arguments. WARNING: These are different than smart tool arguments Smart Initialization

21. calc Arguments (cont.) Smart Initialization

22. calc Arguments • NOT the Fcst database as in smart tools • Can use any combination of source/destination weather elements in the calcXXX functions. • At present time, cannot easily have multiple input source databases Smart Initialization

23. Forecaster EtaForecaster MyEtaForecaster We will redefine the equation for SnowAmt Overriding Algorithms • Why override? • I don’t like existing algorithms. • I need additional sensible wx elements. def calcSnowAmt() def calcSnowAmt() Smart Initialization

24. Steps to Override • Create new smart initialization module. • If you modify the ones in etc/BASE they will go away when you upgrade! • Test out new smart initialization modules. • Put it into routine operations. Smart Initialization

25. Format of Override File • Similar to existing smart init modules. • Subtle differences, mainly due to inheritance. Smart Initialization

26. Override Eta New class name, differentinheritence structure Override function Needs to match class name Format of override file from Eta import *class MyEtaForecaster(EtaForecaster): def __init__(self):EtaForecaster.__init__(self) def calcSnowAmt(self, T, QPF): m2 = less_equal(T, 32) snowamt = where(m2, 10.0*QPF, 0) return snowamt def main():MyEtaForecaster().run() if __name__ == “—main__”: main() Smart Initialization

27. Testing your new module • Can run from command line • ifpInit –t modeltime algFile • ifpInit –t modeltime –a algFile • Model time in yyyymmdd_hhmm format. • -a switch forces all grids to be generated ifpInit –t 20011116_1200 –a MyEta Smart Initialization

28. Testing your new module • Run from the command line to see progress and problems. • Check the log files for errors • …/data/logfiles/dateDirectory/EtaInit • Check the data in the GFE. Smart Initialization

29. Okay, it works, now what? • Put it into operational use by: • Redefining INITMODULES in localConfig serverConfig.INITMODULES[“MyEta”] = [“ETA”] del serverConfig.INITMODULES[“Eta”] Don’t forget to disable the original module! Smart Initialization

30. Adding New Algorithms • Define new algorithm module • In localConfig, add new weather elements to destination database(s) before testing. • Test • Modify localConfig’s INITMODULES. Smart Initialization

31. Adding Local Models • netCDF files must have certain information in order for GFESuite to recognize them • If displayable on d2d, probably close to ok. • Need geographic information in netCDF. • Inherit from Forecaster, look at Eta.py for example. • Similar procedure: write module, test, modify INITMODULES. • May need to modify D2DDIRS. • Don’t forget - you can see these models on GFESuite without smart init! Smart Initialization

32. Useful functions in Init.py def linear(self, xmin, xmax, ymin, ymax, we): -- linear interpolation def FtoK(self, t): -- convert F to Kelvin def KtoF(self, t): -- convert Kelvin to Farhenheit def esat(self, temp): -- saturation vapor pressure def self._empty –- returns grid of all zeros def self.levels() –- returns list of levels (MB500, MB450,) def self._minus –- return grid of all –1s def self.pres –- return list of levels as numbers (500, 450,) Smart Initialization

33. Accessing Data Types • Scalar • T, simply a numerical grid • Vector • tuple, V[0] is magnitude, V[1] is direction • Weather • tuple, W[0] is grid, W[1] is key • Key is set of Weather ugly strings • Grid value indexed into key gives real value • Smart Tool lecture/lab covered specifics of types. Smart Initialization

34. Gotchas • Conditional statements • Unlike regular Python, both the true and the false are always executed in “where” statements • Standard Python if x != 0: y = z / xelse: y = 0.0 • Numerical Python where(not_equal(x, 0.0), z/x, 0.0)) where(not_equal(x, 0.0), z/(x+0.00001), 0.00) Smart Initialization

35. Future Enhancements • Ability to access multiple sources • Useful for model blending • Syntax identical for numerical smart tools and smart initialization • Fairly close now. Smart Initialization

36. Trim down levels Rh<50%, then 0 Create masks, 50-70%, 70-85%, > 85% Calc sky cover at each level Examples of Algorithms - sky def calcSky(self, gh_c, rh_c, topo): # only use the first levels (up to MB600) gh_c = gh_c[:9,:,:] rh_c = rh_c[:9,:,:] rh_c = where(less(rh_c, 50), 0, rh_c) m1 = logical_and(greater(rh_c, 50), less(rh_c, 70)) m2 = logical_and(greater(rh_c, 70), less(rh_c, 85)) m3 = greater(rh_c, 85) skylayer = where(m1, self.linear(50, 70, 0, 25, rh_c), where(m2, self.linear(70, 100, 25, 100, rh_c), where(m3, 100, rh_c))) Smart Initialization

37. Set to 0->1, from 0->100 Start at 1st level Sum up the column Examples of Algorithms - sky skylayer = skylayer / 100 sky = skylayer[0] for i in xrange(1, skylayer.shape[0]): sky = sky + skylayer[i] - sky * skylayer[i] sky = clip(sky, 0, 1) sky = sky * 100 return sky So…what is wrong with this algorithm? Smart Initialization

38. Laboratory Exercises • Override the Eta algorithm for T • Simply use 2m FHAG directly • Enhance to use dry adiabatic rate for topo corrections • Add a new model (UKMET) • Add UKMET to localConfig • Calculate QPF, Wind, and T for UKMET Smart Initialization