1 / 90

Extending FEWS with External functionalities

Extending FEWS with External functionalities. Zhengtao Cui NOAA National Weather Services Office of Hydrologic Development. Outline. Day 1 FEWS Workflow (Page 3-7) The FEWS General Adapter Concept(Pages 8 – 19) OHD FEWS adapter framework (Pages 20 – 60) Examples (Pages 61 - 90) Day 2

paiva
Download Presentation

Extending FEWS with External functionalities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Extending FEWS with External functionalities Zhengtao Cui NOAA National Weather Services Office of Hydrologic Development

  2. Outline • Day 1 • FEWS Workflow (Page 3-7) • The FEWS General Adapter Concept(Pages 8 – 19) • OHD FEWS adapter framework (Pages 20 – 60) • Examples (Pages 61 - 90) • Day 2 • Hands on Examples

  3. FEWS workflow, module and model • A workflow specifies a sequence of other workflows or modules to execute • Usually executed to generate a forecast • A module is a unit of execution • Can be a FEWS provided internal module, for example, the import module • Flow of data/files is hidden from the user • http://publicwiki.deltares.nl/display/FEWSDOC/05+Configuring+the+available+DELFT-FEWS+modules • A General Adapter is a reference to an external model • A FEWS General Adapter connects workflow with external models • Flow of data/files is visible to the user • Can have multiple execution units • Model/External model • Numerical code (source, binary, scripts, etc.) that does some kinds of calculations based a particular physical process, • Rainfall-runoff, routing, snow melt, etc. • Java model • NWSRFS models written in Java • Legacy model • NWSRFS models written in C++/C/Fortran • http://publicwiki.deltares.nl/display/FEWSDOC/Models+linked+to+Delft-Fews

  4. Workflow example

  5. Module example

  6. Model example • SAC-SMA • SNOW-17 • ……

  7. Quiz • Describe the relationship of FEWS workflow and module. Answer:A module is a part of a workflow. • Which of the following could be listed inside a workflow? • A FEWS internal module • Another workflow • A general adapter module • A binary executable of Snow-17 Answer:1,2, and 3.

  8. The FEWS General Adapter Concept • The general adapter invokes an external model allowing for FEWS workflow to execute that model • preAdapter, postAdapter, Model • The user must provide a model adapter to run an external model • The general adapter communicates with external model using the Published Interface (PI) • PI is an XML based data exchange schema • Pre/Post adapter converts PI to/from native format FEWS Model Adapter/External Model

  9. The FEWS Published Interface (PI) • FEWS has no knowledge of any external models • Initiate by the FEWS General Adapter • An external model has no knowledge about FEWS or any other external models • Read/Write PI XML data • Including diagnostic information and model states • physical model calculations • Communication between FEWS and external models is done through PI • XML data files on disk

  10. FEWS Configuration Files • Module Configuration • Provides unique information about the module (name, input/output data, properties, parameter filename, external model, etc.) • Module Instance Descriptors • Defines each module ( e.g. module name, type). Type can be FEWSTransformationor GeneralAdapter • FEWSTransformationare modules that come with FEWS • GeneralAdapterare modules developed by FEWS users • Workflows • Controls and defines the order in which to execute a series of workflows or modules • Workflow Descriptors • Defines each workflow’s characteristics • Such as a descriptive name for a workflow

  11. WorkflowDescriptor.xml … <workflowDescriptor … id=“workflow_file” </workflowDescriptor> … Workflow_file.xml … <activity> <moduleInstanceId>moduleConfigFile</moduleInstanceId> </activity> … ModuleInstanceDescriptors.xml … <moduleInstanceDescriptor id = “moduleConfigFile”> <moduleId>GeneralAdapter</moduleId> </ModuleInstanceDescriptor> … moduleConfigFile.xml FEWS Adapter … < executeActivity> <command>cmd</command> </executeActivity> …

  12. Structure of General Adapter Module Configuration File ‘general’ section defines several some common tags that will be used by other sections. ‘startUpActivity’ defines activities that are run prior to a module run and export any data. ‘exportActivities’ defines all items to be exported through the PI XML to the external module. ‘executeActivities’ defines external executables or Java classes to be run. ‘importActivities’ defines all items to be imported Following successful completion of the module run.

  13. Example: Basic Dummy AdapterDummyAdapter • Has only one “main” method. • Prints “Hello World!” to the screen. • No input/output, parameters, properties, etc. • No external model is executed. package dummy; public class DummyAdapter { public static void main(final String[] args) { System.out.println("Hello World"); } }

  14. General Adapter Module Configuration for the DummyAdapter

  15. Example: Basic Dummy AdapterDummyAdapter2 • Has only one “main” method. • Copy the output file in a property file to another file named “rootDir/output/outputs.xml”. • No external model is executed. args.file rootDirectory=Modules/dummy_module/dummy_basin2 outputFile=output_benchmark.xml

  16. DummyAdapter2.java package dummy; import java.io.BufferedInputStream; import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.IOException; import java.util.Properties; import org.apache.commons.io.FileUtils; public class DummyAdapter2 { public static void main(final String[] args) { final File propertyFile = new File(args[0]); BufferedInputStreambis = null; try { bis= new BufferedInputStream(new FileInputStream(propertyFile)); } catch(final FileNotFoundException e) { e.printStackTrace(); } final Properties prop = new Properties(); try { prop.load(bis); } catch(final IOException e) { e.printStackTrace(); } //get property identifying root directory final String rootDir = prop.getProperty("rootDirectory"); //get property identifying output file final String outputFile = prop.getProperty("outputFile"); //copy output file to output directory under root dir try { FileUtils.copyFile(new File(rootDir, outputFile), new File(rootDir + "/output/outputs.xml")); } catch(final IOException e) { e.printStackTrace(); } } 2 1

  17. General adapter configuration for DummyAdapter2

  18. The <command> option can be any executable, for example,

  19. Quiz • How does FEWS exchange information with external modules? Answer:Published Interface XML. • Which configuration file specifies the model to run? Answer:The module configuration file. • FEWS must have some knowledge about external modules, true or false? Answer:False. • What method a Java model adapter must implement? Answer:The main method.

  20. OHD FEWS Adapter Framework • An instance of the FEWS General Adapter concept. • Conforms the latest version of FEWS PI XML schema • Abstracts some common tasks for all OHD models. • Encourage code reuse • Leveraging the effort of creating an adapter for each of the models. • Can be used for both Java based model or legacy C/FORTRAN based model (a prebuilt binary executable). • Configured through the general adapter configuration • Need a run info file to pass information to the OHD model, such as input/output file locations, diagnostics files, etc. • The run info file is created by the <exportRunFileActivity> activity in the module configuration file. • Has an executable jar in the <binDir> folder.

  21. OHD NWSRFS Java and non-Java (legacy) models Java models Non-Java (Legacy) models CHANLOSS(FORTRAN) BASEFLOW(FORTRAN) SARROUTE(FORTRAN) CONSUSE(FORTRAN) APICONT(FORTRAN) gridded APICONT for MARFC's FFG application(PCRASTER) RESSNGL(FORTRAN) RESJ(C++) SSARRESV(FORTRAN) • SACSMA • UNITHG • SNOW17 • gridded SNOW17 for MARFC's FFG application • TATUN • RSNWELEV • MUSKROUT • GLACIER • LAYCOEF • LAGK

  22. Data Flow (Java model) PI params.xml OHDFewsAdapter Model specific files Loaded into OHDFewsAdapter PI input.xml PI mods.xml ModelDriver Extends xxxModelDriver FEWS states.xml PI Data are exchanged in memory. stateI.txt PI output.xml PI diag.xml Model output files are PI XML files

  23. Model Parameter File Example Follow the pi_moduleParameter.xsd schema XML tags are used to identify the parameter Use basic data types, string, double, int, table

  24. Model State files • states.xml • statesI.txt/statesO.txt

  25. Important Java Classes (1) • OHDFewsAdapter • Entry point of OHD models • Executed by the FEWS general adapter. • Invoke OHD model execution • Read in model configuration, timeseries, parameters, and states • Calls the ModelDriver.executeDriver() method • Can be executed independent of FEWS • Given all necessary information

  26. Important Java Classes (2) • The ModelDriverAbstract Class • Abstracts all NWSRFS models • All models must implement the ‘execute()’ method • Executed by the ModelDriver.executeDriver() method • Output results and the diagnostics file in PI XML format • Execute the model using common data structures (RegularTimeSeries, ModelState, ModelParameters)

  27. Important Java Classes (3) • Common xml handling classes for • Parsing FEWS pi files • Input time series • Input states meta data • Input model parameters • Writing FEWS pi files • Output time series

  28. How is it implemented? <executeActivity> <className>ohd.model.util.fews.OHDFewsAdapter</className> </executeActivity> <exportRunFileActivity> <properties> <string key=“model” value=“ohd.model.foo”/> </properties> </exportRunFileActivity> XML class OHDFewsAdapter { … void main() { … //instantiate modelDriver= Class.forName( “ohd.model.foo”); modelDriver.executeDriver(); … } … } class ModelDriver { … void executeDriver() { … //the physical model //starts execute(); … } … } Java Java

  29. Class Hierarchy IDriver Interface Abstract class Implements User class Driver OHD class Extends ModelDriver Extends Instantiate OHDFewsAdapter XXModelDriver

  30. The ModelDriver Abstract Class (1)

  31. The ModelDriver Abstract Class(2)

  32. The ModelDriver Abstract Class(3)

  33. OHD Java External Model Structure (1)

  34. OHD Java Model Structure (2)

  35. If rootDir is defined as the %TEMP_DIR%, the directory is temporary; will be removed after the adapter run. This behavior can be modified by the F12 menu.

  36. How to configure OHDFewsAdapter(1) • Header Info Every external module is a generalAdapterRun Can be missing. Set <ignoreDiagnostics> in <executeAcitvity>

  37. How to configure OHDFewsAdapter(2) • Startup activity Create working directories if they don’t exist Most are standard directories that must be cleaned out prior to model execution.

  38. How to configure OHDFewsAdapter(3) • Export activity: Export States In order to use relative dating, states must be exported. States can be empty.

  39. How to configure OHDFewsAdapter(4) • Export activity: Input Time Series Id of module that generated the needed data Relative dating requires states be exported

  40. How to configure OHDFewsAdapter(5) • Export activity: Parameters If not specified, current module instance id is used.

  41. How to configure OHDFewsAdapter(6) • Export activity: run info file • Pass general information to the Adapter • Read by the OHDFewsAdapter

  42. Example of OHDFewsAdapterRuninfo File This is the <properites> entry in the model configure file.

  43. Runinfo properties • “model” entry • The name of the Java class that implements the ModelDriver interface • “printDebugInfo” entry • Controls how detailed messages are printed to the diagnostic file. • Five levels (0 to 4). • Higher values mean more detailed information • Do not to be confused with the “Error Levels” in diag.xml • 0 – fatal level, when an exception occurs • 1 – error level, similar to the fatal level • 2 - warning level, warning messages are printed • 3 – the regular level • 4 – debug level, the most detailed information about the execution of the model. • All other entries are model specific.

  44. How to configure OHDFewsAdapter(7) • Execute activity The runinfo file is passed as an argument to the adapter.

  45. How to configure OHDFewsAdapter(8) • Import Activity: Import model output time series The module instance ID of this module.

  46. Example: Basin Dummy ModelDummy Model • Use the OHDFewsAdapter to drive the dummy model. • The dummy model is written in Java • Extends the OHD ModelDriver abstract class • Input a time series, add a value, and then subtract a value from the time series • Configure the OHDFewsAdapter in FEWS • Set input time series • The value to be added • The value to be subtracted

  47. package dummy.models; import java.util.List; import ohd.hseb.measurement.Measurement; import ohd.hseb.measurement.MeasuringUnit; import ohd.hseb.measurement.RegularTimeSeries; import ohd.hseb.ohdmodels.ModelDriver; import ohd.hseb.util.fews.FewsRegularTimeSeries; public class DummyModelDriverextends ModelDriver { public DummyModelDriver() { super._state = new ModelState(); super._parameters = new ModelParameters(); } @Override protected void execute() throws Exception { // get list of input time Series final List<RegularTimeSeries> inputTsList= getTsList(); _drivingTs= inputTsList.get(0); super.runModelDriverValidation(); final long startTime = _drivingTs.getStartTime(); final long endTime = _drivingTs.getEndTime(); final int interval = _drivingTs.getIntervalInHours(); final MeasuringUnitmeasuringUnit = _drivingTs.getMeasuringUnit(); // get measurement list; just a copy of input values final List<Measurement> measurementList = _drivingTs.getMeasurementList(); // create an output time series using properties of input time series final FewsRegularTimeSeriestempTimeseries = new FewsRegularTimeSeries(startTime, endTime, nterval, measuringUnit); // copy values in input time series to output time series final FewsRegularTimeSeriesoutputTimeSeries = new FewsRegularTimeSeries(tempTimeseries, _drivingTs); // get run info properties final Properties runInfoProperties = getDriverProperties(); // get some properties in run_info.xml file final intvalueToAdd = Integer.parseInt(runInfoProperties.getProperty("valueToAdd")); final intvalueToSubtract = Integer.parseInt(runInfoProperties.getProperty("valueToSubtract")); for(int index = 0; index < measurementList.size(); index++) { final Measurement measurement = easurementList.get(index); final double newValue = measurement.getValue() + valueToAdd - valueToSubtract; measurementList.set(index, new Measurement(newValue, measurement.getUnit())); } outputTimeSeries.setMeasurementList(measurementList); super.addTStoResultMap(outputTimeSeries); // each model driver must set the end state time in the states.xml file // this is important for validating that the state time is updated for the end of the run super._state.setDateTime(endTime); } } DummModelDriver.Java

  48. Dummy Model Configuration … <exportRunFileActivity> <exportFile>%ROOT_DIR%/run_info.xml</exportFile> <properties> <string key="model" value="dummy.models.DummyModelDriver"/> <string key=“outputLocationId" value=“SELWE"/> <int key=“valueToAdd” value=8/> <int key=“valueToSubstract” value=8/> <int key="printDebugInfo" value="1"/> </properties> </exportRunFileActivity> … <executeActivities> <executeActivity> <command> <className>ohd.hseb.ohdfewsadapter.OHDFewsAdapter</className> <binDir>$OHDBINDIR$</binDir> </command> <arguments> <argument>%ROOT_DIR%/run_info.xml</argument> </arguments> <timeOut>300000</timeOut> </executeActivity> </executeActivities> Set name of the Java class that implements the model. Explicitly set the location ID for model output time series

  49. OHD Legacy Models • Uses the OHDFewsAdapter • Extends the ModelDriver • A framework has been developed to alleviate the effort • Uses the LegacyModelAdapter to execute the legacy model • Aseparate System Process (called from Java) • LegacyModelAdapter has many helper methods such as writeStandAloneFiles().

More Related