1 / 55

Interactive Verification Program Training for AWIPS OB8.2

Interactive Verification Program Training for AWIPS OB8.2. Hank Herr Office of Hydrologic Development. RFC Verification Workshop, 08/15/2007. Overview. Introduction IVP Terminology Software Components Database Usage Output Files Design Demonstration Tips Future Possibilities

suzy
Download Presentation

Interactive Verification Program Training for AWIPS OB8.2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interactive Verification ProgramTraining for AWIPS OB8.2 Hank Herr Office of Hydrologic Development RFC Verification Workshop, 08/15/2007

  2. Overview • Introduction • IVP Terminology • Software Components • Database Usage • Output Files • Design • Demonstration • Tips • Future Possibilities • Exercises • Discussion

  3. IntroductionHistory of IVP • Original IVP written around 2000-2001 for old HP based verify software • Verification moved to archive database for OB4 • IVP Batch Program software written • Old IVP GUI still ran, but used pairs files created during batch processing; it did not access the archive database directly • New IVP GUI written for OB7.2 • Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Program • Accessed archive database directly • Limitations: • Designed for stage data • Accessed only three data tables of the archive database • Limited number of statistics • No quality control of pairs • Inflexibility in building batch files from GUI • JClass Desktop Views used to render charts (limited charting capabilities; upgrade cost too much money)

  4. IntroductionNew for OB8.2 • Designed for verification of stage, flow, temperature, and precipitation forecasts • Can do other data types as long as data is stored within accessible tables • Accesses 10 data tables of the archive database • Uses all data tables except pempsep, pemrsep, and pairedvalues • Additional statistics • Moments, CDF Plot, ROC, etc. • Quality control of pairs via GUI • View pairs in a time series plot, in addition to existing scatter plot • Mark pairs as ‘bad’ so that verification ignores them • Change forecast or observed values within a pair and save changes to database • More flexible batch file construction • Batch file can be built that will construct current chart for other combinations of locations and analysis periods • Chart Director used for rendering • Relatively quick, memory efficient chart rendering • Used previously for HydroGen, so a run-time license already exists

  5. IVP Terminology • Verification Location • A forecast point or location and a data type for which verification is to be performed • Uniquely specified by location id (lid within database) and SHEF pedtsep code. • Includes following information: • Forecast and observed categories • Observation type (raw or processed) • Active flag (yes or no) • National flag (yes or no) • Sensor preferences (ordered list of observed type source, representing preferences for which observed values to pair with a forecast)

  6. IVP Terminology • Verification Group • A set of locations and parameters that define the data to load and other parameters of verification • Locations must have the same number of categories and must be of the same type: stage (height), flow (discharge), temperature, precipitation • Includes following parameters: • Analysis period and intervals • Lead time period and intervals • Issuance time-of-day (0-24 hours) period and intervals • Restrictions on location response time, active flag, and SHEF components (forecast type source, physical element, duration, and extremum) • Whether or not to compute statistics independently for each lid or forecast type source

  7. IVP Terminology • Forecast-Observed Data Pair • A forecast value paired with an observed value • Observed value can be “raw” or “processed” depending on if the value was posted to the database by the raw or processed SHEF decoder • Each forecast can have one raw pair and one processed pair associated with it • “Best” candidate observation is chosen for each forecast based on following criteria: • SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order) • sensor preference (observation type source) rank • difference between forecast validtime and observation obstime • A pair can be “good” or “bad” • Bad pairs are not used in verification computations

  8. Software Components • Vfyruninfo Editor GUI • Manage vfyruninfo table • IVP • Data quality control and define verification products graphically • IVP Batch Program • Build pairs and verification products via a batch language • IVP Batch Builder • Edit IVP Batch Program batch files graphically • Verify Pairs Ingestor • Ingest pairs directly from text files of specific format into the vfypairs or vfyprocpairs tables • del_bad_pairs • Removes bad pairs from the vfypairs and vfyprocpairs table

  9. Database Usage • Verification Location Information • Vfyruninfo table • Defines locations for which verification can be performed. • Individual entries for each verification location and sensor preference component (observed type source) • Edited via Vfyruninfo Editor GUI • Location table • Information used in building labels, includes RFC name, full location name, etc. • Rivercrit table • Flood stages and flows (if location is of appropriate data type).

  10. Database Usage • Building Pairs • Tables of data for pairing: • Forecasts: pedfsep, pehfsep, peqfsep, pedcsep • Raw Observations: pecrsep, pedrsep, peoosep • Processed Observations: pedpsep, pehpsep, peqpsep, peoosep • Vaddadjust table • Adjust observations prior to pairing with a forecast • Only affects forecast-observed pairs • Is not dated; there can be only one entry per location • Sensok table • Allows for specifying time ranges during which sensors should not be used (i.e. observations should not be used for pairing)

  11. Database Usage • Storing Pairs • Vfypairs table • Stores pairs constructed using raw observations • Field quality_code defines if the pair is good or bad • Can be very large (millions of records) • Vfyprocpairs table • Stores pairs constructed using processed observations • Field quality_code defines if the pair is good or bad • Can be very large, but will likely be a fraction of the size of vfypairs • A forecast can have up to two pairs built for it: one stored in vfypairs and one stored in vfyprocpairs • Only one of the two pairs can be analyzed at one time

  12. Output Files • Output Statistics File (batch only) • Verification statistics output in an ASCII tabular format • Pairs File • Pairs used to compute statistics • ASCII tabular format • Output Image File (.png, .jpg) • PNG or JPG files displaying a plot of verification statistics • Image Data File (.dat) • Statistic values comprising an output image plot • ASCII tabular format

  13. Archive Database DesignFoundation User • Archive database provides the following: • Forecasts and observations • Verification location information • Forecast-observed data pairs

  14. Archive Database DesignVfyruninfo Editor User • Vfyruninfo Editor is used to specify verification locations and saves them to the vfyruninfo table of the archive database Vfyruninfo Editor Vfyruninfo rows

  15. Archive Database Design IVP Batch Program User • IVP Batch Program provides the user the ability to: • Pair forecasts with observed values according to user specified parameters Batch file Vfyruninfo Editor Batch parameters Pairing Processor forecasts, observations, location info pairs Vfyruninfo rows

  16. Archive Database DesignIVP Batch Program User • IVP Batch Program provides the user the ability to: • Calculate statistics and output results to tabular ASCII files Batch file Vfyruninfo Editor Batch parameters ASCII Pairing Processor Verification Processor pairs, statistics forecasts, observations, location info pairs pairs, location info Vfyruninfo rows

  17. Archive Database DesignIVP Batch Program User • IVP Batch Program provides the user the ability to: • Generate user customizable graphical verification products displaying statistics Graphics Batch file Vfyruninfo Editor Batch Graphics Processor parameters ASCII params statistics Pairing Processor Verification Processor pairs, statistics forecasts, observations, location info pairs pairs, location info Vfyruninfo rows

  18. Archive Database DesignIVP GUI User • IVP provides the user the ability to: • Specify, build, and view graphics Graphics Batch file GUI Vfyruninfo Editor params Batch Graphics Processor parameters ASCII params statistics Pairing Processor Verification Processor pairs, statistics forecasts, observations, location info pairs pairs, location info Vfyruninfo rows

  19. Archive Database DesignIVP GUI User • IVP provides the user the ability to: • Modify forecast-observed data pairs Graphics Batch file GUI Vfyruninfo Editor pairs params Batch Graphics Processor parameters ASCII params statistics Pairing Processor Verification Processor pairs, statistics forecasts, observations, location info pairs pairs, location info Vfyruninfo rows

  20. Archive Database DesignIVP GUI User • IVP provides the user the ability to: • Create batch files to produce graphics via IVP Batch Program Graphics batch tokens Batch file GUI Vfyruninfo Editor pairs params Batch Graphics Processor parameters ASCII params statistics Pairing Processor Verification Processor pairs, statistics forecasts, observations, location info pairs pairs, location info Vfyruninfo rows

  21. DesignIVP Window Layout 2. View and QC Pairs 1. Load Pairs 4. View Products 3. Define Products

  22. Demonstration • Feature availability: • Both OB7.2 and OB8.2 • OB8.2 only*

  23. DemonstrationVfyruninfo Editor • Start Vfyruninfo Editor • Filter ingestfilter table contents* • Select locations from ingestfilter table* • Create locations from scratch • Specify location parameters • Active* • National* • Response time • Sensor preferences • Save to archive database ref: Vfyruninfo Editor User’s Manual

  24. DemonstrationPairing Batch File • Commands: • Time window: START_TIME, END_TIME • Specify locations: LOCATION, DUR*, EXTREMUM*, FCST_TS*, PE* • Pairing window: PAIRING_WINDOW • Observation type: OBS_TYPE (replaces OBS_TABLE)* • Persistence flag: PERSISTENCE • Action: • Build the pairs: BUILD_PAIRS ref: IVP Batch Program User’s Manual for Pairing

  25. DemonstrationPairing Batch File • Example (buildpairs_template.bat): # Pair all locations. Use default settings of # ALL for all other location specifying commands location = ALL # Build pairs for a two week window prior to # current system time start_time = "* - 14 days" end_time = "*" # Parameters pairing_window = 1 obs_type = RAW build_pairs = true ref: IVP Batch Program User’s Manual for Pairing

  26. DemonstrationNational Statistics Batch File • Commands: • Time Window: START_TIME, END_TIME • Specify Locations: DUR*, EXTREMUM*, FCST_TS, PE • Action: • Compute statistics: NATLSTATS • Recommendation*: • Set the national flag for all locations that are part of the national program to ‘Y’ (use Vfyruninfo Editor) • Give the NATLSTATS action a value of NATIONAL ref: IVP Batch Program User’s Manual for Verification

  27. DemonstrationNational Statistics Batch File • Example (natlstats_template.bat): # Compute stats for September, 2003. START_TIME = “2003-09-01 00:00:00” END_TIME = “2003-09-30 23:59:59” # Default Values are ‘ALL’ # FCST_TS = ALL # PE = ALL # DUR = ALL # EXTREMUM = ALL NATLSTATS = NATIONAL ref: IVP Batch Program User’s Manual for Verification

  28. DemonstrationIVP: Verification Group Manager • Start IVP go vsys_scripts ivp • Specify parameters of a verification group • Analysis period and intervals • Lead time period and intervals • Issuance time period and intervals* • Miscellaneous parameters ref: IVP User’s Manual, Section 9

  29. DemonstrationIVP: Verification Location Manager • Specify verification locations that comprise a group • Filter available locations* • Choose locations • Specify categories • Specify observation type (from which pairs table to acquire data)* ref: IVP User’s Manual, Section 13

  30. DemonstrationIVP: IVP Data Display • Create IVP Data Display • Loading data progress window • Open the Verification Group Manager and Verification Location Manager • Save the image ref: IVP User’s Manual, Section 15

  31. DemonstrationIVP: IVP Data Display • Examine data through two views: • Scatter Plot View • Select a location* • In OB7.2, use Show button in Verification Location Manager • Display info about a location* • Line x = y • Time Series View* • Specify time series to view • Emphasize a time series ref: IVP User’s Manual, Sections 15.1, 15.2

  32. DemonstrationIVP: IVP Data Display • Draw a selected region rectangle • Zoom and the navigator panel • Zoom in on scatter view • Zoom in on time series view* • Panning through the time series by zooming • Shift the zoomed region via the navigator panel and keyboard* • Zoom out ref: IVP User’s Manual, Sections 15.6, 15.7

  33. DemonstrationIVP: IVP Data Display • View Pairs via the Verify Pairs Viewer • Select pairs • Sort table • Create a pairs file • WARNING: Do not view very large (30,000+) sets of pairs… the table rendering may be too slow ref: IVP User’s Manual, Section 16

  34. DemonstrationIVP: IVP Data Display • Mark pairs as good or bad via IVP Data Display* • Edit pairs via the Verify Pairs Editor* • Select pairs • Mark pairs as good or bad • Edit forecast or observed value manually • Focus on a pair • Edit forecast or observed value graphically • Scatter view • Time series view • Change the mouse mode* • WARNING: Do not select large sets (1000+) of pairs for editing… the table rendering speed may be too slow ref: IVP User’s Manual, Sections 15.11, 17

  35. DemonstrationIVP: Verification Plot Definition Manager • Open the Plot Definition Manager • Select statistics • IVP Statistic Chooser Manager • Select plot type • Select x-axis variable • Select comparison variable* • Edit statistics parameters* • Edit group parameters* • Spawns a window nearly identical to Verification Group Manager • Note how variable selection affects analysis, lead time, and issuance time-of-day intervals • Note how variable selection affects category choice boxes ref: IVP User’s Manual, Sections 18, 19

  36. DemonstrationIVP: The Statistics • Errors • Categorical Statistics • Quantiles and Extremes • Computed relative to whichever variable whose category used is set to “Do Not Use” • Moments* • Sample Size • The number of pairs used in the computations –or– • Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories) • Misc.* • Special Plots* ref: IVP Batch Builder User’s Manual, Appendix B

  37. DemonstrationIVP: The Statistics • Categorical Statistics use 3x3 Contingency Table • Examples: • HFAR = D/(D + E + F) • TFAR = (D + F)/(D + E + F) • CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F) • In a 2x2 contingency table • HFAR is always 0 for the lowermost category • TFAR and HFAR are equivalent for uppermost category • POD + UFR + OFR = 1 in all cases ref: IVP Batch Builder User’s Manual, Appendix B

  38. DemonstrationIVP: Verification Plot Definition Manager Statistic short name (matches output files) primary y-axis comparison variable secondary y-axis x-axis

  39. DemonstrationIVP: Verification Plot Definition Manager • Choose categories used • Disabled if x-axis or comparison variable is forecast or observed categories • Select “Do Not Use” • Only one of the two choice boxes can be “Do Not Use” • Impact of selections on how statistics are computed • Category used restricts pairs used in computations to only those within the category • Some statistics are calculated relative to the variable not used to categorize data (e.g. quantiles, cdf plot*, pdf plot*) • Special case: ROC plot* • Category used defines observation threshold ref: IVP User’s Manual, Section 18.7

  40. DemonstrationIVP: IVP Statistic Display • Create IVP Statistic Display • Data is always loaded from memory even if the overridden group parameters imply additional data is needed (such as a wider analysis interval, for example) • Allows for relatively quick plot generation • Save the image • View the data via the IVP Statistics Data Viewer • Create an ASCII format image data file ref: IVP User’s Manual, Section 21

  41. DemonstrationIVP: IVP Statistics Display • Create a batch file • Change the analysis period* • Change the included locations* • Create other groups for which to generate products* • Specify output and template file names • Replacement strings are needed if images are to be produced for multiple groups in order to avoid overwriting output files* • Specify batch file name ref: IVP User’s Manual, Section 23

  42. DemonstrationIVP: Chart Property Manager • Open up the Chart Property Manager • Click on menu item or tool bar item • Click on plot component* • Change labels • Change axis limits or chart dimensions • Change legend components • Change x-axis labels (for IVP Statistic Display) • Create a template file • Don’t modify manually unless absolutely necessary • Apply changes from a template file ref: IVP User’s Manual, Sections 24, Appendix D

  43. DemonstrationIVP: Chart Property Manager Legend Title Plot Title Y-Axis 1 Title Legend Y-Axis 2 Title Tick Mark Font X-Axis Title

  44. DemonstrationIVP Batch Builder and the IVP Batch Program • Start IVP Batch Builder • Load batch file • Insert, append, delete command lines • Edit command lines • Comment and uncomment command lines • Check for errors • Run batch file ref: IVP Batch Builder User’s Manual

  45. DemonstrationIVP Batch Builder and the IVP Batch Program • Create one batch file to generate products for many locations • @FILE Command to refer to other files • Replacement strings to avoid overwriting output files ref: IVP Batch Builder User’s Manual

  46. DemonstrationIVP Batch Builder and the IVP Batch Program • Organize output by using subdirectories of $(vsys_output)* • If producing graphics for each location, use @ONELID or @ONELIDSHEF to define directories: OUTPUT_FILE = “@ONELID/output.txt,c” • If producing graphics for time periods, use @STARTTIME or @ENDTIME: OUTPUT_FILE = “@STARTTIME{[CCYY]}/output.txt,c” ref: IVP Batch Builder User’s Manual

  47. Tips:Verification Group Manager Usage • To minimize time spent querying database, load all needed pairs at once • Only specify the overall analysis period and choose the locations within the Verification Group Manager and Verification Location Manager • Specify other parameters via the Plot Definition version of the Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager) • Important when generating many products • To minimize the chart rendering time for the IVP Data Display, keep the number of pairs small • Do one location at a time for a couple years • Do several locations at a time for one month • Important when quality controlling pairs data • See the Memory Limitations Tips later for recommended numbers of pairs to load

  48. Tips:Deciding Between PROCESSED or RAW • Use OBS_TYPE “RAW” for paired observations if… • Observed data was posted to archive database via raw SHEF decoder, AND • First letter of type source is ‘R’. • Use OBS_TYPE “PROCESSED” if… • Observed data was posted to archive database via processed SHEF decoder, OR • First letter of type source is not ‘R’ • Example: • If processed observations were posted via raw SHEF decoder (to peoosep table), then specify PROCESSED

  49. Tips:System Settings Files • Specifies parameters of graphical user interfaces • Fonts • Window Sizes • Colors • Others • Location • Default: $(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt • Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIR • Can allow for user specific system settings • Format • ESPADP/IVP batch language standard “<token> = <value>” format ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

  50. Tips: Memory Limitations • Recommended number of pairs: • Limits can be changed by using the ivp and ivpbatch script –m and –s options (run “ivp –hh” or “ivpbatch –hh” to read about these options) • Limits above are for IVP GUI; IVP Batch Program can deal with more pairs because not all pairs need to be stored in memory at one time • If IVP GUI runs out of memory, an error message will pop-up and the pairs will be discarded (in most cases) • The software may crash in other cases • Verify Pairs Viewer window is a memory hog • Add about 25% to the recommended value if not using the Viewer ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

More Related