1 / 40

Alphanumeric Variables – The Long Trek

Alphanumeric Variables – The Long Trek. KHUG 2009. Peter Heweston. Alpha numeric variables. In HYDSYS V1, sites and variables were numeric. In the late 1980s we made sites alphanumeric …if only we had done the same for variables! Why?

venus
Download Presentation

Alphanumeric Variables – The Long Trek

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Alphanumeric Variables – The Long Trek KHUG 2009 Peter Heweston

  2. Alpha numeric variables • In HYDSYS V1, sites and variables were numeric. • In the late 1980s we made sites alphanumeric • …if only we had done the same for variables! • Why? • Because alphas are better, but changing over now is hard!

  3. Alpha numeric is better??? • One of the drivers of this development is the alignment of Hydstra/TS and Kisters TSM server concepts • A necessary step on the road to integration of all Kisters products • Another is that we’re simply running out of numbers • Current limit = 9999 • Proposed new limit ~ 342,000,000,000,000,000,000,000 • WLEV.LOG is much better than 100.00 • Training new users to remember numbers can be a battle • I want calculated flow • I gotta enter “100.00” and “140” into them thar edit boxes? • WTF?

  4. … but change is hard! • Every DB table that has a variable column • Every TS file has the variables coded inside it (won’t be able to exchange new TS files with clients running older versions) • Oodles of INI files… • A plethora of batch files, Perl scripts… • Some loggers are designed with numeric variables in mind… • Hydstra always allows for variable mapping • Some clients have upstream and downstream systemsthat share the same variable IDs… • Shouldn't rely on same variables • HYTRAN provides for mapping

  5. Sorting • The commonly-used vars are <999, and appear “at the top” since they are sorted numerically • Alphanumeric variables will be sorted alphabetically • (and leading spaces will not be allowed) • Will it be a problem if your commonly-used variables are scattered through a large list of little-used or unused ones? • Even if you use “numbers”, they’ll still sort differently: 1 10 2 20 …

  6. Variable Domains • Introduced a new concept of variable Domain • Domains are major groupings of variables • We have proposed some basic domains • DAM - Dams and Storages • GW - Groundwater • INS - Instrumentation and Equipment • MAN - Management and KPI • MET - Meteorology • OTH - Other • SEW - Sewers • SW - Surface Water • WQ - Water Quality

  7. Variable Domains • At present Domains are System codes and controlled by Kisters • Do we have the right list? • Should users be able to make their own? • Should everyone use the same ones? • See Variable manage • Domain is an attribute of a variable - should it be part of the variable name? • SW_LEVEL, SEW_LEVEL, GW_LEVEL • WQ_NO3

  8. Variable Domains • Domains will assist users to locate variables • Dropdowns in all programs will enumerate variables • By Domain • By Reporting Type (LEVEL, TOTAL etc) • By Units • By Word in Name • By Variable (ungrouped) • UPGRADE will try and guess the domain • Users will have to correct the guesses • HYCLIPIN might be helpful

  9. Subvars are not simple • Under the existing system, .00 is the “default” subvar. • If you leave it off (ie 100) it is assumed to be 100.00 • But “100” sometimes means “variable without subvar” or “variable with whatever subvar” (eg VarTo) • If we switch to alphas – what then? • Default subvar (.AAA? hyconfig?) • Omitted from reports • See it in HYDMWB, tables etc • Subvars are not consistent • 100.01 - meaning depends on location • Consult VARSUB to find out

  10. We’ve done some of the work already (in V9) • Internally, we’ve gotten rid of all code that assumes that a variable is a number. • We’ve also identified areas which may cause problems – internally to the system • The CHAIN variable conversion method was one benefit to come from this preparatory work

  11. This decision may affect users differently • New users • Without question, alphas are better – easier to learn • No transition pain because they will “start off” with them • The default dataset we deliver will be all alphanumeric • Small to medium existing users • Have already learnt the numbers, may not see them as a problem • May experience transition pain as they migrate their archive and learn the new Ids • Large corporate users • Will see the problem, as they train new staff more regularly • Transition pain more manageable as they have procedures for upgrading archive / synchronisation with regions etc.

  12. The two-phased approach: Phase 1 • Hydstra/TS V10The Big Preparation • Alphanumeric fields added to VARIABLE, VARSUB and WQVAR • Over the lifetime of V10 (“at your leisure”), you fill these with new values to replace the numbers • Powerful reporting tools to help you get it right • You only change the variables you want to change • Opportunity for cooperation with other agencies…

  13. The two-phased approach: Phase 1 • Opportunity for cooperation with other agencies… • Federal Variable Naming Act of 2009?

  14. The two-phased approach: Phase 2 • Hydstra/TS V11The Big Commitment • UPGRADE applies all the changes you have indicated • TS files will be incompatible with V10(&V9) • All tables will have alphanumeric fields for variables • Likewise for edits in parameter screens,dialogs etc. • Some scripts & INI files may be modified automatically, but many will require manual attention

  15. Phase 1 - Preparation details: VARIABLE • New column in VARIABLE • ALPHAVAR (C15) • Over the lifetime of V10, the system administrator (at leisure) specifies a new alphanumeric ID for each variable to be changed • The VARIABLE. ALPHAVAR will eventually be the “new name” for that variable • Blank = “no change” (new variable will be a string equivalent of the numeric variable)

  16. Phase 1 - Preparation details: VARTRACE • New table: VARTRACE • Will replace VARSUB in V11 • NB: does not have site and variable in the key • Initial contents: • “PRIMARY” • A collection of mnemonics: “LOG”, “DAILY”, “UPSTR”, “10M” etc. • “00” through “99”, with description "Unassigned" • NB: COMPRESSTS and MAXGAP (from VARSUB) do not exist in this table, as there is no provision for site- or variable- control • They will go to another new table (TSCONTROL) to be introduced in Phase #2 (V11)

  17. Phase 1 - Preparation details: VARSUB • New column in VARSUB • VARTRACE (C8) • Specify the new VarTrace to use instead of a subvar • Blank = “no change” (new vartrace will be string equivalent of subvar) • Value must exist in the VARTRACE table

  18. Phase 1 - Preparation details: WQVAR • Currently (V9), WQVAR is problematic: • Overrides NAME and SHORTNAME from VARIABLE • Can have different overrides for same variable (different subvars) • This must change. In V10 we will introduce some rules, which you must comply with before upgrading to V11. • Columns which must not vary within variable: • NAME (will be removed in V11) • SHORTNAME (will be removed in V11) • ELECCHARGE • MOLAREQUIV • Columns which may vary within variable: • ACCURACY, COMMENT, MAXIMUM, MINIMUM, CASREF

  19. Phase 1 - Preparation details: WQVAR (contd.) • In V11, WQVAR will have a lesser role than at present • NAME and SHORTNAME will be withdrawn. The names from VARIABLE will apply • Rather than overriding values in VARIABLE, it will extend VARIABLE, providing additional WQ-related information • This is similar to the way the STATION table extends the SITE table, providing additional gauging-station-related information

  20. Phase 1 - Preparation details: HYVARCHK • All this looks like a lot of hard work • HYVARCHK will assist you in this job • It will provide an extensive report on all issues relating to managing the transition to alphanumeric variables • It will also be used to give a “stamp of approval” that everything is in order, so the V11 upgrade (the big commitment) cannot proceed if anything is amiss.

  21. Phase 1 - Preparation details: HYVARCHK (contd.) • HYVARHK will report on: • Variable holdings: which variables are actually used in the system • TS data (archive & work) • Ratings, Shifts • VARCON (including “intermediate” variables) • Although many VARCON entries may not actively be being used • TTABHED (time tables) • Gaugings, Gaugings measurements • Logger processing (xxxCHAN) • WQ data (archive & public work areas) • GW measurements • (and other tables with variable references)

  22. Phase 1 - Preparation details: HYVARCHK (contd.) • HYVARCHK will also report on: • VARIABLE collision detection: • No two VARIABLE.ALPHAVAR values can be the same • VARTRACE ambiguity / merging report: • Report entries in VARSUB that map to the same VARTRACE entry • This is for informative purposes only: we would encourage it in many places • WQVAR “pre-mapping” conformance report: • No WQVAR entries with the same variable may have different names, accuracies, electric charges or molar equivalents • WQVAR “pre-mapping” report: • For EACH WQVAR variable to be "pre-mapped", list its WQ archive holdings • VARIABLE mapping report: • For EACH variable that is to be renamed, list complete holdings

  23. Phase 2 – Commitment details • In Hydstra/TS V11 – the Upgrade: • All variables through out the system (table columns & TS file codes) are made alpha • They are replaced with the corresponding value from the VARIABLE.ALPHAVAR column. • The ALPHAVAR column is dropped from the VARIABLE table. • All “subvars” through out the system (table columns & TS file codes) are replaced with the corresponding value from the VARSUB.VARTRACE column. • VARSUB table withdrawn, we now use VARTRACE and TSCONTROL instead. • We stop talking about “subvar”, and start calling it “trace”

  24. Phase 2 – Commitment details (Cont.) • New table: TSCONTROL • Some settings of the old VARSUB table must be able to vary by site and/or variable. • COMPRESS • MAXGAP • VARTRACE has no site or variable in the key, so we can’t have them there. • TSCONTROL will be created and populated during the V11 upgrade, from values in SUBVAR

  25. Sharing data between versions • V11 will not be able to read directly TS files from earlier versions • The previous two changes to TS file format were/are 2003(V9.0) and 2008(V10.0) • V9/10 TS files store the variable as a number • V9/10 index files have a floating point number in the file – the very formatis incompatible • So how can Hydstra discover which alphanumeric variable and vartrace is stored in the earlier file? • In some cases, the mappingsyou have meticulously craftedduring V10 are all that is needed • But won’t they be lost once UPGRADE has completed?

  26. Sharing data between versions (contd.) • ALPHAVAR.INI to the rescue * • The mappings from numeric variables and subvars will be preserved in an INI file called ALPHAVAR.INI. • This INI file will have named sections, one of which will be [Default] • When any Hydstra application reads a V9 or earlier TS file, it will attempt to use the [Default] mapping in ALPHAVAR.INI. • To assist you reading earlier files fromother organisations, you can createother mapping sections in ALPHAVAR.INI • This system will also allow HYIMPEXP to upgrade data from earlier versions • There will be a new command in HYFILERthat will let you upgrade a TS file usinga nominated mapping section * … or should that be .XML? ... or TRANSLAT and HYTRAN?

  27. Variable Naming Standards • Do we care about national uniformity in variable naming? • At least the major variables could be named the same everywhere • Guidelines will be published on how to name variables • Possible Australian working group on variables? • Kisters? • AHA? • BOM? • ... • Same in US?

  28. Variable Names • Variable names need to include an indication of units • Should they include DOMAIN? • Should they include Unitcode? • 100 -> SW_LEV_M • 100 -> LEVEL_M • 140 -> SW_FLOW_CUMC • 140 -> FLOW_CUMC • Use Unitcode on non-default? • 100 -> LEVEL • 140 -> FLOW • 141 -> FLOW_MLD

  29. BOM Variables • BOM have defined variables for data interchange • They are very wordy, and include units • WaterCourseDischarge_m3s • ReflectedGlobalSolarIrradianceStandardDeviation_wm2 • There is some notion of directionality • UrbanInflow_ML • UrbanOutflow_ML • StorageTransfer_MLd • Originally BOM wanted inflows separated from outflows everywhere • One man's outflow is another man's inflow • Is direction a property of the variable or of its use?

  30. Variable Names • Do we distinguish between the location of a variable and what it measures? • Stream flow • Sewer flow • Spillway flow • Channel flow • In Hydstra sometimes YES since the variable conversion system may require different pathways • Stream flow uses rating • Sewer flow uses velocity/depth • etc

  31. Variable Names • Do we care HOW something is measured? • Flow from rating • Direct measured ultrasonic • Direct measured magflow • Flow from slope/roughness? • We suggest that subvars are better suited to distinguishing HOW as a general rule

  32. Water Quality Variables • Should we have more DOMAINS? • WQI - Inorganic • WQO - Organic • WQB - Biological • WQP - Physical • How would you name the following? • Nitrate + nitrite as N - total (WQI_NATE_NITE_N_MGL) TOO LONG • Nitrate + nitrite as N - non filterable • Endosulfan(alpha isomer)959-98-8 • Endosulfan(beta isomer)33213-65-9 • Pediastrum duplex • Pediastrum duplex var gracillimum

  33. Water Quality Vars • Nitrate as N • Nitrite as N • Nitrate + nitrite as N - total • Free Ammonia as N • Ammonia as N - total • Kjeldahl Nitrogen • Total Nitrogen • Albuminoid Nitrogen • Organic Nitrogen • Inorganic Nitrogen • Detrital Nitrogen • Nitrate + nitrite as N - soluble • Nitrate + nitrite as N - non filterable

  34. Water Quality Vars • Difficult to encode Nitzschia acicularis var closteroide in 15 chars • WQ_NIZACCLO_CNT • NITZACIC_CLO_CNT • 8704 • use some catalogue

  35. Water Quality Variables • Water Quality variables • Try to keep mnemonics? • There are a large number, some differ very subtly • Sulphate as SO4Sulphate as SSulphite as SO3Sulphite as SSulphide as S • Elements: use common name or symbol? • LEAD or PB • Use CAS registry number instead? • Sulphate = 14808-79-8 • ACX number? • (Available Chemicals eXchange) • System from CambridgeSoft • Sulphate = X1013850-6

  36. Issues on Naming Conventions • What would be a good first set of variable names? • LEVEL, STAGE, HEIGHT? • Q, FLOW, DISCH? • RAIN • Should units be included in the identifier? • No for commonly-used variables where it can be assumed? • Yes when there are more than one in common use? • Variations on a theme: consider multiple “FLOW” entries: • differ by units: FLOW_CUM, FLOW_MLD • differ by time-dimension: FLOW_RATE_CUM, FLOW_VOL_ML • differ by domain: SEW_FLOW_RATE_CUM, SW_FLOW_VOL_ML • Gets silly quickly

  37. Biologicals • Biologicals are a challenge • A few indexing projects have been proposed • Based on the idea of taking the first two characters of each token of the taxonomic tree • A lot of work would be required, even if a definitive list could be obtained…

  38. Sharing data between organisations (contd.) • Kisters Pty Ltd is currently considering our ability to assist in defining a common set of variables for the Australian market • Facilitator? • Leader? • Proposing a set of initial recommendations? • Do we give a s**t?

  39. Where To Now? • It's up to you!

  40. Alphanumeric Variables – The Long Trek KHUG 2009 Peter Heweston

More Related