KISTERS TSM Update Michael Natschke

1. KISTERS TSM UpdateMichael Natschke

2. KISTERS TSM in a nutshell... • KISTERS TSM is a time series management server • Completely written in Java • Responsible for time series operations • Storage management (storage definition, read and write data) • Configuration of agents (calculation and validation) • Execution of agents (automatic update) • API to Time Series Client applications (TCA) such as WISKI and Hydstra • Part of the entire KISTERS Server architecture/ landscape

3. CUAHSI WaterOneFlow / WaterML OGC WFS/WMS OGC SOS / O&M Data Provider Framework KISTERS TSM Time Series Client Such as Hydstra/ WISKI KiWorkFlow KISTERS TSM KiAlarm Manager KI IO Server KIDSM KISTERS TSM

4. KISTERS TSM Time Series Client Such as Hydstra/ WISKI API updated new new Agent Framework Operation Framework Station Configuration KISTERS TSM

5. Overview and definitions: Agents • What are agents good for? • An agent is used to derive other time series • An agent is used to validate time series • Where does an agent live? • Agents live inside KiTSM and are executed on the server and not at the desktop machine

6. Which agents do exist? • Creating production time series • Import, copy, merging time series • Calculation and analysis • Summary statistics, rating curve, tidal peaks (Kunz Kövec), peaks over threshold, baseflow, cubic spline, standard deviations, long term values • Validation and Corrections • Threshold exceedance, gap detection, rate of change, delta detection, KiScript Validation, KiScript Correction • Helper and self defined agents • Roll out profiles, KiScripting

7. Agents and time series types • How can agents been defined? • Agents are defined for time series (persistent configuration) • Agents can be defined globally (runtime agents)

8. Reoccurrent configuration objects of agents • Assigning source time series • Source time series • Source column • Minimum valid quality • Stepper (for merger and KiScript) • Pre-processing • Transformation (such as force interpolation type etc) • Post-processing • Aggregation options • Validation agents • Action on implausible data • Action on plausible data • Alarm configuration

9. KISTER TSM: Agent Framework Time Series Client Such as Hydstra/ WISKI XML UI Request for dialog Retrieve results Configure Execute API updated Agent Frame work KISTERS TSM

10. KISTERS TSM Time Series Client Such as Hydstra/ WISKI API updated new new Agent Framework Operation Framework Station Configuration KISTERS TSM

11. Station configuration • Talking WISKI 6 • All time parameter and time series with their calculation and validation routines had to be created for each station • Mechanism were in place to automated this procedure • Corrections were difficult, maintenance effort were high • Learning from Hydstra and TimeStudio • It was a requirement on TSM to deliver an open architecture, providing the highest level of flexibility • KISTERS team introduced virtual time series and global calculations into TSM

12. Station configuration • Station configuration nowadays • Combination of all: • persistent time series, • virtual time series and • runtime calculation • Persistent and virtual time series can be distributed through the client (XML) to multiple stations automatically

13. Station configuration

14. KISTERS TSM Time Series Client Such as Hydstra/ WISKI API updated new new Agent Frame work Operation Frame Work Station Configuration KISTERS TSM

15. Operation Framework • What is meant by an Operation ? • Operations are algorithms that consumes TSM and/or TCA data to produce a result • The results are not limited to produce time series data • They can produce XY data structures such as returned by • duration curve analysis, • double mass analysis • regressions • etc • Operations can also include data exports, station configurations, batch graphing etc • Operarations are execute server side • Operation do follow a similar pattern to configure and execute

16. Operation Framework Data Time Range Operation Result

17. Operation Framework Grouped data Data Data Mask Ad hoc queried Time Range Operation Result

18. Operation Framework Data Absolute time range Time Range Restriction time range(s) Relative time range Operation Result

19. Operation Framework Bulks (Create time series…) Graphs (Layouts) Reports (KiScripting Reports) Data Statistic (Duration Curve, Double Mass…) Time Range KISTERS Operations Operation Third Party Operation Result

20. Operation Framework Data Time Range Operation Output configuration (optional) Result

21. Operation Framework Data Time Range Operation Result

22. Operation: Double Mass Curve Analysis Data Time Range Operation Result

23. Operation: batch graphing Data Time Range Operation Result

24. Operation: Integration into KISTERS Software

25. KISTERS TSM: Operation Framework Time Series Client Such as Hydstra/ WISKI XML UI Request for dialog Retrieve results Configure Execute API new Operation Framework KISTERS TSM

26. Summary • TSM learnt from WISKI, Hydstra and TimeStudio • Provides a powerful and open calculation and validation framework (agents) • Operation framework provides plugin functionality for almost every thing • Server side implementation of XML UI minimises compilation and installation efforts and gives KISTERS the chance to deliver new features with short response times • Station configuration has never been that easy

27. Consuming WaterML in KISTERS Products