Analysis of time series

1. Analysis of time series Riccardo Bellazzi Dipartimento di Informatica e Sistemistica Università di Pavia Italy riccardo.bellazzi@unipv.it

3. Time series • Time series: a collection of observations made sequentially in time • Many application fields: • Economic time series • Physical time series • Marketing time series • Process control • Characteristics: • successive observations are NOT independent • The order of observation is crucial

4. Why time series analysis • Description • Explanation • Prediction • Control Understand, then act

5. Outline • Dynamic systems basics • Basic concepts • Linear and non linear dynamic systems • Structural and black box models of dynamic systems • Time series analysis • AI approaches for the analysis of time series • Knowledge-based Temporal Abstractions • Knowledge-discovery through clustering of time series

6. Outline • Dynamic systems basics • Basic concepts • Linear and non linear dynamic systems • Structural and black box models of dynamic systems • Time series analysis • AI approaches for the analysis of time series • Knowledge-discovery through clustering of time series • Knowledge-based Temporal Abstractions

7. Dynamical systems • System: a (physical) entity which can be manipulated with actions, called inputs (u) and that, as a consequence of the actions, gives a measurable reaction, called output (y) • Dynamic: the system changes over time; in general, the output does not only depend on the input, but also on the current “state” of the system (x), i.e. on the system history u y x

8. A dynamical system (example) • A simple circuit with two lamps and one switch with values 0 (u1) or 1 (u2). The output can be y={y1 (lamp1 on), y2 (lamp2 on), y3 (off)}. The system is configured to have four states, x1, x2, x3, x4 u1 x1 y1 x1 x4 x2 y2 u2 u2 x3 y3 x4 x2 x3 u1

9. Dynamical system definition • A dynamical system is a process in which a function's value changes over time according to a rule that is defined in terms of the function's current value and the current time.

10. Modeling a dynamical system • Two ingredients: • A state transition functionX(t)=f(t,t0,X0,u(.)) • An output transformationY(t)=h(t,x(t)) u1 x1 x4 u2 u2 x2 x3 u1 x1 y1 x2 y2 x3 y3 x4

11. Main classes of dynamical systems • Continuous / discrete • Linear / nonlinear • Time invariant / variant systems • Single / Multiple Input / Outputs • Deterministic / stochastic

12. Discrete and continuous systems • Discrete: the time set is the set of integer numbers (t=1,2,…,k,…). The system is typically modeled with difference equations • Continuous: the time set is the set of non -negative real numbers. The system is typically modeled with differential equations

13. Equilibrium The pair defines an equilibrium if and only if The output at the equilibrium is given by

14. ingestion u1 injection u2 k1 Gastrointestinal compartment Hematic compartment k2 elimination Compartmental models x1 =drug concentration in the gastrointestinal compartment (mg/cc) x2 = drug concentration in the hematic compartment (mg/cc) k1 = transfer coefficient for the gastrointestinal compartment (h-1) k2 = transfer coefficient for metabolic and excretory systems (h-1) States and inputs x1 , x2, u1, u2

15. Equilibrium Given constant inputs, u1 and u2,

16. Stability of equilibria An equilibrium x = a is asymptotically stable if all the solutions starting in the neighbourhood of a moves towards it.

17. Stability of trajectories Stable Unstable Asymptotically stable

18. Phase portrait The locus in the x1-x2 plane of the solution x(t) for all t > 0 is a curve that passes through the point x0. The x1-x2 plane is usually called the state plane or phase plane. For easy visualization, we represent f(x)=(f1(x),f2(x)), x= (x1,x2 ), as a vector, that is, we assign to x the directed line segment from x to x + f(x). The family of all trajectories or solution curves is called the phase portrait.

19. M=g=l=1 equilibrio instabile equilibrio as. stabile A Phase portrait of a pendulum

20. The phase portraits • Fixed or equilibrium points • Periodic orbits or limit cycles • Quasi periodic-attactors • Chaotic of strange attractors Non linear dynamic systems theory studies the property of the system in the phase plan

21. Linear systems • Linear systems: f and g are linear in x and u • Linear Time Invariant (LTI) Systems Theorem: An equilibrium point of a LTI system is stable, asymptotically stable or unstable if and only if every equilibrium point of the system is stable, asymptotically stable or unstable respectively

22. Linear systems • The dynamics is characterized by the eigenvalues of the matrix A

23. Linear systems: input/output representation • A linear system can be represented in the frequency domain u(t) y(t) g(t) G(s) U(s) Y(s)

24. Reachability Definition: A state is reachable if there exists a finite time instant and an input , defined from 0 to , such that A system such that all its states are reachable is called completely reachable

25. Observability Definition: A state is called unobservable if, for any finite , A system without unobservable states is called completely observable

26. Reachable and unobservable Reachable and observable Output transformation Unreachable and non observable Unreachable and observable Decomposition

27. Outline • Dynamic systems basics • Basic concepts • Linear and non linear dynamic systems • Structural and black box models of dynamic systems • Time series analysis • Some AI approaches for the analysis of time series • Knowledge-discovery through clustering of time series • Knowledge-based Temporal Abstractions

28. Data Models • Input/output or black box • Description of the system only by knowing measurable data • Typically based on minimal assumptions on the system • No infos on the internal structure of the system

29. Reachable and unobservable Reachable and observable Output transformation Unreachable and observable Unreachable and non observable Modeling with black-box

30. Data Models SYSTEM DATA Modeling PURPOSE INPUT-OUTPUT RELATIONSHIP PARAMETER ESTIMATE MODEL

31. Data Models • Time series • Impulse response • Transfer functions (linear models) • Convolution / deconvolution (linear models)

32. 15 10 y u SYSTEM CONCENTRAZIONE 5 0 0 20 40 60 80 100 120 140 160 TEMPO Data models (Input-output)Example Unknown parameters

33. System Models • White or grey box • Description of the internal structure of the system based on physical principles and on explicit hypotesis on causal relationships • After comparison with experimental data are aimed at understanding the principles of the system

34. System Models SYSTEM DATA A priori knowledge Modeling Purpose PARAMETER ESTIMATE STRUCTURE Assumptions MODEL

35. y1 = x1/V1 u k21 x1 x2 V1 k12 k01 SYSTEM MODELS (STRUCTURAL) COMPARTMENTAL MODELS y1 = x1/V1 u x1 V1 k01 Unknown parameters p=[k01, V1]T Unknown parameters p=[k01, k12, k21, V1]T

36. Reachable and unobservable Reachable and observable Output transformation Unreachable and observable Unreachable and non observable Structural models Guesses/ Prior kb Guesses/ Prior kb

37. Modeling time series • Time series: data are correlated; data are realizations of stochastic processes • Stochastic linear discrete input-output models • Two approaches: • Model the data as a function of time (a regression over time) • Model the data as a function of its past values: ARMA models • Often, assumption of stationarity (the mean and variance of the process generating the data do not change over time)

38. Autoregressive (AR) models • AR(h) is a regression model that regresses each point on the previous h time points. Example is AR(1) • Each value is affected by random noise with zero mean and variance s2 • Can be learned with linear estimation algorithm

39. Moving Average (MA) • A different kind of model is the Moving Average model (MA(h)) • It propagates over time the effect of the random fluctuations • The autocorrelation function may help in choosing proper models • An iterative estimation process is needed

40. ARMA • It can be used to obtain a more parsimonious model, with “difficult” autocorrelation functions

41. Exogenous inputs • The system can be driven not only by noise but also by eXogenous inputs This is the general ARMAX model

42. Non linear models • Also non-linear stochastic models have been proposed in the literature • Examples are NARX models • NARX models can be easily learned from data with Neural Nets

43. Non linear AR models • Dynamic Bayesian Nets Y1k-1 Y1k-1 Y2k-1 Y2k-1

44. From black-box to structural stochastic models Y1 Y1 X1 X1 Examples: - Kalman filters - Dynamic BNs - Hidden Markov Models X2 X2 Y2 Y2

45. Y1 Y1 X1 X1 X2 X2 Y2 Y2 Observable and partially observable models k k+1 k k+1 X1 X1 X2 X2 Y2 Y2 Fully observable Partially observable

46. Delay coordinate embedding • How to reconstruct a state-space representation from a uni-dimensional time series y • Sampled data • Idea: add n state variables using the values of y with a delay of tau

47. Example Data generated by a linear system with two state variables

48. Example Time Y1 0 0 0.0100 0.0092 0.0200 0.0171 0.0300 0.0238 0.0400 0.0295 0.0500 0.0343 0.0600 0.0383 0.0700 0.0415 0.0800 0.0441 0.0900 0.0462 Time X1 X2 0 0 0.0343 0.0100 0.0092 0.0383 0.0200 0.0171 0.0415 0.0300 0.0238 0.0441 0.0400 0.0295 0.0462 Embedding Delay=0.05 To 2 dimensions From 1 dimension

49. Plots Tau=0.265 Tau=0.0442 True

50. Challenges • Finding the embedding parameters • Estimate the number of state variable • Estimate the delay • Algorithms proposed in the literature • Autocorrelation • Pineda-Somerer • False near neighbour