Process Query Systems

1. Process Query Systems ENGS 112 Lecture 7

2. Process Query Systems (PQS) vs Data Base Systems (DBS) Data Base System Field Oriented Data Field Oriented Query (eg SQL) Business Requirement Data Sources Data Base System Query Responses Field Oriented Data Process Query System Process Oriented Query Business Requirement Data Sources Process Query System Data Base System Process Query Responses

3. When do we use “processes”? • Is there a large ground vehicle convoy moving towards our position? (Tactical C4ISR) • Is there an unusual pattern of network and system calls on a server? (Cyber-security) • Is there a pattern of unusual document accesses within the enterprise document control system? (Insider Threat Detection) • Is there a pattern of unusual transactions? (Homeland Security) • Is my software operating normally? (Autonomic computing) • Is the workflow system working normally? (Business Process Engineering) IMPORTANT – All are “adversarial” processes, not cooperative so the observations are not necessarily labeled for easy identification and association with a process!

4. How are processes defined or specified? (viz SQL) An “observation” A “state” State transition Non-branching process (A then B then C then D then …) Branching process (A then (B or C or D) then (if B then E or F) or if then …)

5. How are processes defined or specified? (viz SQL) cont’d An “observation” A “state” State transition kinematic state at time t + Dt kinematic state at time t F = ma Laws of motion Continuous kinematic processes (constrained by physical laws) p( s(t) = si | s(t-1) = sj ) = pij, p( o(t) = ok | s(t) = sj ) State (observation) at time t is s(t) (o(t), state space = { si }, observation space = { oj } Markovian models (eg, source code generators and channel models) Petri network models

6. Process descriptions • Using RDF or DAML ontologies • Describe states, possible state transitions and observables using either RDF/DAML and/or remote objects and methods • Detection of identifiable “allowed” processes leaves a residue of anomalies in the DBS so this can be used for anomaly-based detection as well as signature-based detection

7. Example – vehicle tracking Time t Time t+1 Time t+2

8. Example – vehicle tracking Time t Time t+1 Time t+2

9. Example – vehicle tracking Time t Time t+1 Time t+2

10. Example – vehicle tracking Time t Time t+1 Time t+2

11. Example – vehicle tracking Time t Time t+1 Time t+2

12. Example – vehicle tracking Time t Time t+1 Time t+2 Gates Predicted tracks Hypotheses

13. Common Logic • Track-hypothesis initiation • Hypothesis management – pruning • Handling missed detections • Handling track termination • Subscription via gates • Publication of hypothesized tracks

14. Process Query System algorithms • Continuous kinematics – Kalman filtering • Markov processes – Viterbi-like algorithms • Multiple process disambiguation – multi-target multiple hypothesis tracking (MHT) algorithms • Model estimation and optimization -Estimation-Maximization (EM) Methods • Many-to-one (multispectral sensing) and one-to-many (coincidental evidence) observation-to-process resolution methods

15. Generic PQS Operation • A user defines a “query” in terms of a process description • The process query is submitted to the PQS • The PQS searches the DBS for evidence of process instances (may be more than 1) • The evidence consists of ordinary DBS records • The PQS manages the complexity of having multiple instances of the processes • The PQS maintains a pool of the most likely hypotheses about which processes exist and what their states are

16. A PQS Implementation - TRAFEN • TRacking And Fusion ENgine • Currently in alpha version • Handles ground vehicle tracking using acoustic sensors • Handles ICMP-T3 event analysis for worm detection • Uses RDF, DAML and web services for resource description, discovery and access

17. Data mining and Expert Systems • Data mining • used to discover unknown patterns in data • can be used to define processes (front end) • Expert Systems • TRAFEN implements a specialized logic for process discovery • ES’s do not handle statistical inferences as needed in this domain

18. To do list • Graphical programming tools for processes and semantic descriptions • Control/activation of processes and/or sensor infrastructure – ie. “feedback loops” • Performance models and analysis (sampling rates, hypothesis growth, etc) • Hierarchical process capabilities (ie, identified processes are observables of higher level processes, such as convoys are aggregations of vehicles, etc) • Truly generic Kalman, HMM, MHT algorithms