A Complex Event Recognition Architecture

A Complex Event Recognition Architecture Will Fitzgerald Kalamazoo College R. James Firby I/NET, Inc.

A Complex Event Recognition Architecture Protecting us from the Metal Horde! Will Fitzgerald R. James Firby

What is … • A Complex Event? • Complex events are hierarchical, discrete, time-stamped structures inferred from multi-channel, asynchronous signals.

What is … • A Complex Event Recognition Architecture? • A description or implementation of typical patterns and recognition algorithms for complex events.

A real example… • Water Recovery System at NASA's Johnson Space Center • Four complex subsystems, • About 200 sensors and actuators, • Each subsystem asynchronously signals data.

Detecting “Safe Mode” • When a problem is detected internally, the Water Recovery System attempts to go into “safe mode,” which occurs when the four subsystems are safed. • “Safing” of the four subsystems happen asynchronously. • “Safing” detection for each subsystem differs from one another. • On recognizing that the WRS has gone into safe mode, signal an event that all subsystems have been safed.

Another example… To get directions to a location on the on-board map, the user says: • “Go here” and • Taps the display location within 200 ms. (CNN photo)

Parsing the world • Dynamic Predictive Memory Architecture (DPMA) • KR and Semantic parsing • Task execution and dialogue management • “complex, dynamic environments” • Do similar techniques apply to … • multi-channel, asynchronous sensors? • multi-modal interface input?

A Complex Event Recognition Architecture • What assumptions are reasonable to make about the form of input data? • What useful general patterns are there in the data? • What recognition algorithms do we need?

NLP Assumptions • Input to Natural Language Processing systems are typically assumed to be: • Discrete events of one type (“words”) • Single channel • Totally ordered by position; duration irrelevant

More generally… • Events of various types • Over multiple channels and asynchronous • Duration of event often important • “Hierarchical” model still useful

Assumptions about Events • Discrete: Individually distinict, non-continuous data (could be discretized). • Time-stamped: Event carries the start and end times (defining the event duration, which could be instanteneous). • Typed: Events form distinct types (e.g., words vs. taps). • Structured: Event may internal, hierarchical structure (“complex”).

Standard Event Patterns • Are there patterns of events which are particularly useful to identify? • Are there recognition algorithms to identify those patterns? • Yes. • ONE and BINDING • IN-ORDER, ALL, ONE-OF • Allen patterns • WITHIN and WITHOUT

ONE: The simple pattern of looking for a single event (of a particular type). BINDING:ONE pattern plus collecting and constraining state. Essentially “event-driven” programming; the “stimulus” in S-R. ON-CLICK A ONE pattern if just looking for the ‘click’ A BINDING pattern if x,ycoordinates are significant. ONE and BINDING patterns

IN-ORDER patterns • Events will occur “in order” • That is, saying two events, A and B, occur in order, the start time of B is  the end time of A. • (IN-ORDER A B C D) • First an event of type A, then B, etc.

IN-ORDER as NLP • Combined with BINDING and signaling of subpatterns this is essentially a classic natural language processing pattern. S  NP VP NP  DET N VP  V NP The boy saw the girl. [S [NP [DET the][N boy]] [VP [V saw] [NP [DET the] [N girl]]]]

ALL Patterns • Events will all occur, but in any order • With this, we leave (our) standard NLP approaches. • For example, user will choose from all of the sets of options. • For example, all subsystems will be “safed”, but in any order.

ALL patterns and contradiction • The problem: user or system “undoing” an event that has already been seen (interpreting events as state changes). • Example: Class will start when all the students, Alice, Bob, Charles, Dominique, have arrived.

Consider this sequence for (ALL A B C D): • Charles arrives. • Alice and Bob arrive together. • Alice starts to sing. • Charles leaves. • Dominique arrives. • Charles arrives. Order is not relevant; Alice’s singing is not relevant; but Charles’s leaving undoes his earlier arrival.

ONE-OF Pattern • Look for any of a set of event forms • Example: Office hours begin as soon as one of the professors A,B,C or D arrives. • (ONE-OF A B C D)

Time-based patterns • Allen relationships • WITHIN patterns • WITHOUT patterns

James Allen described the relationships between two intervals. Allen patterns look for temporal relationships between 2 events or an event and an interval. contains finishes starts before meets overlaps equal overlapped by after met by started by finished by during Allen Patterns A contains B A … overlaps B

WITHIN patterns reflect that the duration of an event is no longer than a certain amount of time. E.g., an ALL pattern wrapped in a WITHIN pattern. WITHOUT patterns reflect that an interval of time will pass without the occurrence of an event. E.g., Sherlock Holmes’s “significance of the barking dog.” WITHIN and WITHOUT

Pattern Combination • “Go here” and a tap within 200 ms. (within (all (in-order go here) (tap ?x ?y)) 200 ms) (CNN photo)

Safe mode recognizer (define-recognizer (safing-complete) (pattern '(all (safing (system pbbwp) (status on)) (safing (system ro) (status on)) (safing (system aes) (status on)) (safing (system pps) (status on)))) (on-complete (st end) (signal-event '(all-safed) st end))) Some details elided…

Parsing Algorithms • The parsing algorithms and recognizer semantics are more fully described in the paper.

Implementation Details

Conclusions • Standard patterns of events. • Standard recognizers for these patterns. • Good for monitoring complex (internal) system state. • Useful for recognizing patterns of complex events over multiple modes, over time.

Acknowledgments • Work done under NASA SBIR contract NAS9-00122. • We would like to especially acknowledge collaborators at NASA, including Debra Schreckenghost, Pete Bonasso, Carrol Thronesbery and others. • Pulp Images from “Pulp of the Day”:groups.yahoo.com/group/pulpoftheday

Questions?

A Complex Event Recognition Architecture