Situation Awareness Telcordia’s E2A Architecture and Three Case Studies

Situation AwarenessTelcordia’s E2A Architecture and Three Case Studies Dimitrios Georgakopoulosdimitris@research.telcordia.com EPS, SF, November, 2006

Awareness is information packages (complex event objects, their pedigree, and related evidence) that are highly relevant to the situated needs of a user/event consumer Contextual relevancy Events must be cast in terms of concepts (e.g., space, time, objects) familiar to the user Situational relevancy Delivered events must help each user perform the specific activities he/she is working on or is responsible for Temporal relevancy Events must be delivered in timely fashion to permit effective response Awareness

Events to Awareness Concept of Operations Subscribe to awareness Decompose subscriptions to event sources and events they can detect Perform more specialized analysis & extraction tasks Task an expert to evaluate a situation and route related evidense Continuously analyze and (re)contextualize events Perform event extraction & analysis tasks Continuously detect event patterns in awareness specifications E2A activities and roles Receive alerts and related evidence Detect anomalies from observation of normal situations Generate and route alerts providing evidence Administrators Users/other systems E2A-based system Event Processing System Capture context info Author awarenessspecifications

Legend Event flow Utilization Tasking Sensor Interface Sensor Interface Event Sources and Interfaces Context Context contexts Telcordia’s Events to Awareness Architecture (E2A) Event Ontology Awareness Specifications Routing & Task Specifications Event Contextualization (EC) Awareness Computation (AC) Content Routing and Coordination Awareness PrimitiveEvents Contextualized Events Axtionable Events (i.e. alerts & task requests) Users Event Extraction & Analysis (EA) Event repositiry Proactive Event analysis Tasking • Continuous stream processing of events for real time event detection  • Event Subscriptions and tasks

Event contextualization Injects primitive events Contextualizes and fuses events Awareness Computation Utilizes user-specified awareness specifications to compute complex events continuously and incrementally Proactively seek missing events Coordination Manages alert and tasking interactions with end-users Manages tasking of event sources Application context(s), event ontology, awareness specifications, and task specifications Permits application-specific customization Achieve contextual, situational, and temporal relevancy E2A Component responsibilities

Complex event sensing Surveillance Critical Infrastructure protection Reconnaissance Broadcast news analysis UAVs/UASs Coordination and adaptation Intelligence gathering involving collaboration of large multi-organizational teams Disaster/crisis mitigation …at a large scale Blue Force Tracking (DoDs Net-Centric Data Strategy) Situation Awareness Case Studies

The Surveillance Problem

Provide situation awareness by automatically delivering alerts and related evidence to the appropriate users Situation understanding involves determining the causes of an alert Supports situation understanding via event drill down Users can view constituent events and evidence Providing Situation Awareness in Video Surveillance

Surveillance Case Study • Event sources • Video cameras, IR, radar, acoustic, images • RFID readers, badge scanners, biometric • People • Surveillance case study characteristics • Video, sound, and images must be analyzed to extract events • Event extraction and analysis by far the costliest operation and this makes resource optimization hard • Events emerge over time and space • Out of order events are typical due to analysis overhead • To provide situation awareness complex events must be mapped into the context of the specific facility/retain under surveillance (i.e., must be re-contextualized form the context of the specific sensors to the context understood by the users) • Windows do not make much sense • Events are often uncertain due to the complexity of the activity they report on (e.g., human behavior) • Events must be detected in “human” real-time to enable responce to security threats Situational relevancy • …..

Event-driven collaboration of large, multi-organizational teams using CT analysis tools and operating in dynamically changing situations Reduce information overload, and improve decision-making Real time enterprise adaptation as the situation evolves The Intelligence Gathering Problem

Event sources Information/knowledge sources (e.g., open sources in the web), people Policies, processes, resources, Analysis algorithms (e.g., text analysis, evidential reasoning) Intelligence Gathering case study characteristic Events are typically heterogeneous Events must be mapped and evaluated into many different contexts reflecting jurisdictions, organizations, teams, and activities To determine compliance with a policy defined in another context To determine whether to start or adapt a process defined in a different context Out of order events due to analysis overhead and human decision making Events are often uncertain due to the complexity of the activities monitored (e.g., human behavior) and due to gaps in available information Events must be detected in “human” real-time to be able to respond to threats Event-driven process adaptation is common Intelligence Gathering Case Study

A Context Network for Intelligence Gathering Federal Relations Policy & resource flow Event flow DHS 1 n Policies: 1 - Federal Search Warrant 2 - FBI Affidavit 3 - NJ Search Warrant 4 - DHS Notification 5 – Information sharing FBI 4 2 Texas 2 3 2 NJ 5 6 k Activities and processes: 1 - CBP Admission 2 - DHS Notification 3 - Search Warrant 4 - Database Search 5 - Investigation 6 - Event subscription 3 Task force 4 Austin CBP 5 Events/Resources: 1 - Person enters the US 2 - Group active in the US 3 - Person belongs to group 4 – Person belongs to active group in the US m 1 1 Mary Carol Bob Yanni Alice Xavier John 3 4

Situation awareness Teamwork awareness Ongoing policy compliance Dynamic adaptation to reflect changes in the events Process adaptation Context net adaptation Providing Situation Awareness in Intelligence Gathering

Enabling Net-Centricity  Data Strategy The Department of Defense Strategy To move from privately owned and stored data in disparate networks and within legacy systems/applications to an enterprise information environment where authorized known and authorized unanticipated users can access any information and can post their contributions for enterprise-wide access. Producer and Developer Consumer Consumer Producer Ubiquitous Global Network System 1 Data Security Services (e.g., PKI, SAML) Metadata Catalogs System 2 Data Shared Data Space Enterprise & Community Services . . . Application Services (e.g., Web) Metadata Registries System N Data Developer • From Producer-centric: • Multiple calls to find data • Private data – only supports planned consumers • Data translation needed for understanding when pulled from multiple sources • To Consumer-centric: • Data is visible, accessible and understandable • Shared data – supports planned and unplanned consumers • Shared meaning of the data enables understanding

B A R R I E RB A R R I E R B A R R I E R B A R R I E R End-User Producer End-User Consumer Barriers to Identifying, Accessing and Understanding Data “What data exists?“ “How do I access the data?” “How do I know this data is what I need?” “How can I tell someone what data I need?” “How do I share my data with others?” “How do I describe my data so others can understand it?” User knows data exists and can access it but may not know how to make use of it due to lack of understanding of what data represents ? User is unaware this data exists User knows this data existsbut cannot access itbecause of organizational and/or technical barriers Organization “C” Organization “A” Organization “B” Data Strategy Approach: Communities of Interest, Metadata Registry Data Strategy Approach: Discovery Metadata Data Strategy Approach: Web Enabling, Web-service Enabling

Publishing and Subscribing of Data & ServicesSupporting Both Known and Unanticipated Authorized Users Data Producer Data Consumer Data exchanged across engineered, well-defined interfaces System B System A Known User of System A Data Publish Structural and Semantic Metadata Publish Data and Services All Data Assets are Tagged with DoD Discovery Metadata Specification (DDMS) Metadata Publish Discovery Metadata DoD Metadata Registry “Pull” Structural and Semantic Metadata “Pull” Data Query Catalogs and Registry DoD Discovery Catalogs DoD Service Registry System X “Shared Space” Leverages Service Oriented Architecture Unanticipated Authorized User of System A Data

Thank you for your attention! Dimitrios Georgakopoulos (dimitris@ research.telcordia.com)

Backup Slides

Legend Event flow Utilization Tasking Sensor Interface Sensor Interface Event Sources and Interfaces Context Context contexts Telcordia’s Events to Awareness Architecture (E2A) Event Ontology Awareness Specifications Routing & Task Specifications Event Contextualization (EC) Awareness Computation (AC) Content Routing and Coordination Awareness PrimitiveEvents Contextualized Events Axtionable Events (i.e. alerts & task requests) Users Event Extraction & Analysis (EA) Event repository Proactive Event analysis Tasking • Continuous stream processing of events for real time event detection  • Event Subscriptions and tasks

A Context typically contain information about: Entities (e.g., actors or objects or interest) Activities and state changes of the entities Time interval of those activities and state changes Spatial coordinates in which the entities are situated Relationships of entities and activities to other contexts Contexts contain both current and historical info Context management E2A permits the initial modeling of one or more application specific contexts the relationships between them Event Contexts and Context Management

Tracks of people within facility Lobby Hall Rm1 Rm2 Rm3 Rm4 Rm5 Rm6 A Simple Context for Surveillance Facility context dynamically correlates and tracks events from multiple cameras • Facility Space Hierarchy • Spaces are organized into a containment hierarchy with the rooms interconnected by portals • Site-specific attributes: e.g., name, secure, public, etc. • Identities • Partial information on specific people who may use the facility • Site-specific attributes: employee, security clearance,group, etc. • Entities that move about the facility over time • Usually people, though the idea extends to portable objects, like brief cases and documents • Have a source-independent sequence of locations (supported by object tracking) of how the it changed positions over time • Identity of the movable object may be known with some degree of certainty • Pedigree information concerning the above

Steps performed upon receipt of a primitive event: Correlate event parameters and event source metadata with the information of the target and other related contexts Incrementally fuse the primitive event with the info already present in the context Incrementally publish the resulting contextualized events to its subscribers Example: When a person enter a room in a facility, the location of the person is updated in the facility context and fused with the location of the camera Event Contextualization

VEAS-provided customization permits users to specify: What types of events are of interest How to detect them When Where Which method to use Who should be alerted What/how event evidence and pedigree should be presented to each user Awareness Specification

E2A surveillance ontology defines what type of events are of interest: Event types are defined formally in OWL Existing event ontologies can be imported and used New event ontologies can be created and existing ones can be modified via Protégé to provide site-specific and situation-specific customizations Ontology provides an agreement about situation- and site-specific events of interest Example: ZoneVisit Supported by: Protégé, Awareness Computation Event Ontology

Specifications Build from interconnected event operators Example: “Gale’s desk monitor” detects if an object has been taken from her desk during her absence Operators Perform processing on events Examples: generic filter, custom set difference “Anybody but owner in target office” Interconnections define contracts Specify the event flow between operators Define event types of the flowing events VEAS users author interconnections by utilizing event types defined in the surveillance ontology Example: ZoneVisit event type flows from “Owner in target office” to “Anybody but owner in target office” Awareness Specification (How Event Patterns are Specified)

Contextualized event operators Subscribe to contextualized events and can be customized to filter such events Alert delivery operators submit alerts requests (by issuing actionable events) to E2A’s Coordination component Proactive event production operators submit task requests (by issuing actionable events) to E2A’s Coordination component Stream processing operators OR: computes a union of its input streams Difference: computes a set of difference of input streams Relational algebra operators Filtering: culling of uninteresting events Joining: combines related events from multiple sources into a composite event Groupingand aggregation: regrouping and aggregations of events or multiple events Statistical and sampling operators Sampling operators can be added to compute changes in rate of occurrence of a specific event type Statistical operators can be introduce to utilize learned patterns of normal behavior to detect statistical anomalies Extensible pallet of operators Core Awareness Operator Classes

E2A’s coordination component embodies the capabilities of a workflow management system Rich-media dataflow type Accepts actionable events from Alert Delivery and Proactive Event Production operators Routes alerts and evidence to the user role(s) specified in the alert delivery operators Integrates external programs that can interact with event sources for tasking them to produce a specific event or events or a specific type managing them (e.g., changing their settings) Coordination for Alert Delivery and Proactive Event Production

Situation Awareness Telcordia’s E2A Architecture and Three Case Studies