Representation and Exploitation of Contextual Knowledge in Maritime Surveillance

1. Representation and Exploitation of Contextual Knowledge in Maritime Surveillance Galina Rogova Encompass Consulting Jesús García University Carlos III of Madrid

2. Outline Introduction Piracy Problem Maritime surveillance Fusion in Maritime Surveillance Level 1 HLF Context in Information Fusion Reasoning and uncertainty Ontological representation Ontology categorization Knowledge representation Conclusions

3. Piracy Threat Intent: Goals (profit) Objectives Directives Intent Capability • Opportunity* : • limitless range of vulnerable targets • enormous volume of commercial ships • the need for ships to pass through congested (and ambush-prone) maritime choke points • vast territorial waters • skeleton crews • limited resources for monitoring territorial waters and ports • lack of international laws Opportunity Immanent Threat (Tri-partite Whole) Intent Capability Opportunity Potential Threat (two-part whole) *Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.

4. Piracy Threat* (cont’d) Intent Capability • Opportunity (cont’d): • lax naval and coastal security • corruption and easily compromised judicial structures • situation in Somalia • ready willingness of ship owners to pay ransom • limited inter-government cooperation • Capability: • supply of automatic weapons available thanks to the proliferation of small arms • well organized crime/terrorist rings possesseing: • ability to manufacture false identity papers for the crew and vessel and fake cargo documents • a broker network to sell the stolen goods • general poverty Opportunity Immanent Threat (Tri-partite Whole) Intent Capability Opportunity Potential Threat (two-part whole) *Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.

5. Piracy and Terrorism: Differences • Different goals • Piracy is an economically driven phenomenon • Terrorism aims at undermining the oceanic environment to secure political, ideological or religious imperatives • Piracy specific consequences: undermining and weakening government legitimacy by encouraging corruption among elected officials and bureaucrats. • Pirates are not martyrs, which affects the attack implementation and means, e.g. no suicide attacks, no bomb building capabilities • May require specific surveillance means

6. Piracy and Terrorism: Commonality • Possible common consequences: • a direct threat to the lives and welfare • direct economic impact in terms of fraud, stolen cargos and delayed trips, which could undermine a maritime state’s trading ability • major environmental catastrophe • Similar opportunities due to similar limitless range of vulnerable targets: • enormous volume of commercial ships • necessity to pass through congested (and ambush-prone) maritime choke points *A Dali

7. Piracy and Terrorism: Commonality (cont’d) • Common tactics are possible, e.g. terrorists could hijack ships carrying huge loads of highly flammable natural gas to undertake a suicide mission or to ram a hijacked vessels against the cruise center, the container terminal or an oil refinery • Piracy and terrorist attacks are carefully planed and orchestrated, both types require significant resources and well organized crime/terrorist rings. • Willingness to risk their lives although often with different goals

8. Piracy and Maritime security Piracy problem is to a great degree similar to the terrorism problem and represents a part of the general problem of maritime security. Surveillance technology developed in the maritime security domain can be adopted to fight piracy

9. Maritime Surveillance: Why and How “In a world where small water craft can be turned into weapons against navy destroyers and pirates can hold ships for ransom, surveillance of the sea is of increasing importance.” Potential capability monitoring requirements: Small arm trafficking (capability monitoring) Pirate organization network (intent monitoring) Immanent threat monitoring: Monitoring of high vulnerability areas (ports, navigation routes) Realized threat Secondary underlying threat *http://www.nurc.nato.int/research/msa.htm

10. Maritime Surveillance “Surveillance resources are inadequate to monitor the world’s shipping channels tools that help maritime surveillance analysts identify suspicious activity are extremely valuable.” * *http://www.nurc.nato.int/research/msa.htm Overall goal: Alert decision makers on vessel behaviors of interest with minimal false alarm

11. Data & Information sources(examples) • Vessel Traffic Systems (VTS) • Port Traffic Management System (PTMS) • Special sensors and systems for oil and gas ports increase the safety of vessel transit and berthing. • AIS • Civilian and military sensors • Coastal patrols • Unmanned aerial vehicles (UAVs), • Unmanned surface vehicles (USVs) • Spot reports • Visual sightings • General communication reports from coastal patrols. • Ex: shore-based emergency communications, search and rescue service, etc • Documents, procedure • Open source information (web-based, radio,…) • Ref.

12. Data and Information Categories Observables: obtained by persistent surveillance, i.e. continuous tracking and tracing of vessels with observations systems Ex.: statements about the size of vessels and ships (large, small), speed(slow, high), and track behavior (loitering, stopped, and continuously ahead). A priori knowledge including vessel characteristics (size, speed…), current practices and trade activities about information about recent activities of groups and persons and events . Learned knowledge Contextual information. 12

13. Maritime Surveillance (Challenges) • Large and heterogeneous areas • Enormous volume of commercial ships • Diverse operations and Decision makers • Huge amount of Information of variable quality • Dynamic and unpredictable environment • Heterogeneous sources of information (sensors, human reports) • Delay in data transmission 13

14. Fusion and Surviellance L1: continuous detection, ID, tracking and tracing of vessels with observation systems • Observables by persistent surveillance • Sensors • Open source information • Intelligence reports • Observers’ reports • Essential facility reports • Knowledge base: • Rules • A priori beliefs • … Context L2,3: Situation & threat assessment: Where, what, who

15. Situation and Threat assessment • Higher Level Fusion (situation and threat assessment) produces knowledge about the state of the environment by evaluating relations between entities, their aggregations, characteristics and behavior within a specific context • HLF provides multiple decision makers with answers to the questions such as: • What is going on • Is anything unexpected or suspicious going on? • Where? • What are the possible explanations (what does it mean)? • What is the impact (what can be expected in the future)?

16. Challenges: Situation and Threat Assessment How to transform data and information into knowledge and deal with: • complex distributed system modeling • scalability problem • time constraints and time delays • communication within and between systems, decision makers, ,various agencies, countries,… • secure information exchange • heterogeneous data, e.g. structured (databases) and unstructured data (text messages) • designing a mechanism for inferencing from a given state of knowledge to a possible explanation for a hypothesized situation • Taking into account information quality (reliability, trust, relevance) • Uncertain and unknown context.

17. Context The Merriam-Webster dictionary: “The events or circumstances that form, or influence, the environment, within which something exists or takes place.” The problem “Given an entity of interest (a physical object, a situational item, and an event) what context or a sequence of contexts can be formed , such that a task about this entity can be accomplished.”* L. Gong, 2005

18. Context Context* “Context-for X” items externally related to X and selected to help better understand and manage a given situation. “Context-of X” a situation of interest containing a set of entities and relations between them and providing constraints and expectationsabout X How to represent context-for X? How to choose context-of X? X (“reference items”) represents any physical or conceptual entity and event to be considered, e.g. a boat moving towards a cruse ship. * L. Gong, 2005 Steinberg & Rogova (2008) 18

19. “Context Of” (CO) • Guides response to a situation (“what can be done given a certain context” ). • Provides information for detection of possible underlying threat. • Offers dynamics of the situation (in the context of he willingness to pay ransom) • Facilitates effective communications between actors by constraining the meanings of messages. • Constrainsgoals, objective, functions, actions of the situation responders Example: the geophysical and geopolitical world situation such as situation in Somalia, relations between different countries in Malacca Strait. CO is characterized by “problem variables” (entities, relationships, and activities). CO is either given, estimated, or discovered

20. “Context For” (CF) • CF is used for: • Action optimization • Better situation understanding • CF is characterized by “contextual variables” (auxiliary) • May be static e.g. maps, or dynamic e.g. weather or geopolitical situation. • CF is either selected or obtained by direct observations

21. Situation Management • Re-planning • Response • Prevention • Resource management Context in Fusion and Situation managements • Actions • Decisions FUSION PROCESSES • Observations • Sensors • Open source information • Intelligence reports • Observers’ reports • Essential facility reports • Situational items • Predicted potential threat • Immanent threat • Realized Threat L2 • Objects • Relations • Vulnerabilities • Situations • Threats L1 L3 L4 Expected critical situations • Contextual variables • Constrains for: • Objects • Processes, • Preferences • Hypothesis • Context: • Discovered • Estimated • Predicted • Characteristics • Planned actions • Required resources Domain ontology • Operational knowledge • Goals, • Objectives • Functions • Plans, Actions • Operational requirements • Constrains for: • Goals • objectives • Functions • Actions • Message Meanings • Plans • Constrains for: • Ontology • Observations • Expected situational items, characteristics CONTEXTS • Knowledge base: • Rules • A priori beliefs • … Rogova, 2009

22. Harbor Context Multiple areas and operations Congestion

23. Malacca Strait Context • Piracy and international issues* • dense trafficking, between neighboring countries with possible tensions between countries and conflicts in the past: • smuggling of weapons, • presence of international peace forces,… • Lack of international piracy laws • Numerous vessels with ‘normal’ missions * A. Dali 23 http://www.??????

24. Harbor traffic rules as context for maritime surveillance Harbor general rules Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destination, arrival/departure times, passengers/cargo, ... Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden Procedures for special vessels Some ships are obliged to use pilotage service: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc) Depending on maneuvering capability, there may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel. 24

25. Domain ontology L1 L2 Fusion Processes L4 L2 Approach • Constraints for: • Objects • Processes • relations CF Current Context (CO) • Constrains for: • Objects • Processes, • Beliefs • Preferences • Hypothesis • Arguments • Environment monitoring • Sensors • Open source information • Intelligence reports • Observers’ reports • Essential facility reports Statistics Rules Arguments Objects Relations • Expected: • objects • situational items • characteristics • behavior • Assessed: • objects • relations • situational items • characteristics • behavior Consistent? Framework for contextual ontology-based reasoning no • Anomaly? • OR • Insufficient Quality of: • Contextual knowledge? • Observations • Fusion processes? Is the belief in anomaly justified?

26. Dealing with Inconsistency • How and whether to change the state of the knowledge (e.g. threat/no threat) due to the arrival of new information if new information contradicts to prior knowledge in the context under consideration. • Involves: • methods for detecting inconsistency • abduction: a means to determine the sources of the contradiction (e.g., whether this contradiction is because of insufficient quality of contextual knowledge, observations, fusion processes) • Anomaly? • OR • Insufficient Quality of: • Contextual knowledge • Observations • Fusion processes? Requires reasoning under uncertainty

27. Inconsistency Detection • May be based on explicit “normalcy” or “anomaly” models by using: • Values of characteristics or behavior of situational items obtained from databases. • Rules, e.g. presence or absence of certain characteristics. • Statistical information obtained from databases (e.g. hypothesis testing) • Problems • Scalability: There are too many patterns of normal behavior • There are less abnormal patterns than normal but the problem with the “black swan.” What is normal depends on context (geophysical, vessel type,…)

28. Inconsistency Detection (cont’d) • Incremental learning operating on evolving data (“on-line stream classification problem”) • Accumulated statistics • Neural networks • Case-based reasoning • Advantages: • Can identify unseen earlier patterns of behavior or characteristics • Drawbacks • A pattern that might be considered as an anomaly, could become normal with time when more information is available. • Anomaly detection triggers the process of believe update to find the source of inconsistency, which requires reasoning under uncertainty • Incremental learning operating not only on training sets but also on evolving data (“on-line stream classification problem*”) • Accumulated statistics • Neural networks • Case-based reasoning • Advantages: • Can identify unseen earlier patterns of behavior or characteristics • Drawbacks A pattern that might be considered as an anomaly, could become normal with time when more information is available. • Requires continuous learning (adaptive processes) • What is normal depend on context (vessel type, weather, tidal status, etc

29. Reasoning Under Uncertainty Quantitative methods (Uncertainty encoded by numbers) Qualitative methods (Uncertainty is handled by manipulation of symbols) • Probability theory • Classical probability (chance) • Bayesian (subjective belief) • Possibility theory (incompleteness) • Fuzzy theory (vague information) • Evidence theory (ignorance, ambiguity) • Dempster-Shafer • Transferable belief • … • Default logic • (incompleteness) • Argumentation • (inconsistency) • … Hybrid Methods Selection of a single formalism of a hybrid system depends on a particular problem

30. Belief Based Argumentation Belief Based Argumentation* is an approach to non-monotonic reasoning under uncertainty combining symbolic logic with belief theories for judging hypotheses about the unknown world by utilizing given knowledge. Quantitative part Computes and combines beliefs that hypotheses and the arguments, which bear on them, are valid. Qualitative part Finds arguments in favor and against a hypothesis about a possible cause consistent with observations. Belief Based Argumentation * Rogova et al (2004,2005) ,R. Haenni, et al, (2000) * Rogova et al (2004,2005) based on R. Haenni, et al, (2000)

31. Why Belief Based Argumentation Allows for: • contextual reasoning with harbour rules and regulations • dealing with uncertainty • reasonng under the open world assumption (incomplet information and non-exhaustive hypotheses) • incorporating subjective knowlegdeinclude both numeric and symbolic information

32. Example Discovery of possible threat from a boat based on: Boat features (speed, direction, type, flag, etc.) Spatio-temporal relations between the suspicious boat and others, or relations between the boat and harbor zones Beliefs assigned to assumptions are based on the observed spatio-temporal relations and the correspondence of the boat behaviour to the rules and regulation • An argument pro hypothesis “treat” for a vessel under a flag of convenience built as a conjunction of the following uncertain assumptions: • A1: the suspicious boat is too close to a vessel sailing in the opposite direction • A3: The vessel following in the opposite direction is a big cruise ship • A4: The suspicious boat is increasing its speed 32

33. Fusion and Surveillance L1: continuous detection, ID, tracking and tracing of vessels with observation systems • Observables by persistent surveillance • Sensors • Open source information • Intelligence reports • Observers’ reports • Essential facility reports • Knowledge base: • Rules • A priori beliefs • … Context L2,3: Situation & threat assessment: Where, what, who

34. Fusion Challenges: Level l Tracking: accurate detection and estimation of targets dynamics Design aspects: Analysis of appropriate fusion architectures and algorithms in this environment Experimental analysis: demonstrate through an operative prototype Focus in efficiency, point to the highest computation load to allow acceptable performance in real time. Use of contextualinformation for adjustment to adapt the algorithms to operating conditions 34

35. Fusion Challenges: Level l Introduction Piracy Problem Scenarios IF in MD Context Info Ontologies Reasoning Conclusions TRACKS MULTI-ALGORITHM TRACKING SYSTEM SENSOR INPUTS 35

36. Sensor Fusion in MD • Example: AIS-radar Fusion system • Distributed architecture with specific logic for each source • Output: global tracks: ID, reference time, WGS84 location, speed/course over ground

37. Sample Scenario • Simulated scenarios and real data analysis with Cape Verde Islands, 7 available sesors (4 RADAR, 3 AIS Stations)

38. Sensor Fusion in MD • Specific sensor processing • Radar with clutter, noise, detection • AIS: ID, speed processing, aperiodic

39. L1 Algorithm design • Local Tracks Management

40. L1 Algorithm Design • Global Tracks Management

41. Geometric context • Parameter tuning • Initializtaion areas • Speed threshold • Maeuvering index • Management cycles

42. Context-aided tracking Context-aided tracking: adaptive estimation of target dynamics accordingly to configuration and uncertainty Ex.: use of known geometrical conditions and sensor performance Use of expected behaviours Examples: radar data processing in airport surface traffic monitorization 42

43. Context-aided tracking Rule-based tracking: rule-based video processing system: split/merge effect Examples: video in airport surface traffic monitorization, sports, parking surveillance, … Introduction Piracy Problem Scenarios IF in MD Context Info Ontologies Reasoning Conclusions 43

44. Fusion Challenges: Level l Adaptable tracking processes Data association/correlation (video) Track initialization / management logic Appropriate tracking filters (ex. EKF/IMM tuning) Object dynamics characterization (ex. manueverability) Cycle management (efficiency) Scenario Context Maps Regulation (speed, size, …) Restricted areas Trafficability maps Introduction Piracy Problem Scenarios IF in MD Context Info Ontologies Reasoning Conclusions 44

45. Knowledge representation: ontologies Ontology is the study of the kinds of things that exist. Provides formally structured context-dependent information about dynamic reality capable of capturing situational entities and the wealth of relations between them. Domain-specific ontological analysis to represent domain specific characteristics of Real World Formal upper level ontological analysis to represent Real World in general Informs Constrains Formal Ontology of a Specific Domain Situation Ontology Uncertainty Dynamic Environment

46. Types of Domain specific Ontologies Domain Specific ontologies: elements in the application domain, such as known types of vessels and ports To acquire data from information sources in a meaningful and consistent manner Situation ontologies. capture a situation or series of states in the application space using concepts from the content ontologies. Situation ontologies are input to search patterns of interest, e.g. the specific intents that we want the system to recognize. Ex.: a search pattern for smuggle might: rendez-vous with another vessel mid-sea or the vessel type does fit the current location and time. [] Ref: A. van den Broek FUSION 2011 Introduction Piracy Problem Scenarios IF in MD Context Info Ontologies Reasoning Conclusions 46

47. Use of Ontologies for Knowledge Representation in Fusion capture domain knowledge to represent objects, objet attributes, and relations between them capture domain knowledge not only about about entities and status, but also domain semantics (concepts, relationships, etc). Explicit representation of formal semantics such as Description Logics enables automated reasoning Ontology-based systems usually have their logical basis in any type of classical logic (extensions of rule systems) 47

48. Representation A priori specialized knowledge (terminological axioms) Geometrical elements of harbor: moving lines, buoys, restricted areas, etc. Vessel classification: vessel types and properties Rules of operation: margin speeds, procedures (crossing only orthogonal), rules of priority Instantiation (extensional axioms) Properties of actual entities of the current scenario Static (e.g., delimitation of harbor zones) Dynamic (e.g., vessel a position) 48

49. Ex.: Harbor traffic rules Harbor general rules Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destiny, arrival/departure times, passengers/cargo, ... Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden Procedures for especial vessels Some ships are obliged to use pilotageservice: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc) There may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel External procedures Outer channels: Merchant ships have to proceed following a “safe speed” avoiding waves to small boats/vessels Zones considered dangerous for maritime traffic due to military exercises 49

50. Representing areas (static) RCC: Region Connection Calculus 50