http://www.fi-ware.eu http://www.fi-ppp.eu

1. FIWARE WP6 GEs OverviewSergio García GomezTelefónica Digital, WP6 Leader & Architectsergg@tid.es http://www.fi-ware.eu http://www.fi-ppp.eu

2. WP6 overview WP6 provides a set of GEs for gathering, processing, interchanging and exploiting data at large scale. CDVA

3. Publish Subscribe (Context) Broker

4. Publish Subscribe (Context) Broker • It enables publication of context information by entities (Context Producers), so that published context information becomes available to other entities (Context Consumers), which are interested in processing the published context information. • Applications or even other GEs in the FI-WARE platform may play the role of Context Producers, Context Consumers or both. • Producers and Consumers are totallydecoupled. • Two ways communications: push and pull towards both the Context Producer and the Context Consumer. • Data persistence of contextevents. • Basedon NGSI-9 and NGSI-10 specifications. • Twodifferentimplementationsavailable in FI-WARE, by Telecom Italia and Telefonica.

5. Big Data

6. Big Data • Data analysts and applications that want to extract some knowledge from big sets of data through Map&Reduce algorithms. • Two working modes • Batch processing mode (latency is not important) based on Hadoop (de facto standard) and MongoDB • Stream processing mode (latency is important). Stream injectors + data buckets feeding the processing module. • Plugable intelligence modules both for batch & stream, without taking care of distribution/scalability. • Following the EC recommendation, the existing implementation is being superseded by the one used in Telefonica for commercial services (COSMOS). • Planned to be available by the end of 2nd major release.

7. Big Data Architecture

8. Complex Event Processing CEP

9. Complex Event Processing • Users: applications playing the role of event producers (source point of events) and/or event consumers (sink point of events). • Total decoupling of both roles is realized and complex situations rather than simple events are detected. • It analyses event data in real-time, generates immediate insight and enables instant response to changing conditions.

10. Complex Event Processing • While standard reactive applications are based on reactions to single events, the CEP GE reacts to situations rather than to single events. • A situation is a condition that is based on a series of events that have occurred within a dynamic time window called processing context. • Situations include composite events (e.g., sequence), counting operators on events (e.g., aggregation) and absence operators.

11. Architecture / How it works EventProcessing Networks (EPNs)

12. Architecture / How it works (II) - Start up Upon startup it is supplied with server connector which handles all communication of the adapter with CEP runtime. - Run time Constantly polls data source for changes (1,2), transforms an entry within the data source into CEP readable event object (3), sends the information via server connector to the server (4), without being aware of the underlying communication infrastructure which the connector uses to establish the connection &send the data to the server.

13. Compressed Domain Video AnalysisCDVA

14. CDVA • Users: all applications that want to extract meaningful information from video content and that need to automatically find characteristics in video streams on given tasks. • CDVA is a set of tools for analysing video streams in the compressed domain. Its purpose is to avoid costly video content decoding prior to the actual analysis. • Thereby, the tool set processes video streams by analysing compressed or just partially decoded syntax elements. • The GE can work for previously stored video data as well as for video data streams (e.g., received from a camera in real time).

15. Architecture / How it works

16. CDVA APIs & standards • ONVIF specifications and OASIS Webservices Notification standard define XML structures and elements used within the notification module of CDVA. • Codoan tools use RTP (RFC3550) and RTSP (2326) as transport protocols. • Common video codecs are integrated in the tool stack. E.g. H.242/AVC, MPEG-4. • Descriptions of RTP/RTSP payload can be used, for instance RFC3984 for H.242/AVC. • ISO Base Media File Format as standardized in ISO/IEC 14496-12

17. Location Server

18. Location Server • Users: Any application that aims to retrieve mobile device positions and Location area events. • 3rd partylocationclients: uses OMA MLP or OMA RESTful Network API • Mobile end-users: OMA SUPL (SecureUserPlane) • Addresses issues related to Location of mobile devices in difficult environments such as urban canyons and light indoor environments where the GPS receiver in the mobile device is not able to acquire GPS signals. • It improves GPS coverage whilst waiting for a GPS position, which helps to enhance the user experience of end-users (mobile), and the performance of applications requesting the position of mobile devices.

19. Architecture / How it works • Elements: • 1. SLP = SUPL Location Platform, containing: SPC (SUPL Positioning Centre), QoP manager and Access Control&Privacy Management. • 2. SET = SUPL Enabled Terminal. Talks to SLP with SMS and SUPL over TCP/IP. • 3. GPS&SBAS Receivers. Talk to SLP over TCP/IP. • 4. Third Parties. Talk to SLP with MLP (or NetAPI Terminal Location, with limited functionallity) over TCP/IP.

20. Query Broker

21. Query Broker • Unified access to distributed & heterogeneous data (especially, MM) repositories. • Abstraction from heterogeneous retrieval paradigms in the underlying DB & search engines. • Capable to handle queries formulated in any of a defined set of query languages/APIs (SQL, XQuery, SPARQL) • Incoming queries converted into an internal abstract format MPQF (extends XQuery functionalities). • MPEG Query Format (MPQF) supports most functions in traditional query languages and also incorporates several types of multimedia specific queries (temporal, spatial…).

22. Query Broker Architecture • Two main query processing strategies are supported: • Local processing: when registered and participating retrieval systems are able to process the whole query locally. • Distributed processing: registered and participating retrieval systems that allow distributed processing on the basis of a global data set.

23. Metadata Pre-processing

24. Metadata Pre-processing • Usage/Users: applications that need to convert metadata formats or need to generate objects (as instantiation of classes) that carry metadata information. • In real life various components implementing different metadata formats need to inter-work and then requirements to transform metadata appear. • Typically products from different vendors are plugged together. In this case, the “Metadata Pre-processing” GE acts as a mediator between the various products.

25. Metadata Pre-processing features • Timed meta-data is received and transformed. Not restrictive to specific metadata schemes. • The input stream is encapsulated in the Real-time Transport Protocol (RTP). • Input and output data format is XML. Alternatively, the input data can be transformed into serialized Java classes using XSLT. • Encapsulation of transport and metadata transformation can be implemented as-a-service, usable from other web applications or components.

26. Semantic Annotation

27. Semantic Annotation • Users/Usage: applications willing to enrich content by the means of: • Augmented content (news, books, etc): additional information + links to LOD • Filtering and search: LOD resources used as categories/tags. • “Html snippets”, describing content for dbpedia entries. • Itperforms "NamedEntityRecognition” & semantically links themwithLinked Open Data (LOD) objects: persons, places and organizations in a text. • Each entity is passed to a semantic broker, which tries to identify the correct correspondence over the most used linked open data repositories (dbpedia, geonames).

28. Semantic Application Support

29. Semantic Application Support • Users/Usage: Semantic web apps needing: • an infrastructure for semantic web apps that support large scale applications including: metadata storage in RDF, publication of RDF triples, querying by SPARQL and inference. • a framework for supporting methodologies and engineering processes related with metadata management and ontology development. • It provides a framework for ontology engineers and developers of semantically-enabled applications offering RDF/OWL management, storage and retrieval capabilities. • An infrastructure for metadata publication, retrieval and subscription(scalable, distributed and secure).

30. KIARA communication middleware

31. KIARA – FI-WARE Advanced Middleware • Flexible, efficient, scalable and secure communication between distributed applications & GEs, based on Data Distribution System • Simple for developers, through simplified declarative IDL and APIs for the usage of the application native data types. • Dynamic and transparent negotiation & selection of the communication mechanisms, protocols and data representations (from SOAP/REST to binary protocols). • High level specification of QoS and security (policies) requirements. • Some additional features: • Run time optimization with embedded compiler/Interpreter • Different communication patterns (RPC, N-to-N, Publish-Subscribe, ...) • Synchronous, asynchronous operations • Extended transport protocols and mechanisms (Shared Memory, UDP, SDN Plugin,...) • High performance dispatching agent (RPC) • C & C++ support in first release

32. KIARA – Components & Advanced features • Application describes its data types and structures • Middleware generates support code at runtime (from IDL& declaration) • Data mapping between app and IDL (renaming, type transformation, ...) • Support of different marshaling formats (CDR, XML, JSON) • Optimized by embedded compiler • Meta-data annotations (QoS, Security, Transport) • Negotiation • Best Transport mechanism / protocol (SDN Support) • Definition of Security policies Application Declarative Data/Functiondescription Security / QoSPolicy API / Data Access IDL Parser Data Mapping Embedded Runtime Compiler/Interpreter Security / QoS Parameter Marshalling / Serialization Interaction / Service Protocols • IDL Interface Definitions • KIARA IDLwith SecurityAnnotations(Based on OMG IDL) • WADL CDR XML JSON DDS REST Monitoring QoS Negotiation Service Transport- / Wire- Protocols RTPS HTTP Security - Network Transport Protocols Transport Mechanisms UDP TCP TLS, DTLS Shared Memory Backpla/ Fabric SDN Plugin Data Transfer

33. Thanks !! http://www.fi-ppp.eu