We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Data.Middleware - FIWARE Forge Wiki

FIWARE.OpenSpecification.Data.Middleware

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Data.Middleware
Chapter Data/Context Management,
Catalogue-Link to Implementation <Example GE>
Owner FI-WARE KIARA..., Christof Marti...


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


Copyright

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

This specification describes the Advanced Communication Middleware GE, which enables flexible, efficient, scalable, and secure communication between distributed applications and to/between FI-WARE GEs.

In contrast to other GEs, the Advanced Communication Middleware GE is not a standalone service running in the network, but a set of compile-/runtime tools and a communication library to be delivered with the application.

It supports various communication patterns, like Publish-/Subscribe (PubSub), Point-To-Point or Request-/Reply (RPC). Besides the advanced mode supporting binary data encoding, enhanced Quality of Service (QoS) and security features it also provides backward compatibility to traditional REST based WebServices.

The following layer diagram shows the main components of the Advanced Communication Middleware GE.

Advanced Middleware Architecture Overview
Advanced Middleware Architecture Overview


In the above layer diagram the principle communication flow goes from top to bottom for sending data, respectively from bottom to top for receiving data. As in a typical layer diagram, each layer is responsible for specific features and builds on top of the layers below. Some modules are cross cutting and go therefore over several layers (e.g. Security and Transport Mechanisms).

What follows is a short description of the different layers and components.

API & Data Access

Applications access the communication middleware using a set of defined function calls provided by the API-layer. The usage may vary depending on the communication pattern (see below) that the application uses.

The main functionality of the API & Data Access Layer is to provide the mapping of data types and Function Stubs/Skeletons (for the request/response pattern) or DataReaders/-Writers (for the publish/subscribe or point-to-point pattern).

The Advanced Middleware GE provides two variants of this functionality:

  • A basic static compile-time Data-Mapping and generation of Function Stubs/Skeletons or DataReaders/-Writers, created by a compile time IDL-Parser/Compiler from the remote service description, which is provided in an Interface Definition Language (IDL) syntax based on the Object Management Group (OMG) IDL (see below) or, in case of WebService compatibility in Web Application Definition Language (WADL) syntax, which is submitted as a W3C draft.
  • An advanced dynamic runtime Data- and Function-Mapping based on a declarative description of the internal data-structures and functions provided by the application and the IDL description of the remote service with an embedded Runtime Compiler/Interpreter

Quality of Service (QoS) parameters and Security Policies may be provided through the API and/or IDL-Annotations. This information will be used by the QoS and Security modules to ensure the requested guarantees.

Depending on the communication pattern, different communication mechanisms will be used.

  • For publish/subscribe and point-to-point scenarios, the DDS services and operations will be provided. When opening connections, a DataWriter for publishers/sender and a DataReader for subscribers/receivers will be created, which can be used by the application to send or receive DDS messages.
  • For request/reply scenarios the Function Stubs/Skeletons created at compile- or runtime can be used to send or receive requests/replies.

Marshalling

Depending on configuration, communication pattern and type of end-points the data will be serialized to the required transmission format when sending and deserialized to the application data structures when receiving.

  • Common Data Representation (CDR) an OMG specification used for all DDS/RTPS and high-speed communication.
  • Extensible Markup Language (XML) for WebService compatibility
  • JavaScript Object Notation (JSON) for WebService compatibility

Wire Protocols

Depending on configuration, communication pattern and type of end-points the matching Wire-Protocol will be chosen.

  • For publish/subscribe and point-to-point patterns the Real Time Publish Subscribe (RTPS) Protocol is used.
  • For request/reply pattern with WebService compatibility the REST/HTTP Protocol is used
  • For request/reply pattern between DDS end-points the Real Time Publish Subscribe (RTPS) Protocol is used
  • For high-performance communication the wire protocol may be skipped entirely and set up directly on lower layer communication mechanisms and protocols

Dispatching

The dispatching module supports various threading models and scheduling mechanisms. The module provides single-threaded, multi-threaded and thread-pool operation, both in synchronous and asynchronous fashion. Priority or time constraint scheduling mechanisms can be specified through QoS parameters.

Transport Mechanisms

Based on the QoS parameters and the runtime-environment, the QoS module will decide which transport mechanisms and protocols to choose for data transmission.

In Software Defined Networking (SDN) environments, the SDN plugin will be used to get additional network information (e.g. from the I2ND GE) or even provision the network to provide the requested quality of service or privacy.

Transport Protocols

All standard transport protocols (TCP, UDP) as well as encrypted tunnels (TLS,DTLS) are supported. For high-performance communication in specific environments optional optimized protocols will be provided (Memory Mapping, Backplane/Fabric, ...).

Security

The security module is responsible for authentication of communication partners and will ensure in the whole middleware stack, the requested data security and privacy. The required information can be provided with Security Annotations in the IDL and by providing a security policy via the API.

Negotiation

The negotiation module provides mechanisms to discover or negotiate the optimal transmission format and protocols when peers are connecting. It automatically discovers the participants in the distributed system, searches through the different transports available (shared memory and UDP by default, TCP for WebService compatibility) and evaluates the communication paradigms and the corresponding associated QoS parameters and security policies.

Basic Concepts

In this section several basic concepts of the Advanced Communication Middleware are explained. We assume that the reader is familiar with the basic functionality of communication middleware like CORBA or WebServices.

Communication Patterns

We can distinguish between three main different messaging patterns, Publish/Subscribe, Point-to-Point, and Request/Reply, shown schematically bellow:

Publish/Subscribe Pattern
Publish/Subscribe Pattern


Point-To-Point Pattern
Point-To-Point Pattern


Request/Reply Pattern
Request/Reply Pattern


All traditional middleware technologies available implement one or many of these messaging patterns and may incorporate more advanced patterns on top of them. Most of RPC middleware is based on the Request/Reply pattern and more recently, extends towards support of Publish/Subscribe and/or the Point-To-Point pattern.

W3C Web Service standards define a Request/Reply and a Publish/Subscribe pattern which is built on top on that (WS-Notification). CORBA, in a similar way, build its Publish/Subscribe pattern (Notification Service) on top of a Request/Reply infrastructure. In either case the adopted architecture is largely ruled by historical artefacts instead of performance or functional efficiency. The adopted approach is to emulate the Publish/Subscribe pattern on top of the more complex pattern thus inevitably leading to poor performance and complex implementations.

The approach of the Advanced Middleware takes the other direction. It provides native Publish/Subscribe and implements the Request/Reply pattern on top of this infrastructure. Excellent results can be achieved since the Publish/Subscribe is a meta-pattern, in other words a pattern generator for Point-To-Point and Request/Reply and potential alternatives.

Interface Definition Language (IDL)

The Advanced Middleware GE supports a novel IDL to describe the Data Types and Operations. Following is a list of the main features it supports:

  • IDL, Dynamic Types & Application Types: It supports the usual schema of IDL compilation to generate support code for the data types, but also, dynamic runtime type creation, allowing the applications to use its own data structures without forcing to use the IDL compiler generated types. See the data-access feature bellow for a complete description.
  • IDL Grammar: An OMG-like grammar for the IDL as in DDS, Thrift, ZeroC ICE, CORBA, etc.
  • Types: Support of simple set of basic types, structs, and various high level types such as lists, sets, and dictionaries (maps).
  • Type inheritance, Extensible Types, Versioning: Advanced data types, extensions, and inheritance, and other advanced features will be supported.
  • Annotation Language: The IDL is extended with an annotation language to add properties to the data types and operations. These will, for example, allow adding security policies and QoS requirements.
  • Security: The IDL allows for annotating operations and data types though the annotation feature of our IDL, allowing setting up security even at the field level.

For compatibility with REST-based WebServices, the Middleware also supports the W3C draft submission Application Definition Language (WADL).

Data Access Layer

The Advanced Middleware supports an advanced set of data types:

  • Static Data Types: Types generated via the IDL compiler in compliance with traditional approaches to warrant backward compatibility.
  • Dynamic Middleware Data Types: Data types generated by the middleware at runtime.
  • Application Native Data Types (new technique): To use application native data types, where the application provides type marshalling and data management using a declarative and/or procedural approach. To this end the Advanced Middleware GE provides two different mechanisms:
  • Letting the application developer provide his own data type plug-in using calls to low-level routine in the middleware that then perform the required marshalling and other operations. Some basis support for this is already provided by RTI-DDS (and also OpenDDS).
  • Exposing an API to describe the application data type and generating the required marshalling and management operations at run-time by:
  • Interpretation: Generating an intermediate byte-code to implement the operations and interpret this byte-code by a small “virtual machine”, or
  • Compilation: Generating an intermediate representation but compiling this data access code to native code with a JIT compiler (e.g. by an embedded LLVM-based compiler). This includes integrating and optimizing (e.g. inlining) the code for performing the chosen data marshalling and submission to the transport mechanism.

Main Interactions

As explained above, the middleware can be used in different communication scenarios. Depending on the scenario, the interaction mechanisms and the set of API-functions for application developers may vary.

API versions

There will be two versions of APIs provided:

  • Basic API
  • Static compile-time parsing of IDL and generation of Stub-/Skeletons and DataReader/DataWriter
  • Backward compatible to RPC-DDS and DDS applications
  • Advanced API
  • Dynamic runtime parsing of IDL and generation of Stub-/Skeletons
  • Mapping of application data types and functions
  • Advanced security policy and QoS parameters
  • Support for high-performance transport mechanisms and protocols
  • REST Webservice support

Classification of functions

The API-Functions can be classified in the following groups:

  • Preparation: statically at compile-time (Basic API) or dynamically at run-time (Advanced API)
  • Declare the local applications data types/functions (Advanced API only)
  • Parsing the Interface Definition of the remote side (IDL-Parser)
  • Building the data-/function mapping (Advanced API only)
  • Generate Stubs-/Skeletons, DataReader-/Writer (Compiler-/Interpreter)
  • Build your application against the Stubs-/Skeletons, DataReader-/Writer (Basic API only)
  • Initialization:
  • Set up the environment (global QoS/Transport/Security policy,...)
  • Open connection (provide connection specific parameters: QoS/Transport/Security policy, Authentication, Tunnel encryption, Threading policy,...)
  • Communication
  • Send Message/Request/Response (sync/async, enforce security)
  • Receive Message/Request/Response (sync/async, enforce security)
  • Exception Handling
  • Shutdown
  • Close connection (cleanup topics, subscribers, publishers)
  • Free resources

Detailed descriptions of the APIs and tools can be found in the User and Programmers guide, which will be updated for every release of the Advanced Middleware GE.

Basic Design Principles

Implementations of the Advanced Middleware GE have to comply with the following basic design principles:

  • All modules have to provide defined and documented APIs.
  • Modules may only be accessed through these documented APIs and not use any internal undocumented functions of other modules.
  • Modules in the above layer model may only depend on APIs of lower level modules and never access APIs of higher level modules.
  • All information required by lower level modules has to be provided by the higher level modules through the API or from a common configuration.
  • If a module provides variants of internal functionalities (e.g. Protocols, Authentication Mechanisms, ...) these should be encapsulated as Plugins with a defined interface.


Detailed Specifications

Following is a list of Open Specifications linked to this Generic Enabler. Specifications labeled as "PRELIMINARY" are considered stable but subject to minor changes derived from lessons learned during last interactions of the development of a first reference implementation planned for the current Major Release of FI-WARE. Specifications labeled as "DRAFT" are planned for future Major Releases of FI-WARE but they are provided for the sake of future users.

Open API Specifications


Re-utilised Technologies/Specifications

The Advanced Middleware GE is a set of communication libraries and tools to be delivered with applications/services. It is not a RESTful service running as a standalone component, but in the final advanced version it however can be used to provide or consume RESTful web services.

The technologies and specifications used in basic version of this GE are:

The Advanced Version will use and support additional technologies:

  • RESTful web services
  • HTTP/1.1 (RFC2616)
  • JSON and XML data serialization formats.

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions

  • Data refers to information that is produced, generated, collected or observed that may be relevant for processing, carrying out further analysis and knowledge extraction. Data in FIWARE has associated a data type and avalue. FIWARE will support a set of built-in basic data types similar to those existing in most programming languages. Values linked to basic data types supported in FIWARE are referred as basic data values. As an example, basic data values like ‘2’, ‘7’ or ‘365’ belong to the integer basic data type.
  • A data element refers to data whose value is defined as consisting of a sequence of one or more <name, type, value> triplets referred as data element attributes, where the type and value of each attribute is either mapped to a basic data type and a basic data value or mapped to the data type and value of another data element.
  • Context in FIWARE is represented through context elements. A context element extends the concept of data element by associating an EntityId and EntityType to it, uniquely identifying the entity (which in turn may map to a group of entities) in the FIWARE system to which the context element information refers. In addition, there may be some attributes as well as meta-data associated to attributes that we may define as mandatory for context elements as compared to data elements. Context elements are typically created containing the value of attributes characterizing a given entity at a given moment. As an example, a context element may contain values of some of the attributes “last measured temperature”, “square meters” and “wall color” associated to a room in a building. Note that there might be many different context elements referring to the same entity in a system, each containing the value of a different set of attributes. This allows that different applications handle different context elements for the same entity, each containing only those attributes of that entity relevant to the corresponding application. It will also allow representing updates on set of attributes linked to a given entity: each of these updates can actually take the form of a context element and contain only the value of those attributes that have changed.
  • An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. Events typically lead to creation of some data or context element describing or representing the events, thus allowing them to processed. As an example, a sensor device may be measuring the temperature and pressure of a given boiler, sending a context element every five minutes associated to that entity (the boiler) that includes the value of these to attributes (temperature and pressure). The creation and sending of the context element is an event, i.e., what has occurred. Since the data/context elements that are generated linked to an event are the way events get visible in a computing system, it is common to refer to these data/context elements simply as "events".
  • A data event refers to an event leading to creation of a data element.
  • A context event refers to an event leading to creation of a context element.
  • An event object is used to mean a programming entity that represents an event in a computing system [EPIA] like event-aware GEs. Event objects allow to perform operations on event, also known as event processing. Event objects are defined as a data element (or a context element) representing an event to which a number of standard event object properties (similar to a header) are associated internally. These standard event object properties support certain event processing functions.
Personal tools
Create a book