We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.IoT.Edge.DataConsolidation R5 - FIWARE Forge Wiki

FIWARE.OpenSpecification.IoT.Edge.DataConsolidation R5

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.ArchitectureDescription.IoT.Edge.DataConsolidation
Chapter Internet of Things,
Catalogue-Link to Implementation Cepheus
Owner Orange, Christophe Azemar

Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.

Copyright

  • Copyright © 2012, 2013, 2014 by Orange, Atos
  • Copyright © 2015, 2016 by Orange

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

From the view point of Edge Data Consolidation GE, IoT environments are seen as sets of event-generating resources. This enabler addresses the need for pre-filtering, aggregating and consolidating low-level raw data from such different sources into meaningful events. Many applications depend on such value-added events that are relevant to their needs. These events can be genrated in a generic and applicaton-independent manner with Complex Event Processing (CEP) technology. This is also referred to as event stream analysis, or real time event correlation. Events are published by Event Producers and subscribed to by Event Consumers. Consumers can only subscribe to events in accordance with their granted access rights.

Typical applications that require Data Consolidation GE are sensor network applications, supply chain management, scheduling and control of fabrication lines, smart city and smart buildings applications, home automation, among others. These applications have in common the requirement to detect and process events in real time, or near real time. Key considerations for these types of applications are throughput, latency and the complexity of the logic required. The CEP Engine component of the Edge Data Consolidation GE typically processes input data, in order to generate a smaller set of output data, which avoids upper level software overloading and network flooding.

The Edge Data Consolidation GE handles data streams from IoT devices that cannot continuously be online, for various reasons. Among them are possible network disruptions and power resource constraints. The latter typically requires optimizing and limiting the communications with the IoT resources. For this purpose, local storage is desirable, in order to cache the last available processed data and contexts.

The natural surrounding GEs for Data Consolidation are the IoT Broker, the Configuration Management, and the Protocol Adapter GE.

Although it can be operated standalone, the Data Consolidation GE typically receives events from the Gateway Protocol Adapter GE, and propagates the processed data towards the IoT Broker GE.

It should be noted that Publish-Subscribe and CEP features are also available in other GEs. This should not be seen as useless redundancy: some features are deliberately distributed at different level of granularity, in order to provide reliability to the whole architecture, and also to appropriately dispatch the load at different levels.

The next sections discuss the overall architecture

Internal architecture components diagram

Image:Fiware-cepheus-architecture-overview.png
Edge Data Consolidation GE architecture

The interfaces for this GE are compliant with the FIWARE implementation of the Next Generation Services Interface (NGSI), and more precisely, NGSI-9 and NGSI-10 subsections. It should be noted that the official OMA NGSI Context Management standard neither specifies nor imposes any particular implementation regarding the exchange format, but simply describes NGSI data types as belonging to the "XML Schema Part 2" W3C recommendation. Currently, the GE supports both XML and JSON implementations of the NGSI standard: more information can be found here.

The Edge Data Consolidation GE is currently implemented by the Fiware-Cepheus GEi that is composed of two components : the Cepheus Broker application and the Cepheus CEP application. Both interact with each other using the NGSI-9/10 API.

Cepheus Broker component is responsible of the interactions (subscription/publication/forwarding) with the other GE and NGSI-9/10 compliant devices, while the Cepheus CEP is responsible of the manipulation of data (aggregation, transformation, detection, ...).

Main interfaces

NGSI-9/10 API

The Data Consolidation API is NGSI-9/10 compliant. It accepts events from any NGSI compliant Event Producer.

In the setup shown above, an IoT Agent or any NGSI-9/10 compliant device can register as NGSI context providers, by calling the NGSI-9 registerContext method exposed by the Data Consolidation GE.

After context registration, the IoT Agent or NGSI-9/10 device can send events by calling the NGSI-10 updateContext method of the Data Consolidation GE.

Third party software and other external GEs can subscribe to Data Consolidation GE output events by calling its NGSI-10 subscribeContext methods. For every subscriber, events are propagated by calling their NGSI-10 notifyContext method, which means subscribers must implement this method, in order for them to receive events.

CEP Management API

This API allows the configuration and operation of the Complex Event Processor: it provides features such as setting rules, defining event types, and configuring output event recipients.

Main components

NGSI-9/10 Publish-Subscribe component (Cepheus-Broker)

This component is the core implementation of the Data Consolidation GE NGSI functionality. It allows the Generic Enabler to communicate with the outside world and the other GEs, in an ingoing and outgoing way, and also features publish-subscribe functions. It implements the necessary subset of NGSI-9 and NGSI-10 methods for the purpose. It also allows the GE to register itself to a Context Broker, which is a prerequisite to the propagation of events towards the IoT Broker GE or the Data Context Broker GE (Orion).

This organization enables the creation of a Publish/Subscribe network of NGSI compliant GEs, that expose their services to each others.

It should be noted that NGSI compliance does not imply the implementation of the full set of NGSI methods.

CEP Engine component (Cepheus-CEP)

This is the core functionality of the Data Consolidation GE.

The role of the CEP Engine component is to process events in real time, which mainly involves filtering, aggregating and merging data. This processing is based on a set of rules that can be defined by the CEP administrator.


More information can be found in the [Fiware-Cepheus GEi manual]

Basic concepts

Event concepts

Similar to many topics in science and engineering, the term "event" has different meanings based on who is observing the event and the context of the observation. Generally speaking, an event is something notable that happens in the real world or in a system. But in the computer world, an event is understood as a computer representation of such an occurrence, usually for the purpose of computer processing. Events can contain data, are immutable, but more than one event may record the same activity.

Before the actual event processing occurs, optional event preprocessing can be implemented: it is often referred to as data normalization, validation, prefiltering and basic feature extraction. When required, event preprocessing is often the first step in a distributed, heterogeneous CEP solution. Heterogeneous, distributed event processing applications normally require some type of event preprocessing for data normalization, validation and transformation.

An "event object" is a computer representation of an actual event in the real world or in any system. An event object carries data or properties. But the term “event” is actually more often used than “event object”, as a simpler, confusing, but yet improper replacement. However, we can consider the terms “event” and “event object” to be somewhat equivalent when talking about CEP topics.

Complex Event Processing concepts

Complex Event Processing (CEP) is computing that performs operations on events. Common event processing operations include reading, creating, transforming, and deleting events (ETZION & NIBLETT, 2011). It can also be considered as a method of analyzing real-time streams of data about things that have happened and deriving a conclusion from them (LUCKHAM, 2011). It typically processes input events order to produce a smaller set of output events.

It also is a distributed technology that creates actionable, situational knowledge from distributed message-based systems, databases and applications in real time or near real time. It can provide an organization with the capability to define, manage and sometimes predict events, situations, exceptional conditions, opportunities and threats in complex, heterogeneous networks. (BASS, 2007) CEP processes many events happening across all the layers of an organization or system, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.

CEP also refers to state processing, which is the change of state occurring when a defined threshold is exceeded. It requires event monitoring, event reporting, event recording, and event filtering. Any event may be observed as a change of state with any physical or logical, or otherwise discriminated condition, of and in a technical or economical system. Each state change usually comes with an attached time stamp defining the order of occurrence, and a topology mark that can define the location of the occurrence.


Fiware-Cepheus implementation

Fiware-Cepheus relies on on the Esper (by EsperTech) java library. The latter is available for both Java and .Net. It features a dedicated CEP rule (EPL) language. The Java version has been chosen for the FIWARE project.

It is an open-source software under the GNU General Public License GPL.

Esper (by EsperTech) CEP library is mainly used for:

  • Complex computations: applications that detect event correlation, filter events, merge event streams, trigger specific actions when particular conditions are satisfied, etc...
  • High throughput: applications that process large volumes of messages (between 1,000 to 100k messages per second)
  • Low latency: applications that react in real-time to conditions that occur

Native supported event formats are plain Java objects, XML, and map objects. Map object is the one implemented in Fiware-Cepheus.

Esper (by EsperTech) provides an dedicated Event Processing Language (EPL) which allows rules definition. EPL is an SQL-like language, but is specifically optimized for dealing with high frequency event data.

The EPL language allows grouping, aggregating, sorting, filtering, merging, splitting and duplicating event streams, defining time and length sliding windows, matching event pattern, and so on...

Some important concepts drive the CEP Engine component structure:

  • The event types describe the internal structure of incoming events
  • The rules define the way event are processed. Rule triggering occurs when event data match the rule criteria.
  • Sliding windows store event data in a First-In First-Out (FIFO) manner. They can be sized in length or duration.

EPL statement structure

A typical statement looks like the following. The brackets indicate optional features.

[INSERT into insert_into_def] SELECT select_list FROM stream_def [AS name] [, stream_def [AS name]] [,...] [WHERE search_conditions] [GROUP BY grouping_expression_list] [HAVING grouping_search_conditions] [OUTPUT output_specification] [ORDER BY order_by_expression_list]

Examples of rules statement:

  • Simple fields "select" from a single event source:
INSERT INTO StoredTaxiPresenceEventType 
SELECT phoneNumber,status 
FROM PresenceEvent 
WHERE userType='Drive'
  • Merge event sources:
INSERT INTO FraudEventType

SELECT fraud.accountNumber AS accntNum, 
       fraud.warning AS warn, 
       withdraw.amount AS amount,
       MAX(fraud.timestamp, withdraw.timestamp) AS timestamp, 
       'withdrawlFraud' AS desc

FROM FraudWarningEvent.win:time(30 min) AS fraud, 
     WithdrawalEvent.win:time(30 sec) AS withdraw

WHERE fraud.accountNumber = withdraw.accountNumber

NGSI

[1] Ref. NGSI Context Management - Open Mobile Alliance V1.0 May 29, 2012

NGSI 9/10 information model for FI-WARE

FI-WARE NGSI Open RESTful API Specification (PRELIMINARY)

Main Interactions

  • The NGSI 9-10 Publish-Subscribe component registers the Data Consolidation GE to the Configuration Management GE by calling its registerContext method.
  • The NGSI 9-10 Publish-Subscribe component sends events to the IoT Broker by calling its updateContext NGSI method.
  • When needed, the IoT broker can call the queryContext method to synchronously retrieve context data from the Data Consolidation NGSI Publish-Subscribe component.
  • The Protocol Adapter GE registers itself to the Data Consolidation GE by calling its registerContext method.
  • The Protocol Adapter GE sends events towards the Data Consolidation GE by calling its updateContext method.
Data Consolidation GE main interactions with surrounding GEs

Basic design principles

  • The Complex Event Processing API was designed with the main CEP concepts in mind: CEP component engine state, CEP rules management, event types management, and configuration of the actions that are executed on rule triggering.
  • The API was designed in a resource-oriented manner.
  • NGSI interface is also part of the API and it was designed in a restful manner. The Data Consolidation GE has to keep technical consistency on its whole API.

Re-utilised Technologies/Specifications

The Edge Data Consolidation GE is based on RESTful Design Principles. The technologies and specifications used in this GE are:

  • RESTful web services
  • XML and JSON data serialization formats.

Other technology which is re-used

Detailed Open Specifications

Following is a list of Open Specifications linked to this Generic Enabler. Specifications labeled as "PRELIMINARY" are considered stable but subject to minor changes derived from lessons learned during last interactions of the development of a first reference implementation planned for the current Major Release of FI-WARE. Specifications labeled as "DRAFT" are planned for future Major Releases of FI-WARE but they are provided for the sake of future users

Open Restful API Specifications(DRAFT)

Applicable only to the Esper4FastData configuration:

Applicable only to the SOL/CEP configuration:

Applicable to both Esper4FastData and SOL/CEP configurations:

Other Open Specifications

Non Applicable up to now.

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions.

  • Thing. One instance of a physical object, living organism, person or concept interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Examples of physical objects are: building, table, bridge (classes), the Kreml, the tennis table from John’s garden, the Tower bridge (instances). Examples of living organisms are: frog, tree, person (classes), the green frog from the garden, the oak tree in front of this building, Dr. Green.
  • Class of thing. Defines the type of a thing in the style of object-oriented programming: a construct that is used as a blueprint to create instances, defining constituent members which enable class instances to have state and behavior. An example of class of thing is “person”, whereas an instance of a thing is “Dr. Green”, with all the static/dynamically changing properties of this particular person.
  • Group of things. A physical/logical group of things. Examples are all office buildings from Munich, all bridges above the Thames, all screws from a drawer, the planes of Lufthansa currently above Germany, all cars inside a traffic jam driven by a female driver, the trees from the Amazonian forest, the squids from the Mediterranean see the mushrooms from Alice’s garden, all the fish from the aquarium of John., the soccer team of Manchester, the colleagues from OrangeLabs currently working abroad (from the perspective of France), all patients above 60 years of age of Dr. Green, the traffic jams from Budapest, the wheat crop in France in the year 2011, the Research and Development Department of the company Telefónica.
  • Device. Hardware entity, component or system that may be in a relationship with a thing or a group of things called association. A device has the means either to measure properties of a thing/group of things and convert it to an analog or digital signal that can be read by a program or user or to potentially influence the properties of a thing/group of things or both to measure/influence. In case when it can only measure the properties, then we call it a sensor. In case when it can potentially influence the properties of a thing, we call it an actuator. Sensors and actuators may be elementary or composed from a set of elementary sensors/actuators. The simplest elementary sensor is a piece of hardware measuring a simple quantity, such as speed of the wind, and displaying it in a mechanical fashion. Sensors may be the combination of software and hardware components, such as a vehicle on-board unit, that consists of simple sensors measuring various quantities such as the speed of the car, fuel consumption etc and a wireless module transmitting the measurement result to an application platform. The simplest elementary actuator is a light switch. We have emphasized potentially because as a result of the light switch being switched on it is not sure that the effect will be that light bulb goes on: only an associated sensor will be able to determine whether it went on or not. Other examples of devices are smart meters, mobile POS devices. More sophisticated devices may have a unique identifier, embedded computing capabilities and local data storage utilities, as well as embedded communication capabilities over short and/or long distances to communicate with IoT backend. Simple devices may not have all these capabilities and they can for instance only disclose the measurement results via mechanical means. In this latter case the further disclosure of the measurement result towards IoT backend requires another kind of device, called IoT Gateway.
  • Association. It is a physical/logical relationship between a device and a thing or a device and a group of things or a group of devices and a group of things from the perspective of a FI-WARE instance, an application within a usage area project or other stakeholder. A device is associated with a thing if it can sense or (potentially) influence at least one property of the thing, property capturing an aspect of the thing interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Devices are associated with things in fully dynamic or mainly static or fully static manner. The association may have several different embodiments: physical attachment, physical embedding, physical neighborhood, logical relation etc. Physical attachment means that the device is physically attached to the thing in order to monitor and interact with it, enabling the thing to be connected to the Internet. An example is the on-board device installed inside the vehicle to allow sending sensor data from the car to the FI-WARE instances. Physical embedding means that the device is deeply built inside the thing or almost part of the thing. An example of physical embedding is the GPS sensor inside a mobile phone. Physical neighborhood means that the device is in the physical neighborhood of the thing, but not in physical contact with it. An example of physical neighborhood is the association of the mobile phone of a subscriber who is caught in a traffic jam with the traffic jam itself: the mobile phone is the device, the traffic jam is the thing and the association means that the mobile phone is in the physical neighborhood of the area inside which the traffic jam is constrained (e.g. within 100 m from the region where the average speed of cars is below 5 km/h). Logical association means that there is a relationship between the device and the thing which is neither fully physical in the sense of attachment or embedding, nor fully physical proximity related. An example of logical association is between the car and the garage door opener of the garage where the car usually parks during the night.
  • IoT gateway. A device that additionally to or instead of sensing/actuating provides inter-networking and protocol conversion functionalities between devices and IoT backend potentially in any combination of these hosts a number of features of one or several Generic Enablers of the IoT Service Enablement. It is usually located at proximity of the devices to be connected. An example of an IoT gateway is a home gateway that may represent an aggregation point for all the sensors/actuators inside a smart home. The IoT gateway will support all the IoT backend features, taking into consideration the local constraints of devices such as the available computing, power, storage and energy consumption. The level of functional split between the IoT backend and the IoT gateway will also depend on the available resources on the IoT gateway, the cost and quality of connectivity and the desired level for the distribution of intelligence and service abstraction.
  • IoT resource. Computational elements (software) that provide the technical means to perform sensing and/or actuation on the device. There may be one-to-one or one-to-many relationship between a device and its IoT resource. Actuation capabilities exposed by the IoT resource may comprise configuration of the management/application features of the device, such as connectivity, access control, information, while sensing may comprise the gathering of faults, performance metrics, accounting/administration data from the device, as well as application data about the properties of the thing with which the device is associated. The resource is usually hosted on the device.
  • Management service. It is the feature of the IoT resource providing programmatic access to readable and/or writable data belonging to the functioning of the device, comprising a subset of the FCAPS categories, that is, configuration management, fault management, accounting /access management/administration, performance management/provisioning and security management. The extent of the subset depends on the usage area. Example of configuration management data is the IP endpoint of the IoT backend instance to which the device communicates. Example of fault management is an error given as a result of the overheating of the device because of extensive exposure to the sun. Example of accounting/administration is setting the link quota for applications using data from a certain sensor. Example of security management is the provisioning of a list of device ID-s of peer devices with which the device may directly communicate.
  • Application service. It is the feature of the IoT resource providing programmatic access to readable or writable data in connection with the thing which is associated with the device hosting the resource. The application service exchanges application data with another device (including IoT gateway) and/or the IoT backend. Measured sensory data are example of application data flowing from devices to sensors. Setting the steering angle of a security camera or sending a “start wetting” command to the irrigation system are examples of application data flowing from the application towards the sensor.
  • Event. An event can be defined as an activity that happens, occurs in a device, gateway, IoT backend or is created by a software component inside the IoT service enablement. The digital representation of this activity in a device, IoT resource, IoT gateway or IoT backend instance, or more generally, in a FI-WARE instance, is also called an event. Events may be simple and complex. A simple event is an event that is not an abstraction or composition of other events. An example of a simple event is that “smart meter X got broken”. A complex event is an abstraction of other events called member events. For example a stock market crash or a cellular network blackout is an abstraction denoting many thousand of member events. In many cases a complex event references the set of its members, the implication being that the event contains a reference. For example, the cellular network blackout may be caused by a power failure of the air conditioning in the operator’s data center. In other cases such a reference may not exist. For example there is no accepted agreement as to which events are members of a stock market crash complex event. Complex events may be created of simple events or other complex events by an IoT resource or the IoT backend instance.
  • IoT Backend. Provides management functionalities for the devices and IoT domain-specific support for the applications. Integrant component of FI-WARE instances.
  • IoT application. An application that uses the application programming interface of the IoT service enablement component. May have parts running on one/set of devices, one/set of IoT gateways and one/set of IoT backends.
  • Virtual thing. It is the digital representation of a thing inside the IoT service enablement component. Consists of a set of properties which are interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Classes of things have the same set of properties, only the value of the properties change from instance to instance.
Personal tools
Create a book