We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.IoT.Edge.DataConsolidation - FIWARE Forge Wiki

FIWARE.OpenSpecification.IoT.Edge.DataConsolidation

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.ArchitectureDescription.IoT.Edge.DataConsolidation
Chapter Internet of Things,
Catalogue-Link to Implementation Cepheus
Owner Orange, Christophe Azemar

Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.

Copyright

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

The IoT environments adressed by this GE comprise large numbers of networked sensors generating even larger numbers of events to be processed. The GE addresses the need for filtering, merging, aggregating and consolidating real-time data from all these highly heterogeneous sources. By contrats to a similar backend enabler in FIWARE, IDEC operates close to the data sources, for applications that have latency requirements that a return trip to a backend enabler could not meet. Many different such applications, especially control-oriented applciations, may wish to receive such latency-critical pre-precessed and aggregated data according to their specific needs, yet this can be achieved in a generic way thanks to this Complex Event Processing technology (CEP)enabler.

Applications that require Complex Event Processing may be found in supply chain management, smart manufacturing, smart cities, smart buildings, home automatio, to name but a few. These applications have in common the requirement to process events (or messages) in real-time, or near real-time. Key considerations for these types of applications are throughput, latency and the complexity of the logic required. The CEP Engine component of the IDEC GE typically processes input data, in order to generate a smaller set of output data, which avoids upper level software overloading and network flooding.

Events are published by Event Producers and subscribed to by Event Consumers. One can only subscribe to events that are allowed by its granted access rights.

The Edge Data Consolidation GE handles data streams from IoT devices that cannot continuously be online, for various reasons. Among them are possible network disruptions and power resource constraints. The latter typically requires optimizing and limiting the communications with the IoT resources. For this purpose, local storage is desirable, in order to cache the last available processed data and contexts.

The natural surrounding GEs for Data Handling are the IoT Broker GE, the Orion Broker GE and IoT Agents.

Although it can be operated standalone, the Data Handling GE typically receives events from IoT Agents, and propagates the processed data towards the brokers. Prior to propagating any data, the Data Handling GE must be registered to the Configuration Management GE, in order to be identified as a context producer.

It should be noted that Publish-Subscribe and CEP features are also available in other GEs. This should not be seen as useless redundancy: some features are deliberately distributed at different level of granularity, in order to provide reliability to the whole architecture, and also to appropriately dispatch the load at different levels.

The Edge Data Consolidation GE is provided by the Fiware-Cepheus project (previously called EspR4FastData) and targets IoT gateways.

Main components

NGSI-9/10 Publish-Subscribe component

This component is the core implementation of the Data Handling GE NGSI functionality. It allows the Generic Enabler to communicate with the outside world and the other GEs, in an ingoing and outgoing way, and also features publish-subscribe functions. It implements the necessary subset of NGSI-9 and NGSI-10 methods for the purpose.

This organization enables the creation of a Publish/Subscribe network of NGSI compliant GEs, that expose their services to each others.

It should be noted that NGSI compliance does not imply the implementation of the full set of NGSI methods.

In the Fiware-Cepheus project, this component is named Cepheus-Broker.

CEP Engine component

This is another core functionality of the Data Handling GE.

The role of the CEP Engine component is to process events in real time, which mainly involves filtering, aggregating and merging data. This processing is based on a set of rules that can be defined by the CEP administrator.

The CEP Engine component can map NGSI data to its internal event representation, process the event and then emit back new NGSI data.

In the Fiware-Cepheus project, this component is named Cepheus-CEP.

Basic concepts

Event concepts

Similar to many topics in science and engineering, the term "event" has different meanings based on who is observing the event and the context of the observation. Generally speaking, an event is something notable that happens in the real world or in a system. But in the computer world, an event is understood as a computer representation of such an occurrence, usually for the purpose of computer processing. Events can contain data, are immutable, but more than one event may record the same activity.

Before the actual event processing occurs, optional event preprocessing can be implemented: it is often referred to as data normalization, validation, prefiltering and basic feature extraction. When required, event preprocessing is often the first step in a distributed, heterogeneous CEP solution. Heterogeneous, distributed event processing applications normally require some type of event preprocessing for data normalization, validation and transformation.

An "event object" is a computer representation of an actual event in the real world or in any system. An event object carries data or properties. But the term “event” is actually more often used than “event object”, as a simpler, confusing, but yet improper replacement. However, we can consider the terms “event” and “event object” to be somewhat equivalent when talking about CEP topics.

In the FIWARE and NGSI context, an "event" is therefore an update to a Context Entity.

Complex Event Processing concepts

Complex Event Processing (CEP) is computing that performs operations on events. Common event processing operations include reading, creating, transforming, and deleting events (ETZION & NIBLETT, 2011). It can also be considered as a method of analyzing real-time streams of data about things that have happened and deriving a conclusion from them (LUCKHAM, 2011). It typically processes input events order to produce a smaller set of output events.

It also is a distributed technology that creates actionable, situational knowledge from distributed message-based systems, databases and applications in real time or near real time. It can provide an organization with the capability to define, manage and sometimes predict events, situations, exceptional conditions, opportunities and threats in complex, heterogeneous networks. (BASS, 2007) CEP processes many events happening across all the layers of an organization or system, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time.

CEP also refers to state processing, which is the change of state occurring when a defined threshold is exceeded. It requires event monitoring, event reporting, event recording, and event filtering. Any event may be observed as a change of state with any physical or logical, or otherwise discriminated condition, of and in a technical or economical system. Each state change usually comes with an attached time stamp defining the order of occurrence, and a topology mark that can define the location of the occurrence.

Cepheus CEP implementation

The Cepheus-CEP component (from the Fiware-Cepheus project) relies on on the Esper (by EsperTech) java library. It is an open-source software under the GNU General Public License GPL featuring a dedicated CEP rule language (EPL).

The Esper CEP library is used for:

  • Complex computations: applications that detect event correlation, filter events, merge event streams, trigger specific actions when particular conditions are satisfied, etc...
  • High throughput: applications that process large volumes of messages (between 1,000 to 100k messages per second)
  • Low latency: applications that react in real-time to conditions that occur

The Event Processing Language (EPL) allows to define these rules. The EPL language is an SQL-like language specifically optimized for dealing with high frequency event data.

The EPL language allows grouping, aggregating, sorting, filtering, merging, splitting and duplicating event streams, defining time and length sliding windows, matching event pattern, and so on...

Some important concepts drive the CEP component:

  • The event types describe the internal structure of incoming events
  • The rules define the way event are processed. Rule triggering occurs when event data match the rule criteria.
  • Sliding windows store event data in a First-In First-Out (FIFO) manner. They can be sized in length or duration.
  • A listener is executed when a rule is triggered. A listener is always associated to a rule.

EPL statement structure

A typical statement looks like the following. The brackets indicate optional features.

[INSERT into insert_into_def] SELECT select_list FROM stream_def [AS name] [, stream_def [AS name]] [,...] [WHERE search_conditions] [GROUP BY grouping_expression_list] [HAVING grouping_search_conditions] [OUTPUT output_specification] [ORDER BY order_by_expression_list]

For more information on EPL, please read the Esper documentation.

Examples of rules statement:

  • Simple fields "select" from a single event source:
INSERT INTO StoredTaxiPresenceEventType 
SELECT phoneNumber,status 
FROM PresenceEvent 
WHERE userType='Drive'
  • Merge event sources:
INSERT INTO FraudEventType

SELECT fraud.accountNumber AS accntNum, 
       fraud.warning AS warn, 
       withdraw.amount AS amount,
       MAX(fraud.timestamp, withdraw.timestamp) AS timestamp, 
       'withdrawlFraud' AS desc

FROM FraudWarningEvent.win:time(30 min) AS fraud, 
     WithdrawalEvent.win:time(30 sec) AS withdraw

WHERE fraud.accountNumber = withdraw.accountNumber

Basic design principles

The Fiware-Cepheus project is composed of 3 separated components (two applications and one common library):

  • The Cepheus-CEP application handling CEP processing
  • The Cepheus-Broker application handling NGSI pub/sub and request forwarding
  • The NGSI library that contains code for building NGSI v1 client and server applications

These two applications communicate with each other and with other FIWARE GEs using NGSI v1.

More detail on the architecture can be found on the Fiware-Cepheus project

Re-utilised Technologies/Specifications

  • XML and JSON data serialization formats.

Other technologies which are re-used:

Detailed Open Specifications

Following is a list of Open Specifications linked to this Generic Enabler. Specifications labeled as "PRELIMINARY" are considered stable but subject to minor changes derived from lessons learned during last interactions of the development of a first reference implementation planned for the current Major Release of FI-WARE. Specifications labeled as "DRAFT" are planned for future Major Releases of FI-WARE but they are provided for the sake of future users

Open API Specifications

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions.

  • Thing. One instance of a physical object, living organism, person or concept interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Examples of physical objects are: building, table, bridge (classes), the Kreml, the tennis table from John’s garden, the Tower bridge (instances). Examples of living organisms are: frog, tree, person (classes), the green frog from the garden, the oak tree in front of this building, Dr. Green.
  • Class of thing. Defines the type of a thing in the style of object-oriented programming: a construct that is used as a blueprint to create instances, defining constituent members which enable class instances to have state and behavior. An example of class of thing is “person”, whereas an instance of a thing is “Dr. Green”, with all the static/dynamically changing properties of this particular person.
  • Group of things. A physical/logical group of things. Examples are all office buildings from Munich, all bridges above the Thames, all screws from a drawer, the planes of Lufthansa currently above Germany, all cars inside a traffic jam driven by a female driver, the trees from the Amazonian forest, the squids from the Mediterranean see the mushrooms from Alice’s garden, all the fish from the aquarium of John., the soccer team of Manchester, the colleagues from OrangeLabs currently working abroad (from the perspective of France), all patients above 60 years of age of Dr. Green, the traffic jams from Budapest, the wheat crop in France in the year 2011, the Research and Development Department of the company Telefónica.
  • Device. Hardware entity, component or system that may be in a relationship with a thing or a group of things called association. A device has the means either to measure properties of a thing/group of things and convert it to an analog or digital signal that can be read by a program or user or to potentially influence the properties of a thing/group of things or both to measure/influence. In case when it can only measure the properties, then we call it a sensor. In case when it can potentially influence the properties of a thing, we call it an actuator. Sensors and actuators may be elementary or composed from a set of elementary sensors/actuators. The simplest elementary sensor is a piece of hardware measuring a simple quantity, such as speed of the wind, and displaying it in a mechanical fashion. Sensors may be the combination of software and hardware components, such as a vehicle on-board unit, that consists of simple sensors measuring various quantities such as the speed of the car, fuel consumption etc and a wireless module transmitting the measurement result to an application platform. The simplest elementary actuator is a light switch. We have emphasized potentially because as a result of the light switch being switched on it is not sure that the effect will be that light bulb goes on: only an associated sensor will be able to determine whether it went on or not. Other examples of devices are smart meters, mobile POS devices. More sophisticated devices may have a unique identifier, embedded computing capabilities and local data storage utilities, as well as embedded communication capabilities over short and/or long distances to communicate with IoT backend. Simple devices may not have all these capabilities and they can for instance only disclose the measurement results via mechanical means. In this latter case the further disclosure of the measurement result towards IoT backend requires another kind of device, called IoT Gateway.
  • Association. It is a physical/logical relationship between a device and a thing or a device and a group of things or a group of devices and a group of things from the perspective of a FI-WARE instance, an application within a usage area project or other stakeholder. A device is associated with a thing if it can sense or (potentially) influence at least one property of the thing, property capturing an aspect of the thing interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Devices are associated with things in fully dynamic or mainly static or fully static manner. The association may have several different embodiments: physical attachment, physical embedding, physical neighborhood, logical relation etc. Physical attachment means that the device is physically attached to the thing in order to monitor and interact with it, enabling the thing to be connected to the Internet. An example is the on-board device installed inside the vehicle to allow sending sensor data from the car to the FI-WARE instances. Physical embedding means that the device is deeply built inside the thing or almost part of the thing. An example of physical embedding is the GPS sensor inside a mobile phone. Physical neighborhood means that the device is in the physical neighborhood of the thing, but not in physical contact with it. An example of physical neighborhood is the association of the mobile phone of a subscriber who is caught in a traffic jam with the traffic jam itself: the mobile phone is the device, the traffic jam is the thing and the association means that the mobile phone is in the physical neighborhood of the area inside which the traffic jam is constrained (e.g. within 100 m from the region where the average speed of cars is below 5 km/h). Logical association means that there is a relationship between the device and the thing which is neither fully physical in the sense of attachment or embedding, nor fully physical proximity related. An example of logical association is between the car and the garage door opener of the garage where the car usually parks during the night.
  • IoT gateway. A device that additionally to or instead of sensing/actuating provides inter-networking and protocol conversion functionalities between devices and IoT backend potentially in any combination of these hosts a number of features of one or several Generic Enablers of the IoT Service Enablement. It is usually located at proximity of the devices to be connected. An example of an IoT gateway is a home gateway that may represent an aggregation point for all the sensors/actuators inside a smart home. The IoT gateway will support all the IoT backend features, taking into consideration the local constraints of devices such as the available computing, power, storage and energy consumption. The level of functional split between the IoT backend and the IoT gateway will also depend on the available resources on the IoT gateway, the cost and quality of connectivity and the desired level for the distribution of intelligence and service abstraction.
  • IoT resource. Computational elements (software) that provide the technical means to perform sensing and/or actuation on the device. There may be one-to-one or one-to-many relationship between a device and its IoT resource. Actuation capabilities exposed by the IoT resource may comprise configuration of the management/application features of the device, such as connectivity, access control, information, while sensing may comprise the gathering of faults, performance metrics, accounting/administration data from the device, as well as application data about the properties of the thing with which the device is associated. The resource is usually hosted on the device.
  • Management service. It is the feature of the IoT resource providing programmatic access to readable and/or writable data belonging to the functioning of the device, comprising a subset of the FCAPS categories, that is, configuration management, fault management, accounting /access management/administration, performance management/provisioning and security management. The extent of the subset depends on the usage area. Example of configuration management data is the IP endpoint of the IoT backend instance to which the device communicates. Example of fault management is an error given as a result of the overheating of the device because of extensive exposure to the sun. Example of accounting/administration is setting the link quota for applications using data from a certain sensor. Example of security management is the provisioning of a list of device ID-s of peer devices with which the device may directly communicate.
  • Application service. It is the feature of the IoT resource providing programmatic access to readable or writable data in connection with the thing which is associated with the device hosting the resource. The application service exchanges application data with another device (including IoT gateway) and/or the IoT backend. Measured sensory data are example of application data flowing from devices to sensors. Setting the steering angle of a security camera or sending a “start wetting” command to the irrigation system are examples of application data flowing from the application towards the sensor.
  • Event. An event can be defined as an activity that happens, occurs in a device, gateway, IoT backend or is created by a software component inside the IoT service enablement. The digital representation of this activity in a device, IoT resource, IoT gateway or IoT backend instance, or more generally, in a FI-WARE instance, is also called an event. Events may be simple and complex. A simple event is an event that is not an abstraction or composition of other events. An example of a simple event is that “smart meter X got broken”. A complex event is an abstraction of other events called member events. For example a stock market crash or a cellular network blackout is an abstraction denoting many thousand of member events. In many cases a complex event references the set of its members, the implication being that the event contains a reference. For example, the cellular network blackout may be caused by a power failure of the air conditioning in the operator’s data center. In other cases such a reference may not exist. For example there is no accepted agreement as to which events are members of a stock market crash complex event. Complex events may be created of simple events or other complex events by an IoT resource or the IoT backend instance.
  • IoT Backend. Provides management functionalities for the devices and IoT domain-specific support for the applications. Integrant component of FI-WARE instances.
  • IoT application. An application that uses the application programming interface of the IoT service enablement component. May have parts running on one/set of devices, one/set of IoT gateways and one/set of IoT backends.
  • Virtual thing. It is the digital representation of a thing inside the IoT service enablement component. Consists of a set of properties which are interesting to a person or an application from the perspective of a FI-WARE instance, one particular usage area or to several usage areas. Classes of things have the same set of properties, only the value of the properties change from instance to instance.
Personal tools
Create a book