We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Data.CEP - FIWARE Forge Wiki

FIWARE.OpenSpecification.Data.CEP

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Data.CEP
Chapter Data/Context Management,
Catalogue-Link to Implementation Complex Event Processing
Owner IBM Haifa Research Lab, Tali Yatzkar-Haham


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


Copyright

  • Copyright © 2012-2016 by IBM

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

Introduction to the CEP GE

The Complex Event Processing (CEP) GE is intended to support the development, deployment, and maintenance of Complex Event Processing (CEP) applications.

CEP analyses event data in real-time, generates immediate insight and enables instant response to changing conditions. Some functional requirements this technology addresses include event-based routing, observation, monitoring and event correlation. The technology and implementations of CEP provide means to expressively and flexibly define and maintain the event processing logic of the application, and in runtime it is designed to meet all the functional and non-functional requirements without taking a toll on the application performance, removing one issue from the application developer’s and system managers concerns.


The CEP and other entities

Entities connected to the CEP GE (application entities or some other GEs like the Context Broker GE) can play two different roles: the role of Event Producer or the role of Event Consumers. Note that nothing precludes that a given entity plays both roles. Event Producers are the source of events for event processing. Following are some examples of event producers:

  • FIWARE GEs, such as the Context Broker GE that can send context events to the CEP in NGSI format through the CEP REST producer.
  • Sensors reporting on a new measurement. Events generated by such sensors can be consumed directly by the CEP GE. Another alternative is that the sensor events are gathered and processed through the IoT GEs, which publish context events to the Context Broker GE, having the CEP acting as a context consumer of the Context Broker GE as described above.
  • External applications reporting events on user activities such as "user placed of new order", and on operation activities such as "delivery has been shipped".

Event Producers can provide events in two modes:

  • "Push" mode: The Event Producers push events into CEP by means of invoking a standard operation that CEP exports.
  • ”Pull” mode: The Event Producer exports a standard operation that CEP can invoke to retrieve events.

Event Consumers are the destination point of events. Following are some examples of event consumers:

  • FIWARE GEs, such as the Context Broker GE which can connect as an event consumer to the CEP, get the CEP output events as context update events in NGSI format, and forward the context events it consumes to all interested applications based on a subscription model.
  • Dashboard: a type of event consumer that displays alarms defined when certain conditions hold on events related to some user community or produced by a number of devices.
  • Handling process: a type of event consumer that consumes meaningful events (such as opportunities or threats) and performs a concrete action.


Basic Concepts

CEP implements event processing functions based on the design and execution of Event Processing Networks (EPN). An EPN is made up of processing nodes called Event Processing Agents (EPAs) as described in the book “Event Processing in Action” [EPIA]. The network describes the flow of events originating at event producers and flowing through various event processing agents to eventually reach event consumers. See the figure below for an illustration. Here we see that events from Producer 1 are processed by Agent 1. Events derived by Agent 1 are of interest to Consumer 1 but are also processed by Agent 3 together with events derived by Agent 2. Note that the intermediary processing between producers and consumers in every installation is made up of several functions and often the same function is applied to different events for different purposes at different stages of the processing. The EPN approach allows dealing with this in an efficient manner, because a given agent may receive events from different sources. At runtime, this approach also allows for a flexible allocation of agents in physical computing nodes as the entire event processing application can be executed as a single runtime artifact, such as Agent 1 and Agent 2 in Node 1 in the figure below, or as multiple runtime artifacts according to the individual agents that make up the network, such as Agent 1 and Agent 3 running within different nodes. Thus scalability, performance and optimization requirements may be addressed by design.


Illustration of an Event Processing Network made of event producers, agents and event consumers


The reasons for running pieces of the network in different nodes or environments vary, for example:

  • Distributing the processing power
  • Distributing for geographical reasons – process as close to the source as possible for lower networking
  • Optimized and specialized processors that deal with specific event processing logic

Another benefit in representing event processing applications as networks is that entire networks can be nested as agents in other networks allowing for reuse and composition of existing event processing applications.

The event processing agents and their assembly into a network is where most of the functions of CEP are implemented. The behavior of an event-processing agent is specified using a rule-oriented language that is inspired by the ECA (Event-Condition-Action) concept and may better be described as Pattern-Condition-Action. Rules in this language will consist of three parts:

  • A pattern detection that makes a rule of relevance. A pattern is associated with event processing context
  • A set of conditions (logical tests) formulated on events
  • A set of actions to be carried out when all the established conditions are satisfied

Following is an indication of the capabilities to be supported in each part of the rule language.

Pattern Detection

In the pattern detection part the application developer may program patterns over selected events within an event processing context (such as a time window or segmentation) and only if the pattern is matched the rule is of relevance and according to (optional) additional conditions the action part is executed. Examples for such patterns are:

  • Sequence, meaning events need to occur in a specified order for the pattern to be matched. E.g., follow customer transactions, and detect if the same customer bough and later sold the same stock within the time window.
  • Aggregate, compute some aggregation functions on a set of incoming events. E.g., compute the percentage of the sensors events that arrived with a fail status out of all the sensors events arrived in the time window. Alert if the percentage of the failed sensors is higher than 10 percent.
  • Absent, meaning no event holding some condition arrived within the time window for the pattern to match. E.g., alert if within the time window no sensor events arriving from specific source have arrived. This may indicate that the source is down.
  • All, meaning that all the events specified should arrive for the pattern to match. E.g., wait to get status events from all the 4 locations, where each status event arrives with the quantity of reservations. Alert if the total reservations are higher than some threshold.

Event Processing Context

Every pattern is associated with an Event processing context. Event processing context groups event instances so that they can be processed in a related way. It assigns each event instance to one or more processing context partitions. Event processing context can be a temporal processing context, a segmentation processing context, or a composite context that is to say one made up of other processing context specifications.

Temporal processing context defines a time window. If a pattern is associated with a temporal processing context, it will process only the events arriving within this time window. E.g., “within 5 seconds after sensor events with high value”, “A time window bounded by Order-Placed and Order-Delivered events”

Segmentation processing context defined a matching criteria based on the attribute values of the events. E.g., “Sensor’s ID”, “Shipment’s ID”, “Building’s ID”. Events that belong to the same matching criteria (e.g., have the same sensor ID) will be processed together within the same processing context partition.

Conditions

The application developer may add the following kind of conditions in a given rule:

  • Simple conditions, which are established as predicates defined over single events of a certain type
  • Complex conditions, which are established as logical operations on predicates defined over a set of events of a certain type

Actions

As part of the rule definition, the application developer of CEP specifies what should be done when a rule is detected. This can include generation of one or more derived events to be sent to the consumers and actions to be performed by the consumers. These actions definitions include the parameters needed for their execution.

Target Usage

Complex Event Processing (CEP) is the analysis of event patterns in real-time to generate immediate insight and enable instant response to changing conditions. When the need is to respond to a specific event, the Context Broker GE is sufficient. You should consider using the CEP GE when there is a need to detect pattern over the incoming events occurring within some processing context (see the pattern examples in the previous section). Some functional requirements this technology addresses include event-based routing, observation, monitoring and event correlation. The technology and implementations of CEP provide means to expressively and flexibly define and maintain the event processing logic of the application, and in runtime it is designed to meet all the functional and non-functional requirements without taking a toll on the application performance, removing one issue from the application developer’s and system managers concerns.

For the primary user of the real-time processing generic enabler, namely the consumer of the information generated, the Complex Event Processing GE (CEP GE) addresses the user’s concerns of receiving the relevant events at the relevant time with the relevant data in a consumable format (relevant meaning that reacting or making use of the event is meaningful for the consumer/subscriber). The figure below depicts this role through a pseudo API derivedEvent(type,payload) by which, at the very least, an event object is received with the name of the event, derived out of the processing of other events, and its payload.

The designer of the event processing logic is responsible for creating event specifications and definitions (including where to receive them) from the data gathered by the Massive Data Gathering Generic Enabler. The designer should also be able to discover and understand existing event definitions. Therefore FIWARE, in providing an implementation of a Real-time CEP GE, will also provide the tools for the designer. In addition, APIs will be provided to allow generation of event specification and definitions programmatically, by an application or tools. In the figure below these roles are described as Designer and Programs making use of the pseudo API deploy definitions/instructions.

Finally, the CEP GE supports the roles of event system manager and operator, which could be played either by real people or management components. Actors playing these roles are responsible for managing configurations, monitor processing performance, handling problems, and monitoring the system’s health. They make use of the pseudo API configuration/tuning/monitoring for this purpose.

Image: Interactions_with_and_APIs_of_the_Real-time_CEP_Generic_Enabler.jpg
Interactions with and APIs of the Real-time CEP Generic Enabler


Generic Architecture

The CEP architecture is composed from 4 main components:

  1. Build-time web based user interface
  2. Input Adapters
  3. Output Adapters
  4. Runtime Engine

Illustration of those components and their interactions are given in the figure below.


The application definitions, including the EPN, can be written by the application developer using CEP build-time web based user interface, by filling definition forms. The CEP build-time user interface generates a definition file which is sent to the CEP run-time. Alternatively this definition file, in JSON format, can be generated programmatically by any other application. At runtime, CEP receives incoming events through the input adapters or directly through its REST service. CEP processes those incoming events according to the application definitions and sends derived events through the output adapters.


CEP High Level Architecture


CEP semantic layer allows the user to define producers and consumers for event data (see the figure above). Producers produce event data, and consumers consume the event data. The definitions of producers and consumer, which is specified during the application build time are translated into input and output adapters at CEP execution time. The physical entities representing the logical entities of producers and consumers in CEP are adapter instances.


Adapters layer representation


As can be seen in the above figure, an input adapter is defined for each producer, which defines how to pull the data from the source, how to format the data into CEP's object format before delivering it to the engine. The adapter is environment-agnostic but uses the environment-specific connector object, injected into the adapter during its creation, to connect to CEP runtime.

The consumers and their respective output adapters are operated in push mode – each time an event is published by the runtime it is pushed through environment-specific server connectors to the appropriate consumers, represented by their output adapters, which publish the event in the appropriate format to the designated resource.

The server connectors are environment-specific. They hide the implementation of the connectivity layer from the adapters, allowing them to be environment-agnostic.

Adapters design principles

As part of the CEP application design, the user specifies the events producers as sources of event data and the event consumers as sinks for event data. The specification of producers includes the resource from which the adapter pulls the information (whether this resource is a file in a file system). It also includes format settings which allow the adapter to transform the resource-specific information into a CEP event data object. The formatting depends on the kind of resource we are dealing with – for files it can be a tagged file formatter. Likewise, the specification of consumers includes the resource to which the event created by CEP runtime should be published and a formatter describing on how to transform a CEP event data object into resource-specific object.

The design of adapter's layer satisfies the following principles:

  • A producer is a logical entity which holds the specifications of the source of the event data and the format of the event data. The input adapter is the physical entity representing a producer, an entity which actually interacts with the resource and communicates event information to CEP runtime server.
  • A consumer is a logical entity which holds the specifications of the target of the events and the format of the event data. The output adapter is the physical representation of the consumer: it is an entity which is invoked by the CEP runtime when an event instance should be published to the resource.
  • All the input adapters implement a common interface, which is extendable to cover custom input adapter types and allows adding new producers for custom-type resources.
  • All the output adapters implement a common interface, which is extendable to cover custom output adapter types and allows adding new consumers for custom-type resources.
  • A single event instance can have multiple consumers
  • A producer can produce events of different types, a single event instance might serve as input to multiple logical agents within the event processing network, according to the network's specifications
  • Producers operate in pull mode, each input adapter pulling the information from designated resource according to its specifications, each time processing the incremental additions in the resource. However, producers operating in a push mode are planned to be supported as well.
  • Consumers define a list of event types they are interested in, they can also specify a filter condition on each event type – only event instances satisfying this condition will be actually delivered to this consumer.
  • Consumers operate in push mode, each time the CEP runtime publishes an event instance it is pushed to the relevant consumer.
  • The producers and consumers are not directly connected, but the raw event's data supplied by a certain producer can be delivered to a consumer if the consumer specifies this event type in its desirable events list.


Main Interactions

CEP has four main interfaces with its environment as can be seen in the figure above:

1. Input Adapters and REST service for getting incoming events

2. Output Adapters for sending derived events

3. CEP Application definition

4. Administrative REST services


The input and output adapters that are used in a specific application are set according to the Producers and Consumers definitions the user defined as part of the CEP application definition. For example, if the user defined a producer of 'File' type, then a File adapter will be used at run time to read events from a file. If the user defined a consumer of type 'Rest', then at run time, a Rest adapter will activate a Rest service and send it output events. If the user wants to use the CEP Rest service to send input events to the CEP they don't need to define a producer.

Definition of CEP Application

A CEP definition file is created using the CEP build-time web based user interface. Using this UI, the application developer creates the building blocks of the application definitions. This is done by filling up forms without the need to write any code.

The building blocks of a CEP application are:

  • Event type – the events that are expected to be received as input or to be sent as output. An event type definition includes the event name and a list of its attributes.
  • Producers – the event sources and the way CEP gets events from those sources.
  • Consumers – the event consumers and the way they get derived events from CEP.
  • Temporal Contexts – time windows contexts in which event processing agents are active.
  • Segmentation Contexts – semantic contexts that are used to group several events to be used by the event processing agents.
  • Composite Contexts – group together several different contexts.
  • Event Processing Agents – responsible of applying rules on incoming events in specific context as to detect situations and generate derived events.

The UI (see a figure below) provides many functions, including defining a CEP application, validating the CEP definitions and exporting the event processing network definition. The export action creates a JSON format representation of the CEP application. This JSON can be exported either to the engine repository or to a local file (to be later fed to the engine repository).

This JSON file that describes a CEP application, is the input for the CEP engine. It can be generated by the UI, and it can be generated programmatically by any other application and fed to the CEP engine. The format of this definition file is described by a JSON schema for a CEP application while the semantics of the various elements in this schema are described in a user guide


CEP Web based User interface for application definition

Administrative REST services

There are REST services that allow managing the CEP definition repository that holds the CEP application definitions available to the CEP engine instances at run time. These services allow putting a new definition file to the repository, getting a specific definition from the repository, updating a repository definition file or deleting a definition from the repository. In addition, there are REST services that allow controlling the CEP engine instances at run time. These services allow starting and stopping a CEP engine instance, updating CEP engine instance definitions and reading the state of the CEP engine instance (started/stopped and its definition url).

Basic Design Principles

  • The EPN application definition can be done using a user interface without the need to write any code, with intention for visual programming.
  • The CEP application is composed of a network of Event Processing Agents. This allows the agents to run in parallel and to be distributed on several machines.
  • Logical EPN definition which is decoupled from the actual running configuration. The same EPN can run on a single machine or be distributed on several machines.
  • The event producers and event consumers can be distributed among different machines.
  • Event producers and consumers are totally decoupled.
  • Adapter framework that is extensible to allow adding any type of custom adapter for sending or receiving events.
  • The expression language is extensible and functions can be added if needed.

References

[EPIA]

O. Etzion and P. Niblett, Event Processing in Action, Manning Publications, 2010.


Detailed Specifications

API Documentation

Re-utilised Technologies/Specifications

The CEP authoring tool is running on Apache Tomcat web server

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions

  • Data refers to information that is produced, generated, collected or observed that may be relevant for processing, carrying out further analysis and knowledge extraction. Data in FIWARE has associated a data type and avalue. FIWARE will support a set of built-in basic data types similar to those existing in most programming languages. Values linked to basic data types supported in FIWARE are referred as basic data values. As an example, basic data values like ‘2’, ‘7’ or ‘365’ belong to the integer basic data type.
  • A data element refers to data whose value is defined as consisting of a sequence of one or more <name, type, value> triplets referred as data element attributes, where the type and value of each attribute is either mapped to a basic data type and a basic data value or mapped to the data type and value of another data element.
  • Context in FIWARE is represented through context elements. A context element extends the concept of data element by associating an EntityId and EntityType to it, uniquely identifying the entity (which in turn may map to a group of entities) in the FIWARE system to which the context element information refers. In addition, there may be some attributes as well as meta-data associated to attributes that we may define as mandatory for context elements as compared to data elements. Context elements are typically created containing the value of attributes characterizing a given entity at a given moment. As an example, a context element may contain values of some of the attributes “last measured temperature”, “square meters” and “wall color” associated to a room in a building. Note that there might be many different context elements referring to the same entity in a system, each containing the value of a different set of attributes. This allows that different applications handle different context elements for the same entity, each containing only those attributes of that entity relevant to the corresponding application. It will also allow representing updates on set of attributes linked to a given entity: each of these updates can actually take the form of a context element and contain only the value of those attributes that have changed.
  • An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. Events typically lead to creation of some data or context element describing or representing the events, thus allowing them to processed. As an example, a sensor device may be measuring the temperature and pressure of a given boiler, sending a context element every five minutes associated to that entity (the boiler) that includes the value of these to attributes (temperature and pressure). The creation and sending of the context element is an event, i.e., what has occurred. Since the data/context elements that are generated linked to an event are the way events get visible in a computing system, it is common to refer to these data/context elements simply as "events".
  • A data event refers to an event leading to creation of a data element.
  • A context event refers to an event leading to creation of a context element.
  • An event object is used to mean a programming entity that represents an event in a computing system [EPIA] like event-aware GEs. Event objects allow to perform operations on event, also known as event processing. Event objects are defined as a data element (or a context element) representing an event to which a number of standard event object properties (similar to a header) are associated internally. These standard event object properties support certain event processing functions.
Personal tools
Create a book