Context Management Architecture
From FIWARE Forge Wiki
FIWARE enables smarter, more customized/personalized and context-aware applications and services by the means of a set of Generic Enablers (GEs) able to gather, publish, exchange, process and analyze massive data in a fast and efficient way.
Nowadays, several well-known free Internet services are based on business models that exploit massive data provided by end users. This data is exploited in advertising or offered to 3rd parties so that they can build innovative applications. Twitter, Facebook, Amazon, Google and many others are examples of this.
The Data/Context Management FIWARE chapter aims at providing outperforming and platform-like GEs that will ease development and the provisioning of innovative Applications that require management, processing and exploitation of context information as well as data streams in real-time and at massive scale. Combined with GEs coming from the Applications and Services Delivery Framework Chapters, application providers will be able to build innovative business models such as those of the companies mentioned above and beyond.
The open data trend is also covered through the adding of the Open Data GE, which allows to catalogue and store open datasets, while it is a perfect match to help in the publication of open datasources based on context information, which enable the publication of real time open data.
FIWARE Data/Context Management GEs will enable to:
- Generate, subscribe for being notified about and query for context information coming from different sources.
- Model changes in context as events that can be processed to detect complex situations that will lead to generation of actions or the generation of new context information (therefore, leading to changes in context also treatable as events).
- Processing large amounts of context information in an aggregated way, using BigData Map&Reduce techniques, in order to generate new knowledge, and to interact with the store to support off the self bundles of data, algorithms and infastructure. Use context data and social networks data to perform analysis.
- Process data streams (particularly, multimedia video streams) coming from different sources in order to generate new data streams as well as context information that can be further exploited.
- Manage some context information, such as location information, presence, user or terminal profile, etc., in a standard way.
- Manage and publish open data, in particular as context data in real time.
- Use existing data and media to enrich applications.
A cornerstone concept within this chapter is the structural definition of Data Elements enclosing its "Data Type", a number of "Data Element attributes" (which enclose the following: Name, Type, Value) and, optionally, a set of "Metadata Elements" (which have also in turn Data-like attributes: Name, Type, Value). However, this precise definition remains unbound to any specific type of representation and enables the usage of "Data Element" structures to represent "Context Elements" and "Events".
This page is for Release 4 of FIWARE. If you need to see the previous architecture for components of Release 3, please visit Data/Context Management Architecture R3
The following diagram shows the main components (Generic Enablers) that comprise the second release of FIWARE Data/Context chapter architecture.
Guiding Design Principles: Data, Context, Event and Event Object Concepts in FIWARE
Data in FIWARE refers to information that is produced, generated, collected or observed that may be relevant for processing, carrying out further analysis and knowledge extraction. A cornerstone concept in FIWARE is that data elements are not bound to a specific format representation.
The structure associated to a data element is represented in Figure 2.
Data has associated a data type and a value. FIWARE will support a set of built-in basic data types similar to those existing in most programming languages.
A data element refers to data whose value is defined as consisting of a sequence of one or more <name, type, value> triplets referred as data element attributes, where the type and value of each attribute is either mapped to a basic data type and a basic data value or mapped to the data type and value of another data element. Note that each data element has an associated data type in this formalism. This data type determines what concrete sequence of attributes characterizes the data element. There may be meta-data (also referred as semantic data) linked to attributes in a data element.
Applications may assign an identifier to data elements in order to store them in a given Data Storage, e.g. a Data Base. Such identifier will not be considered part of the structure of the data element and the way it can be generated is out of the scope of this specification. Note that a given application may decide to use the value of some attribute linked to a data element as its identifier in a given Data Storage but, again, there is no identifier associated to the representation of a data element.
Context in FIWARE is in turn represented through context elements. A context element extends the concept of data element by associating an EntityId and EntityType to it, uniquely identifying the entity (which in turn may map to a group of entities) in the FIWARE system to which the context element information refers. In addition, there may be some attributes as well as meta-data associated to attributes that we may define as mandatory for context elements as compared to data elements.
The structure of a context element is represented in Figure 3.
An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. Events typically lead to creation of some data or context element, which enables applications or event-aware GEs (e.g., the CEP GE) to handle the information described or associated to it. As an example, a sensor device may be measuring the temperature and pressure of a given boiler, sending a context element every five minutes associated to that entity (the boiler) that includes the value of these to attributes (temperature and pressure) or just the one that has changed. The creation and sending of the context element is an event, i.e., something that has occurred (the sensor device has sent new measures). As another example, a mobile handset may export attributes like “Operating System” or “Screen size”. A given application may query for the value of these two attributes in order to adapt the content to be delivered to the device. As a result, the mobile handset creates and replies a context element back to the application. This response may be considered as well an event, i.e., something that has occurred (the mobile handset has replied to a request issued by an application).
Events get visible in the system by updates in Data/context elements and , therefore, it is common to refer to data/context elements related to events simply as "events". For convenience, we also may use the terms “data event” and “context event”. A “data event” refers to an event leading to creation of a data element, while a “context event” refers to an event leading to creation of a context element.
The word event object is used to mean a programming entity that represents such an occurrence (event) in a computing system [EPIA]. Events are represented as event objects within computing systems to distinguish them from other types of objects and to perform operations on them, also known as event processing.
In FIWARE, event objects are created internally to some GEs like the Complex Event Processing GE or the Context Broker GE. These event objects are defined as a data element (or a context element) representing an event to which a number of standard event object properties (similar to a header) are associated internally. These standard event object properties support certain event processing functions. The concrete set of standard event object properties in FIWARE is still to be defined but we may anticipate that one of these properties would be the time at which the event object is detected by the GE (arrives to the GE). This will, for example, allow supporting functions that can operate on events that exceed certain age in the system. Tools will be provided enabling applications or admin users to assign values to those event object properties based on values of data element attributes (e.g., source of the event or actual capture time). An event object may wrap different characteristics of the data element (i.e., DataType) or the context element (i.e., EntityId and EntityType).
Inter-dependencies and Interaction between GEs
In a first stage, the functionalities set provided by each one of the GEs within the Data/Context architecture are considered meaningful standalone and as such they are delivered to other chapters and third parties exploiting FIWARE. However, some tasks have been already started to identify scenarios involving smart combinations of Data/Context GEs into GE packages thus providing synergic features to other chapters and customer applications beyond FIWARE framework. In Figure 4, the most prominent interactions among GEs are depicted. They all are based on the NGSI APIs of FIWARE, and take the Context Broker implementations as core of the integration.
The most important NGSI integrations in FIWARE are:
- Context Broker - Complex Event Processing. The CEP implements specific NGSI adaptations of the REST connectors to allow the reception and generation of notifications of context events. This integration is only based on ONCHANGE notifications. This integration has been implemented and tested as part of FIWARE live demo.
- Context Broker - Big Data Analysis. An adaptor has been implemented to automatically store all the context notifications related to a context entity into the Big Data storage. This data can be used later for Map&Reduce analysis from raw files or loaded into external tables (Hive).
- Context Broker - Open Data. The Open Data GE can catalog and show context information, defining dataset resources as context queries. Moreover, a connector can deliver historical context information as a tabular dataset, feeding it continuously.
- Stream Oriented (Kurento) . The Stream Oriented GE filters and analyzes media content, among other possibilities. The applications deployed in this GE receive real time information about the media analized. Through Java componets, it is possible to push NGSI context notifications to the Context Broker.
- IoT Broker (IoT Chapter). Through this GE the Data chapter Context Broker can receive notifications about context information generated and published by IoT devices. This integration is fully implemented.
And last but not least, other applications and components can use NGSI as a way to communicate context information. For instance, it is posible to send/receive context information from apps running in smartphones or other devices (e.g. Raspberry Pi). There are already open source components that help to create context providers and producers easily.
Architecture description of GEs
Interfaces to other Chapters
Gathering, processing and exchanging information is subject to privacy concerns. Therefore, Data/Context GEs are expected to use Security APIs and interfaces providing Identity management, data authentication/encryption and confidentiality scope.
Internet of Things
In a first approach, several Data/Context GEs will be exploited by the IoT chapter, providing a high differentiation and thus added-value of the FIWARE core platform for IoT businesses. For instance, the Data/Chapter will provide a NGSI-capable Context Broker engine coping with the management of notifications from/to users, applications and GEs at the IoT chapter. Additionally, other Data/Context GEs such as the CEP, enabling the advanced composition of events, or the BigData analysis are expected to be highly useful processing context information generated by GEs in the IoT chapter.
Secondly, the Data/Context framework knowledge framework may exploit metadata associated to context element updates/notifications coming from GEs in the IoT chapter, thus providing added-value to applications.
Applications and Services Ecosystem and Delivery Framework
Several GEs will be offered "as a Service" as cloud services with their own policies and business models (i.e "BigData as a Service"). Therefore, a set of synergic features resulting from the combination of Data/Context GEs and GEs associated to the FIWARE business framework. For instance, it is possible to deliver private datasets from the Open Data GE through the Store, so that their access can be acquired before using them.
Data/Context assets deployment and operations will take benefit of the dynamic resources allocation and monitoring functions supported in the Cloud chapter. In particular, recipes for automated deployment of several GEs in the Data/Context Management chapter are going to be provided (see PaaS and SDC GEs in the Cloud chapter).
Interface to Networks and Devices
Some relevant contextual and metadata information relevant for third party organizations using FIWARE such as availability, reachability and location of users, applications resources and devices can be actually provided by I2ND platform and exploited by Data/Context Management GEs. Moreover, it is possible in FIWARE to have applications and GEs interacting with Robots through the interface between NGSI and ROS (Robots Operating System).