We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.ArchitectureDescription.IoT.Backend.IoTDiscovery - FIWARE Forge Wiki


From FIWARE Forge Wiki

Revision as of 18:40, 6 September 2016 by Telsaleh (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search



Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.


Data model outline

The IoT-Discovery GE is the part of the Backend tier of the IoT Architecture which is responsible for context source availability management. The underlying data model of this GE is based on the OMA NGSI-9 Context Management Information Model. This model relies on the concept of context entities, which are generic entities whose state is described by the means of values of attributes and associated metadata. In the context of IoT, context entities and context entity attributes can be used to model IoT resources and the variables they measure, respectively. Additionally - and more importantly -arbitrary physical objects (Things) like rooms, people, cars, etc. and their attributes like temperature, geo-location, etc. can be used as well.

Functionality outline

The IoT-Discovery GE is responsible for the context availability registrations from IoT Agents, thus making it the access point for information about entities and their attributes. In particular, the context availability information provided by the IoT-Discovery GE is either forwarded from IoT Agents exposing the FIWARE NGSI-9/10 interfaces. The role of IoT Agents can be played by either the Data Handling GE in IoT Gateways, or the Backend Device Management GE. It is also possible that the context information is provided by other IoT Backend systems.

Information about context source availability is typically forwarded to the FIWARE Context Broker GE so that context information about Things becomes accessible to applications. However, it can be the case that the Context Broker GE may manage context availability information that is not necessarily provided by the IoT-Discovery GE, therefore linked to the Internet of Things, but gathered from other parts of the application.

More precisely, using the FIWARE NGSI-9 interface that the IoT-Discovery GE provides, applications and services will be able to register, discover and subscribe to updates on context availability information, that can be:

  • Information about IoT resources and the variables they measure
  • Information about Things and their attributes
  • Information about Associations between entities, whereby the attributes of a Thing can be derived from attributes of other Things, or from attributes measured by IoT resources.

The IoT-Discovery GE is also specified to optionally different flavors of discovery mechanisms, along with the main mode of discovery. These include (and not limited to):

  • Geographic: where entities can be discovered based on their location.
  • Probabilistic: where entities are discovered similarity in syntax.
  • Semantic: where entities can be semantically-annotated and linked to well-known ontology models that can provide more detailed information about a particular attribute, and increase their interoperability with other information on the Web.

Basic Concepts

The IoT-Discovery GE is based on the OMA NGSI context data model.


The IoT-Discovery GE implements the RESTful FIWARE NGSI-9 binding specification. FIWARE NGSI Context Management specifications are based on the NGSI Context Management specifications defined by OMA (Open Mobile Alliance). They include two RESTful context management interfaces NGSI-9 and NGSI-10 (see Open RESTful API Specification), which also solve some ambiguities in the OMA specification for NGSI and extends them where necessary to realize the FIWARE Vision.

You can visit the FI-WARE NGSI Context Management tutorial to learn the main concepts underlying FIWARE NGSI Context Management specifications.

GE Internal Architecture

The internal architecture of IoT-Discovery consists of two main components, the Registry Manager and the Registry Repository.

Internal architecture for IoT Discovery

Registry Manager

The Registry Manager component consists of a context information registry in which context provider applications can be registered. In addition, components interacting with the Registry Manager can perform discovery operations on that context registration information or subscribe to changes on it.

It detail, the component provides the following functionality:

  • Allow NGSI-9 clients, such as the IoT Broker to discover and subscribe to context information, through the NGSI-9 interface exposed by IoT Discovery.
  • Announce the availability of context information towards the Context Broker GE (or any other component supporting the NGSI-9 interface) through the northbound NGSI-9 interface. The Registry Management component stores the context information in a Registry Repository described in next subsection. This repository is used to answer NGSI-9 request in the exposed interfaces described above.
  • Receive registrations from IoT Gateways and the Thing-level Adapter through the NGSI-9 southbound interface and store this information in the Registry Repository, thus updating the context information registry.

Registry Repository

The Registry Repository stores information on the availability of context information and can be accessed through the Registry Management. When a NGSI-9 client sends a discoverContextAvailabilityRequest operation in order to find out where the desired context information can be found, the Registry Manager returns zero or more ContextRegistration instances that match the client's request.

Results can also include associated context sources if specified in the operation scope field in the request. With this approach, IoT-Discovery can maintain higher-level context information that is not available in the IoT Gateways or Devices. For example, a Gateway might not know the concept of cars, but only maintains a list of sensors and their measurements. The information about which specific sensors provide information about same car could be maintained only by the IoT-Discovery GE.

Additional Concepts

Associations in FIWARE NGSI-9

Associations between entities

IoT-Discovery enables enhanced associations. One of the central features of the FI-WARE IoT Backend is its ability to derive information about arbitrary things (cars, people, houses, etc.) from device level information. The link between these two levels of abstraction is provided by the association concept defined for FI-WARE NGSI. A definition of the concept and its representation in NGSI-9 messages can be found on the NGSI association definition page. An association is an ordered pair consisting of a sensor-level entity (possibly with an attribute name) and a thing-level entity (possibly enhanced with an attribute name). The interpretation of such a pair is that the sensor provides information about the thing.

Optional Features for Discovery

When discovering and subscribing to notification for specific entities, the Operation Scope field in the respective operation i.e. discoverContextAvailability and subscribeContextAvailability can be used to specify a particular type of discovery, which can be:

  • By Association: using the type includeAssociations in the scope field
  • Geographical: using the type FIWARE::Location.
  • Probabilistic: using the type search.

There is also an option for discovering context sources via semantic query methods such as SPARQL. This requires registrations to be available in semantic form. An NGSI-9 message can link to a corresponding semantic description. This can be done by the client by specifying the link to the semantic description the in the registration metadata using the type linked-data.

Main Interactions

IoT-Discovery has a number of interfaces for interacting with applications, users, and other GEs within the FIWARE architecture. For example, the GE communicates with the Context Broker GE via the Northbound Interface and with the IoT Agents on the Southbound Interface. With respect to GEs, the role of IoT Agent can be played by either the Backend Device Management GE, or by the Gateway Data Handling GE. There is also an interface between the IoT Broker GE and the IoT-Discovery GE. In the context of NGSI, IoT Discovery plays the role of the server, and any other actors interacting with it play the role of the client. To illustrate this, in the following diagrams all IoT Agents, GEs, and applications are abstracted using "NGSI-9 Client".


In order for their information on Things and/or Resources to be available at the Backend, IoT Agents (which play the role of "NGSI9 Client") need to register their information to the IoT-Discovery GE. This is done via the RegisterContext operation. Note that the registration can be performed at various levels of granularity: registering specific Entity/Attribute combinations, registering that information about certain entity type is available, or even in general registering that some entity information is available (without getting more specific) are all possible registrations. Also note that in absence of a Context Broker GE instance the registration is still stored in the IoT-Discovery GE.

This sequence is used for both new registrations and registrations updates.

NGSI-9 Register

Registration Update

A context provider can update a registration by using the registration ID that is returned from the previous registration response.
NGSI-9 Register Update


The entities that typically play the "NGSI-9 Client" role in this case are the IoT Broker GE (when requesting context availability information to process NGSI-10 query/updates) or any other NGSI application that wants to know the context availability associated with a given entity/attribute.

NGSI-9 Discover


The entities that potentially could play the "NGSI-9 Client" role are any application that needs to be aware of changes in context availability information. The "Subscribed App" could be the "NGSI-9 Client itself".

NGSI-9 Subscribe


The entities that potentially could play the "NGSI-9 Client" role are IoT Agents, while "Subscribed App" are applications subscribed to context availability information changed by the registration issued by the "NGSI-10 Client" (we assume that such Subscribed Application are previously subscribed, as shown in the sequence diagram in previous section).

NGSI-9 Notify

Added-value (Optional) Features

The following sections describe additional features that IoT-Discovery may implement. However, they are not mandatory for a basic implementation of this GE.

Geographical Discovery

As an optional feature, the IoT-Discovery GE supports the usage of geographic scopes in the operations discoverContextAvailability and subscribeContextAvailability. In particular, the geographic scopes defined under keyword SimpleGeoLocation in Appendix D of the OMA NGSI 9/10 specification [1] will be supported.

For specifying the geographic location of entities and/or context providers, a new type of registration metadata will be defined and implemented. The exact syntax and semantics of this metadata type is under discussion.

Probabilistic Discovery

The Probabilistic Search Engine is a component that uses machine learning (ML) techniques for resolving IoT Descriptions (whether RDF or Non-RDF format). It uses unsupervised probabilistic ML techniques to group similar IoT Descriptions. Its purpose is to allow automated discovery, ranking and recommendation of IoT Descriptions. It provides an efficient mechanism for the search and discovery of registered IoT Descriptions by automatically clustering all the IoT Descriptions and hence allowing the IoT Search Engine to be scalable to large object repositories. It also automates the ranking of results in order of relevance.

Probabilistic Search - Basic Diagram

Figure 6: Probabilistic Search Overview

Training the Engine

For the Probabilistic Search Engine to effectively refine the searching process, it goes through several stages. First it involves a training process whereby all the Descriptions for each model is retrieved from the repository. It will then extract "textual concepts" from the Descriptions using a parsing mechanism. The concepts are essentially the elements, attributes and values of a Description. The extracted textual concepts will then be used to create a "Document-term matrix", which is a mathematical matrix that is used to reflect the frequency of textual concepts in a particular IoT Description, whereby the rows represent the number of IoT Descriptions and the columns represent the numbers of concepts.

Concept Extraction

Figure 7: Concept Extraction

The next stage is another training process for clustering the IoT descriptions. The training involves a ML technique known as the Latent Dirichlet Allocation. The aim of this technique is to discover a hidden dimension behind the vector of concepts. This dimension essentially reflects a "topic" or "theme", which is shared by certain IoT Descriptions. The topic itself is modelled as Latent Factors, which allows any type of Descriptions to be represented in vector form, and hence making the search much more efficient compared to keeping them in text format. The model then associates the Latent Factors with the distribution of Concepts.

Latent Factor Extraction

Figure 8: Latent Factor Extraction

Once the second training process is done, the clusters are then created. The number of clusters created is based on the number of Latent Factors generated. The vector of Latent Factors that corresponds to an IoT Description is used to determine which Latent Factor best describes it.

Folding in new Descriptions

Figure 9: Folding in new Descriptions

After the training process, the Probabilistic Search Engine becomes online and is ready to receive requests. During runtime, the repository is expected to receive more registrations of IoT Descriptions. When this occurs the IoT Search Engine will read the new IoT Description and "fold" it in to one of the clusters based in its probability of similarity with one of the Latent Factors in the vector.

Querying the Engine

For discovery, a user can query the IoT Search Engine by providing a description template or a SPARQL query as the input to the search engine. The template should include concepts that are relevant to what the user is searching for. The search engine will then convert the query into Latent Factor vector- as done previously in the second training stage - and then match the query vector to the vector of Latent Factors that represents the IoT Descriptions stored in the repository. A query example is shown below.

Query Match

Figure 10: Query Match

A query about an Semantic Description is given in the form:

<?xml version="1.0"?>
       <vm:hasAttributeType rdf:resource="http://purl.oclc.org/NET/ssnx/qu/quantity#humidity"/>
       <vm:hasAttributeName rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
   <vm:hasName rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
   >BA 38 01</vm:hasName>
   <vm:hasA rdf:resource="#HumidityAttributeBA38"/>
   <vm:hasType rdf:datatype="http://www.w3.org/2001/XMLSchema#anyURI"

...whereby the query is asking for an Entity - in this case a room (38) - that has Humidity information about it.

The Search Engine does as follows:

  1. The query is converted to a vector of latent factors: {0.038, 0.916, 0.006, 0.006, 0.0, 0.006, 0.01, 0.008, 0.003, 0.0}
  2. After analysing the vector, the query belongs to latent factor #2 with the highest probability.
  3. Query is forwarded to the registry responsible for cluster #2 and query is matched within that cluster.
  4. The first five results obtained are:
  • Rank #1: 38_BA_01.owl with a score of 0.998479530076855
  • Rank #2: 44_BA_01.owl with a score of 0.998479530076855
  • Rank #3: 01_BA_02.owl with a score of 0.998479530076855
  • Rank #4: 37_BA_01.owl with a score of 0.998479530076855
  • Rank #5: 18_BA_02.owl with a score of 0.998479530076855

Once this is done, the respective IoT Descriptions are retrieved from the repository and returned to the user.

Semantic Discovery


NGSI provides a simple, generic and effective means of exchanging information about context entities. Although, it does not specify a global, interoperable naming convention for them. Hence an entity type, attribute, or attribute metadata can be syntactically declared differently from different context producers, even though they might point to the same phenomenon. Or the case could be the other way round, whereby the same syntax has different semantics among different context producers. Therefore there is a need to enable the semantic interoperability of context information among context producers and consumers, so that they are "talking" about the same thing. Here Linked Open Data can play role in increasing interoperability, whereby fields within an NGSI description can be linked to other well-known descriptions that provide more information about it. Since this GE is targeted for IoT entities, then an IoT reference information model is needed to define the linking of an IoT related attribute declared in NGSI to one that is.

This optional functionality within the IoT Discovery GE is targeted for:

  • Context Producers wanting to register their IoT entities using semantic IoT descriptions.
  • Context Consumers using SPARQL for discovering IoT Context Entities.
  • Applications requiring Semantic Discovery of IoT Context Entities.
  • Semantically-enabled GEs
    • As a reference, semantic IoT Descriptions in this specification are based on the IoT-A ontology models.

Linking NGSI Entities to Semantic Descriptions

Within the FIWARE architecture, NGSI-9 is the main protocol for the registration and discovery of sources of context information. The NGSI-9 interface for discovering IoT entities is effective due to its simplicity and generality. But it is also regarded as a loose binding in the sense that it not does provide a global naming convention for entities and their attributes, which will essentially affect interoperability between descriptions that might match in terms of syntax, and not in semantics. Therefore, to increase interoperability, descriptions annotated using NGSI Context Information model should be exposed as linked open data, so that its attributes can be compared with other entities on the web, whether it be spatial, temporal or thematic. In order to do this, a mapping must be done between the information annotated in NGSI to an semantic information model that reflect's IoT entities and their properties. As an initial reference, the ontology models defined by the EU FP7 IoT-A project can be used.
A Context Producer can link an NGSI Entity Description to an Semantic Description by adding a Context Attribute that provides the URI of the corresponding Semantic Description, when registering the NGSI Entity via the registerContextRequest operation. This requires the Semantic Description to be available beforehand.


The Association Engine (AE) is a component that establishes and maintains associations between IoT Entities and Services that expose Resources, which acts as the representative for interacting with a Resource. The associations are made based on spatial, temporal and thematic relativity:

  • Thematic (feature) match - using domain ontologies that capture an Entity's attributes and IoT Service's input/output parameter.
  • Spatial match - using either co-ordinate comparison between IoT Entities and Services for determining close proximity, or location ontologies that model logical locations with properties such as 'contains'
  • Temporal match - utilising temporal aspects of entities which have a temporal aspect, such as meeting rooms with the IoT Service's observation_schedule.
Association process

Figure 11: Association Process Overview

Creating the Associations

Formulation of associations utilises the IoT-A Entity and Service models (i.e. Semantic Descriptions). Dynamic association inference is maintained through rules that incrementally reason on theme, spatial aspects and time. At startup, the AE will retrieve from the database all the different types of Semantic Descriptions - Entities, and Services (N.B. Services represent Resources) - and processes them through an Association Engine for matching. The matching process first looks for similar themes between Entity Attributes and Service observation types, e.g. temperature, light intensity, acoustic level. It will then look for similar locations. Lastly in the matching process it will look for temporal availability of a Service. Once the matching process is done, the associations generated will then be stored, which can then be queried via SPARQL.

Association process

Figure 12: Association Mechanism

Querying for Associations

The association can then be queried by users via the SPARQL interface. In the query the predicate used to retrieve the association is "isAssociatedToService". This will return the URI of the IoT-A service description, which can then be retrieved by querying the IoT-A Service repository. It is this description that will provide access to contextual information (e.g. temperature reading) produced by the IoT-A Resource (e.g. temperature sensor).

Personal tools
Create a book