We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Data.MetadataPreprocessing - FIWARE Forge Wiki

FIWARE.OpenSpecification.Data.MetadataPreprocessing

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Data.MetadataPreprocessing
Chapter Data/Context Management,
Catalogue-Link to Implementation [ <N/A>]
Owner Siemens AG, Peter Amon


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


FIWARE WIKI editorial remark:
This page corresponds to Release 3 of FIWARE. The latest version associated to the latest Release is linked from FIWARE Architecture

Copyright

  • Copyright © 2012-2014 by SIEMENS

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

Target usage

Target users are all stakeholders that need to convert metadata formats or need to generate objects (as instantiation of classes) that carry metadata information. The requirements to transform metadata typically stem from the fact that in real life various components implementing different metadata formats need to inter-work. However, typically products from different vendors are plugged together. In this case, the Metadata Preprocessing GE acts as a mediator between the various products.

Example scenarios and main services exported

The Metadata Preprocessing GE is typically used for preparing metadata coming from a data-gathering device for subsequent use in another device (i.e., another GE or an application or another external component). The data-gathering device can be a sensor, e.g., the analytics component of a surveillance camera. Depending on the manufacturer of the camera, different metadata schemes are used for structuring the metadata. The Metadata Preprocessing GE generally transforms the metadata into a format that is expected by a subsequent component, e.g., a storage device. In addition to performing the transformation of the metadata format (e.g., defined by XML Schema), also some elements of the metadata can be removed from the stream by a filtering component. This is especially useful in case these elements cannot be interpreted by the receiving component. For example, in the Use Case project OUTSMART (more specifically, in the Santander cluster), coma-separated sensor metadata is transformed into XML metadata by the Metadata Preprocessing GE for further processing and storage. The transformation task is described in more detail in the following. OUTSMART's advanced measure & managing system (AMMS) is constantly producing sensor data for the following format:

[...]
06E1E5A2108003821;29/05/2012;01;1;AI;000107;0
06E1E5A2108003821;29/05/2012;01;1;RI;000012;0
[...]

The format can be interpreted as follows: AMMS identifier; date; hour; quarter (of hour); type of measurement; data measured; error code. The Metadata Proprocessing GE transforms this data into XML for further processing.

<io>
    <obs from="urn:outsmart:06E1E5A2108003821">
        <stm>2012-05-29T10:00:00+02.00</stm>
        <what href="phenomenon:activepower"/>
        <param href="identifier:UniversalIdentifierOfLogicalHub"><text>sc.poc.outsmart.eu</text></param>
        <data><quan uom="uom:watts/h">107</quan></data>
    </obs>
    <obs from="urn:outsmart:06E1E5A2108003821">
        <stm>2012-05-29T10:00:00+02.00</stm>
        <what href="phenomenon:reactivepower"/>
        <param href="identifier:UniversalIdentifierOfLogicalHub"><text>sc.poc.outsmart.eu</text></param>
        <data><quan uom="uom:Varh">12</quan></data>
    </obs>
</io>

Note that a special transformation unit (implemented by the GE owner) was necessary to realize this task. (The transformation could not be specified by an XSLT stylesheet, since the input data was not in XML format.)

Basic Concepts

Functional components of the Metadata Preprocessing GE

The following figure depicts the components of the Metadata Preprocessing Generic Enabler. These functional blocks are the Control Interface, the Metadata Interface for inbound streams, the Metadata Transformation & Filtering component, and the Metadata Interface for outbound (processed) streams. The mentioned methods are described in more detail in Section Main Interactions.

Functional components of the Metadata Preprocessing GE

The functionality of the components is described in the following bullet points.

  • Control Interface: The control interface is the entity for configuring and controlling the metadata processing engine. The algorithms used for transformation and filtering as well as the metadata source are configured, i.e., connected using the configureInstance method. Sinks receiving the outbound streams can connect using the information provided in the getMetadata method. More details on the APIs are described in the Section "Main Interactions".
  • Metadata Interface (for inbound streams): Different interchange formats (such as the ones for streaming or for file access) can be realized (i.e., configured or programmed into this interface at a later stage). At the current stage, the Real-time Transport Protocol (RTP) as standardized in RFC 3550 [RFC3550] is supported as interchange format. Different packetization formats for the contained/packetized payload data (i.e., the XML metadata) depending on the application might be used. Usually, the Inbound Metadata Interface" as a (RTSP/RTP) streaming client connected to a (RTP/RTSP) streaming server (i.e., the Streaming Device).
  • Metadata Transformation & Filtering: The Metadata Transformation & Filtering component is the core component of this Generic Enabler. Based on an XML Stylesheet Language for Transformations (XSLT) [XSLT] and a related stylesheet, the processing of the metadata is performed. In principle, other kinds of transformations (other than XSLT) can also be applied (e.g., text-to-XML transformation); however, dedicated changes to the enabler are needed for this. Metadata filtering is an optional step in the processing chain. The filtering can be used, e.g., for thinning and aggregating the metadata, or simple fact generation (i.e., simple reasoning on the transformed metadata). Filtering is usually done during transformation by using XSLT technology. The output of this step is a new encapsulation/formatting of the metadata received.
  • Metadata Interface (for outbound streams): Through this interface, the transformed (and possibly filtered) metadata or metadata stream is accessed. For example, the "Device" connected to the Outbound Metadata Interface can be a (RTSP/RTP) streaming client. In this case, the Outbound Metadata Interface acts as a (RTSP/RTP) streaming server.

Realization by MetadataProcessor asset

The MetadataProcessor asset realizes a specific implementation of the Metadata Preprocessing GE. Timed metadata (i.e., metadata elements with associated timing information, e.g., a timestamp) is received over an RTSP/RTP interface (as specified in RFC 2326 [RFC2326] and RFC 3550 [RFC3550], respectively), which implements the metadata interface for inbound data/streams. Different RTP sessions can be handled; therefore metadata streams can be received from several components/devices (e.g., cameras or other type of sensors). The target in such a realization could be the provision of metadata as facts to a metadata broker, which would be the receiver of the outbound stream.

Main Interactions

The external API is a RESTful API that permits easy integration with web services or other components requiring metadata access and transformation services (e.g., other GEs or external applications/components). The following interface will be supported:

  • getVersion: The version of the Metadata Preprocessing GE is returned.
  • listInstances: All instances (i.e., processing units) of the Metadata Preprocessing GE are listed.
  • createInstance: An instance for processing metadata streams/events is created.
  • getInstanceInfo: The information about a specific instance (i.e., processing unit) is returned.
  • destroyInstance: An existing metadata processing instance is destroyed.
  • startInstance: The metadata processing (e.g., transformation and/or filtering) is started.
  • stopInstance: The metadata processing is stopped/halted.
  • getConfig: The configuration of a specific processing unit is returned.
  • configureInstance: A metadata source (e.g., another GE) is connected to the enabler and/or the metadata processing (e.g., the XSLT stylesheet for the conversion of metadata formats and filtering of metadata streams/events) is configured for a specific instance (i.e., processing unit).
  • getMetadata: The URI (i.e., RTSP URL)of the metadata output stream is returned.

The following figure explains the main interactions in a (very general) example usage. In the first step, a new instance for metadata processing is created. The ID of the instance is returned to the calling application/component. In a second step the processing of the Metadata Preprocessing GE is configured (e.g., by providing an XSLT stylesheet). In a third and fourth step the source and the sink of the metadata processing are configured. Note that the order of the configuration steps (i.e., configureInstance, getMetadata) is arbitrary. Note further that more than one sink can be added as receiving component, but only one source can be configured. (However, additional processing units for metadata transformation can be created using createInstance.) In a fifth step, the processing is started.

Image:mdpp_example_scenario.jpg
Example usage

After the processing is done, the specific instance of the GE is stopped. Note that the instance could be started again afterwards by re-using its instance ID. (A list of all existing instances can be retrieved using the listInstances request.) Also the processing of the source could be reconfigured and sinks can be added or removed. As a final step in this example usage, the specific metadata preprocessing instance is destroyed. Note that it is not necessary to stop the instance before destroying it, since this will be done automatically. A simple but concrete example for metadata transformation and metadata filtering can be found in the Open RESTful API Specification of this GE.

Basic Design Principles

The following basic design principles apply:

  • The Metadata Preprocessing GE realizes a generic metadata transformation approach, which is not restrictive to specific metadata schemes.
  • Encapsulation of transport and metadata transformation is implemented as-a-service, usable from other web applications or components.
  • Transformation is based on standardized and commonly used XML Stylesheet Language for Transformations (XSLT).

References

[RFC2326] H. Schulzrinne, A. Rao, and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, Apr. 1998.
[RFC3550] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: A transport protocol for real-time applications", RFC 3550, Jul. 2003.
[XSLT] W3C / M. Kay (editor), "XSL Transformations (XSLT) Version 2.0", http://www.w3.org/TR/xslt20/, Jan. 2007.


Detailed Specifications

Following is a list of Open Specifications linked to this Generic Enabler. Specifications labeled as "PRELIMINARY" are considered stable but subject to minor changes derived from lessons learned during last interactions of the development of a first reference implementation planned for the current Major Release of FI-WARE. Specifications labeled as "DRAFT" are planned for future Major Releases of FI-WARE but they are provided for the sake of future users.

Open API Specifications


Re-utilised Technologies/Specifications

The following technologies/specifications are incorporated in this GE:

  • Extensible Stylesheet Language Transformation (XSLT) Version 1.0 as defined by the W3C,
  • Real-time Transport Protocol (RTP) / RTP Control Protocol (RTCP) as defined in RFC 3550,
  • Real-Time Streaming Protocol (RTSP) as defined in RFC 2326.

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions

  • Data refers to information that is produced, generated, collected or observed that may be relevant for processing, carrying out further analysis and knowledge extraction. Data in FIWARE has associated a data type and avalue. FIWARE will support a set of built-in basic data types similar to those existing in most programming languages. Values linked to basic data types supported in FIWARE are referred as basic data values. As an example, basic data values like ‘2’, ‘7’ or ‘365’ belong to the integer basic data type.
  • A data element refers to data whose value is defined as consisting of a sequence of one or more <name, type, value> triplets referred as data element attributes, where the type and value of each attribute is either mapped to a basic data type and a basic data value or mapped to the data type and value of another data element.
  • Context in FIWARE is represented through context elements. A context element extends the concept of data element by associating an EntityId and EntityType to it, uniquely identifying the entity (which in turn may map to a group of entities) in the FIWARE system to which the context element information refers. In addition, there may be some attributes as well as meta-data associated to attributes that we may define as mandatory for context elements as compared to data elements. Context elements are typically created containing the value of attributes characterizing a given entity at a given moment. As an example, a context element may contain values of some of the attributes “last measured temperature”, “square meters” and “wall color” associated to a room in a building. Note that there might be many different context elements referring to the same entity in a system, each containing the value of a different set of attributes. This allows that different applications handle different context elements for the same entity, each containing only those attributes of that entity relevant to the corresponding application. It will also allow representing updates on set of attributes linked to a given entity: each of these updates can actually take the form of a context element and contain only the value of those attributes that have changed.
  • An event is an occurrence within a particular system or domain; it is something that has happened, or is contemplated as having happened in that domain. Events typically lead to creation of some data or context element describing or representing the events, thus allowing them to processed. As an example, a sensor device may be measuring the temperature and pressure of a given boiler, sending a context element every five minutes associated to that entity (the boiler) that includes the value of these to attributes (temperature and pressure). The creation and sending of the context element is an event, i.e., what has occurred. Since the data/context elements that are generated linked to an event are the way events get visible in a computing system, it is common to refer to these data/context elements simply as "events".
  • A data event refers to an event leading to creation of a data element.
  • A context event refers to an event leading to creation of a context element.
  • An event object is used to mean a programming entity that represents an event in a computing system [EPIA] like event-aware GEs. Event objects allow to perform operations on event, also known as event processing. Event objects are defined as a data element (or a context element) representing an event to which a number of standard event object properties (similar to a header) are associated internally. These standard event object properties support certain event processing functions.
Personal tools
Create a book