We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Data.MetadataStoreManagementPlatform - FIWARE Forge Wiki

FIWARE.OpenSpecification.Data.MetadataStoreManagementPlatform

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Data.MetadataStoreManagementPlatform
Chapter Data/Context Management,
Catalogue-Link to Implementation [This GE is not present on the catalogue since it was discontinued MetaWare]
Owner Telecom Italia, Fabrizio Verroca (TI)


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.

Copyright

Copyright © 2010-2014 by TI. All Rights Reserved.

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

Metadata is data that describes other data. Meta is a prefix that in most information technology usages means "an underlying definition or description." In our context Metadata is used to described objects in FiWare Big Data environment.

Basic Concepts

Metadata Store GE will allow a smooth approach to a general Big Data platform decoupling from technical usage issues, maintaining platform objects descriptors enabling complex processing flows and controlling the access to the data and data APIs by the authorized users, keeping accounting of the data consumption and supporting its publication. More in generale Metadata store GE enable the ease of use. Everything that a User can use, publish, buy, in the FiWare Big Data environment, can be managed as an object and can be described into the Metadata Store. An object can be private and owned by user owner or an object can be public, owned by user owner and buyed by one or more users. The Metadata Store GE maintains the relation between user and his objects. The object types described by MetaData Store GE are:

  • User
  • Dataset
  • Algorithm
  • Process
  • API
  • Dashboard
  • Application

Technologies Involved

The technologies involved are:

  • Document DB NoSQL products support a whole range of new data types, and this is a major area of innovation in NoSQL. A Document DB is used for metadata repository implementation. A database holds a set of collections. A collection holds a set of documents. A document is a set of key-value pairs. Documents have dynamic schema. Dynamic schema means that documents in the same collection do not need to have the same set of fields or structure, and common fields in a collection’s documents may hold different types of data.
  • DAO Pattern Data Access Object Pattern or DAO pattern is used to separate low level data accessing API or operations from high level business services. For each object described in MetaData Store a set of web services are provided for accessing it.

General Information

Big Data, true to its name, deals with large volumes of data characterized by volume, variety and velocity. Any enterprise that is in the process of or considering a Big data applications deployment has to adress the metadata management problem. Big Data introduces large volumes of unstructured data for analysis. This data colud be in the various form. To bring this data into the fold of information management solution, its metadata should be correctly defined. A good metadata management solution must provide visibility across multiple solutions and bring business users into the fold for a collaborative, active metadata management process. The purpose of MetaData Store Managment Platform GE is to adress the issues described before offering a metadata repository accessible through RESTful Web Services.

How It Works

Metadata is defined by a json with a basic structure that describes the characteristics of each object; inside it there is an array that can contain a list of json objects whose structure is basic object. In this way you can describe categories and subcategories of objects with a single structure that contains all the necessary information obtainable in the same way.

Each object in FiWare Big Data environment is described by its own metadata in MetaData Store management platform GE. Object's metadata are provided through RESTful web services that implements CRUD operations.

MetaDataStore Mangement Platform GE Architecture

The next figure shows the MetaData Store GE high level architecture. The Metadata store management platform describes the objects used by GEs FiWare Big Data environment. It is composed by:

•Metadata Access Layer (via Restful web services)

•Metadata Repository: is the container of metadata describing the objects of the platform (syntax and semantics)

Figure 1.
MetaData Store GE High Level Architecture

Basic Design Principles

The MetaData Store GE is designed with the following the Data Access Object (DAO) pattern. The Data Access Object (DAO) pattern is a mechanism to abstract away the details of persistence in an application. The idea is that instead of having the domain logic communicate directly with the database, file system, web service, or whatever persistence mechanism the application uses, the domain logic communicates to a DAO layer instead. This DAO layer then communicates with the underlying persistence system or service.

DAO
Figure 2.
DAO pattern used in MetaData Store GE

The advantage of the DAO layer is that if you need to change the underlying persistence mechanism you only have to change the DAO layer, and not all the places in the domain logic where the DAO layer is used from. The DAO layer usually consists of a smaller set of classes, than the number of domain logic classes that uses it. Should you need to change what happens behind the scene in the DAO layer, the operation is somewhat smaller, since it only affects the DAO layer. It is also a somewhat more controlled operation, since you can search for all DAO classes, and make sure they are changed to use the new persistence mechanism.

Figure 3.
How DAO pattern works

Data access layer implements DAO classes that provide CRUD(create, read, update, delete) operations for each object in Metadata Repository.

Main Interactions

In this section a couple of interactions example are provided. Both examples refer to the interactions between Metadata Store GE and CKAN GE. In the first case (figure 4) an authorized user is connencted to Big Data GE (Cosmos) trough an application (in the case a Big Data Portal). The user also access to CKAN and select an Open Data dataset to use for his purposes. Before to import the dataset in Big Data GE is needed to import the linked metadata. The Metadata Store GE has an API to do that. From that the dataset is added to the list of datasets owned by the user.

Figure 4.
How retrive metadata from CKAN

In the same case the user can publish a private dataset as Open Data in CKAN. In this case (figure 5) the user choose the dataset to publish and update the linked metadata. Metadata update include also the translation in DCAT data model for Open Data. At the end of this step metadata is exported to CKAN, through the export API, and the dataset can be published.

Figure 5.
How to publish a private dataset as Open Data in CKAN

Detailed Specifications

Please refere to MetaWare on GitHub MetaWare

Re-utilised Technologies/Specifications

The Project Makes use of:


Terms and definitions

The description should not contain specific terms with the need to be explained. However, if this may happen, please contact GE owner for any clarification

Personal tools
Create a book