We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.ArchitectureDescription.Apps.Repository - FIWARE Forge Wiki

FIWARE.ArchitectureDescription.Apps.Repository

From FIWARE Forge Wiki

Jump to: navigation, search

Contents

Copyright

  • Copyright © 2014 by SAP
  • Copyright © 2015 by UPM

Legal Notice

Please check the following FIWARE Open Specification Legal Notice (essential patents license) to understand the rights to use this open specification. This scheme differs from the one chosen for the rest of open specifications that make up the FIWARE platform since it is inherited from the one chosen by SAP between the two FIWARE license schemes allowed in Release 3.

To illustrate this open specification license from our perspective:

  • The specifications of this Generic Enabler are made available under IPR rules that allow for a exploitation and sustainable usage both in Open Source as well as proprietary, closed source products to maximize adoption.
  • This Open Specification is exploitable for proprietary 3rd party products and is exploitable for open source 3rd party products, including open source licenses that require patent pledges.
  • If the owner of this GE spec holds a patent that is essential to create a conforming implementation of the GE spec (i.e. it is impossible to write a conforming implementation without violating the patent) then a license to that patent is deemed granted to the implementation.

Note: SAP provides the software associated to the Repository - RI (Reference Implementation) according to the initial specification. UPM provides the software associated to the Repository - RI according to the current version of the specification as reflected in these pages. Both implementations are provided as open source under the BSD License. Please check the specific terms and conditions linked to this BSD open source license at https://github.com/conwetlab/Repository-RI/blob/master/license.txt. You can also obtain the Repository GE software using the FIWARE Catalogue at http://catalogue.fiware.org/enablers/repository-repository-ri

Overview

Together with the Store GE and the Marketplace GE, the Repository GE is a core enabler of the FIWARE Business Framework. The Repository GE provides a consistent uniform API to access USDL service descriptions and associated media files for applications of the business framework. A service provider can use the Repository GE to publish the description of various aspects of the service according to a unified description language.

USDL is used in its Linked Data version "Linked USDL". Documentation can be found at <http://linked-usdl.org/> . Information about the FIWARE Platform is available at http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php . USDL describes services on a metadata level and can refer to supplemental resources of any media type. Therefore the Repository GE must be able to store resources in arbitrary formats. The RDF datamodel of USDL allows referring to entities of the service description via the resource URL. Therefore Linked-USDL is already well prepared to allow the distribution of service descriptions all over the Internet.

Moreover, the Repository GE provides a uniform API to perform smart searches over the published descriptions. This API is oriented to service descriptions and media files serialized in RDF formats (e.g a Linked-USDL document) and allows the usage of SPARQL as query language.

Target usage

The Repository GE is a place to store service models, especially USDL descriptions but also other models required by components of the overall FIWARE business framework (e.g. technical models for service composition and mashup). The Repository GE provides a common location for storage (centrally or distributed and replicated), reference and/or safety.

The Repository GE contains published descriptions, which can be utilized by any component in respect to privacy and authorization constraints imposed by the business models. Usually a repository is under control of an authority and usually is keeping track of versions, authenticity and publication dates.

User roles

  • The Provider creates services describing basic service information as well as technical information. He needs to upload and publish service descriptions on the repository in order to make them available to other components of the platform, such as the Shops/Stores, Aggregators, etc.
  • The Aggregator will use for example technical and service-level information of existing services in the Repository in order to perform a composition of a value added service or application. So for example, in order to give a valid statement about the availability of the new composed service. The availability of each contained service needs to be considered. The Aggregator also needs information about the technical interfaces of a service in order to develop code to call them correctly . Service descriptions for the newly created composite service can be uploaded and published to the repository again.
  • The Marketplace Provider enables consumers to make comparisons between services therefore needs all kind of business relevant descriptions of services, such as general descriptions, business partners, service-levels, and pricing, to be presented in the store or within the marketplace. Also technical information can be required, on a level to be able to do comparisons between services for the consumer.
  • The Channel Maker. The internet consists of various channels and heterogeneous devices for consuming information such as Web (browser), Android, iOS but also global as well as local social networking and community platforms such as Facebook, LinkedIn, MySpace, Xing, KWICK! The Channel Maker needs detailed information about the channel to ensure the proper channel creation or selection. Further a channel may require embedding or wrapping the service so it can be accessed by the user through the specific channel.
  • The Hoster requires information on service-level descriptions, deployment and hosting platform requirements to provide the necessary infrastructure needed to host applications/services in a reliable and scalable way.

Basic Concepts

Web Citizen

The Repository GE is relying on Web principles:

  • URI to identify resources
  • Consistent URI structure based on REST style protocol
  • HTTP content negotiation to allow the client to choose the appropriate data format supporting HTML, RDF, XML, RSS, JSON, Turtle, ...
  • Human readable output format using HTML rendering ('text/html' accept header) including hyperlinked representation
  • Use of HTTP response codes including ETags (proper caching)
  • Linked Data enablement supporting RDF input and output types

Open Distributed Architecture

The Repository GE Open Specification has to be seen as a specification of the repository abstract functionality. There can be many technologies used to implement the functionality. Often the repository protocol is implemented on top of a Web content management system.

Also we envision a large number of repositories containing service descriptions, which also might to refer to descriptions in other repositories. Repositories can be hosted by the provider or a provider may use repository services of other platform providers. The latter might be an alternative for small sized providers, which don't want to provide an own infrastructure.

The service descriptions in a repository are typically used by different other components of the platform, such as service stores or marketplaces, by extracting information needed for the specific functionality.

Data Model

The Repository GE is structured into core objects, which are resources and collections. These objects constitute also the granularity of access control. Collections are containers for storing resources, which are typically used to maintain all resources belonging to a certain service description in one place.

Resources

The resources are mainly the USDL service descriptions themselves as well as complementary media files that are used within the service descriptions.

Collections

A collection is a container for collecting resources. Multiple collections can be used on the repository for various purposes. Collections can be nested and may provide versioning of the resources. Collections are used to keep all content that is locally referred from the service descriptions together in one place. For example a service description often has additional documentation, depictions and other collateral information, which can be bundled together in one collection.

Recipes

Recipes are virtual containers selecting resources from different collections.

Content Negotiation

For optimal interoperability and flexible use, the Repository GE should be able to deliver the results of an operation in multiple formats. HTTP content negotiations should be used to let the client choose an appropriate content type. Basic content types (mime-type) are

  • HTML - to deliver the results in hyper-linked HTML that can be rendered directly in a Web Browser
  • RDF - various RDF serializations for processing in applications
  • JSON - Javascript Object Notation for easy processing in a mashup environment.

Repository Architecture

The Repository GE is used by various other GEs within the FIWARE platform. Namely Marketplace GE, Store GE, Application Mashup GE as well as the Revenue Settlement and Sharing System GE can access the Repository GE to retrieve detailed information about a service or application. In general, the different services and applications offered in the FIWARE Business Framework are described in Linked-USDL documents that are published in the Repository GE. These descriptions can be later used to discover and compare offerings.


Repository in the context of the Business Framework

Besides the FIWARE platform, Future Internet applications or composite services on top of the FIWARE platform can also use the Repository GE as a service for their own purpose. An example of the inner architecture of the Repository is shown in the following diagram.

Example of high-level architecture of a repository implementation

The architecture shown here is only a blueprint for possible implementations of the Repository GE and depicts the functional components, necessary to realize this functionality. There are many technology options for a concrete implementation, depending on the context and application domain and its nonfunctional requirements. Since the requirements according to repository size, workload and other parameters can be quite different, there is no obvious all-encompassing implementation solution. The implementations can span very simple ones, which provide only few extensions to a standards Web service to very sophisticated ones that utilizes enterprise content management systems (e.g. based on the "CMIS - Content Management Interoperability Services" standard).

The Repository GE only stores and provides access to service descriptions. Since there is no common standard for versioning, and the requirement according to versioning may vary depending on the use case scenario, we do not require version control from a repository implementation, although a real implementation can provide versioning models and mechanism (e.g. using the capabilities of the underlying CMS system).

Also there is no requirement regarding consistency checking of the service descriptions in the Repository GE. The applications themselves have to ensure that the descriptions are consistent. All clients of a repository have to cope with incomplete and inconsistent information by default. This reflects the architecture of the Web, where also no consistency commitment of the pages on different Web servers can be made. To ensure integrity additional measures have to be taken.


Technical Interfaces

Main Interactions

The Repository GE operation protocol is kept very simple. It basically provides operations to get and put resources, such as service descriptions and media content. Additional operations are used to structure the repository into collections of resources.

Managing Resources (mandatory)

The core functionality of the Repository GE is to store resources and retrieve them when necessary. Further resources sometimes need to be updated and eventually deleted. The following diagram shows an example sequence of resource management operations of the Repository GE.

Example sequence of resource management operations

The Get Resource operation can be used to retrieve a resource from the Repository GE. This operation delivers the actual content of the resource and/or metadata about the resource, such as the media type, creator, or modification date, depending on the used technical interface. The following parameters need to be exchanged:

  • resource identifier - Resource identifier of the resource to be returned.

If only information about collections is requested, the collection identifier is used instead of the resource identifier.

  • collection identifier - Identifier for the collection, which contains the resource.
  • resource - Resource which will be returned
  • media type - Media type of the resource which will be returned.


The Post Resource operation is used to store a new resource in the Repository GE. The repository should take precautions to provide inconsistent changes due to concurrent access. The following parameters need to be exchanged:

  • resource identifier - Identifier which contains the resource.
  • collection identifier - Identifier which denotes the collection which the resource will be put. The collection can be a part of the resource identifier. For example, when URL paths are used to identify a resource.
  • resource - the content of the resource to be stored in the repository.

The Put Resource operation is used to store a new resource into the repository or update an existing resource with the same resource identifier. The repository should take precautions to provide inconsistent changes due to concurrent access. The following parameters need to be exchanged:

  • resource identifier - Identifier which contains the resource.
  • collection identifier - Identifier which denotes the collection into which the resource will be put. The collection can be a part of the resource identifier, if for example URL paths are used to identify a resource.
  • resource - the content of the resource to be stored into the repository


In order to delete a resource irrevocably from the repository the Delete Resource operation is used. The following parameters are exchanged:

  • resource identifier - Resource identifier of the resource to be deleted.


Managing Collections

Collections are used to put a structure into the Repository GE. In order to easily access parts of the Repository GE, it allows clients to get information about the contents of individual collections. The following diagram shows an example sequence of collection management operations of the Repository GE.

Example sequence of collection management operations

The Create Collection operation creates a new collection in the Repository GE, containing the necessary details such as owner, policies, and other metadata attributes. It requires the parameter:

  • description - Description of the collection to be created, which contains the location path within the repository and administrative data such as creator and access policies.


To get the details and contents of a collection the Get Collection operation is used. The collection information contains information such as owner, policies, textual descriptions, dates, versions, number of resources, and more. The level of detail of the description may depend on the authorization level of the requester. The following parameters can be involved in the operation:

  • collection identifier - Collection identifier for which a description is to be returned.
  • filter - Optional filter expression to select the properties to be filtered.
  • description - Returned collection description containing information according to the filter expression.


A collection in the Repository GE can be deleted with the Delete Collection operation. This operation can only be successful for a requester that has the appropriate authorization. The delete operation requires the identifier of the collection as input. After this operation the collection is no longer accessible for clients. Only one parameter is necessary:

  • collection - Collection identifier for the collection to be deleted


Listing Content

The List operation lists collections and/or resources contained in the Repository GE, which are accessible by the user. This operation usually is needed for a repository browser and maintenance tool as well as an editor tool in order to select the resource to be maintained.

The operations using the following parameters: No input parameters are required. However, for practical reasons it might be useful to restrict the list of collections and resources by specifying the number of results and the starting offset.

  • collection - Collection for which the list is restricted to.
  • index - Index of the first element of the result set to be returned.
  • limit - Maximal number of results to be returned.
  • filter - A repository implementation might also offer the possibility to filter the list of collections according to specific criteria. An optional filter expression can be used to reduce the number of delivered results. A repository may support different criteria to filter the output
  • result list - The operation results in a list of collections/resources that is returned to the client and contains resource descriptions according to the collection, filter, index, and limit expressions.


List the additional services

Besides the operations described above, a repository might provide additional services, such as search, backup, etc. A repository should list and describe the additional services to the clients when the List Services operation is invoked. If additional services are offered only for specific collections of the Repository, the collection identifier can be used to list the actual available functionalities for this collection. The required parameters for this operation are:

  • collection - Collection identifier for which the additional services will be listed.
  • result list - List of additional services are returned to the client.


Searching the Repository

A Repository might provide searching service, to search service descriptions according to the occurrence search terms in properties of the description and media content. It is desirable that in compliance to OpenSearch the repository provides an OpenSearch description to the search API.

Querying the Repository

The Repository GE also provides a more complex querying service in a specific query language. As an example a query service based on SPARQL [ REP2 ]would allow to execute complex queries on the Linked Data RDF model [ REP1 ].

The following diagrams shows an example sequence of operations to perform queries using SPARQL:

Example sequences of smart query operations

The Get Query operation is a service used to obtain Linked USDL data of the Repository GE by using a query that has less than 2054 characters. The following parameters can be involved in the operation:

  • query - Query to be executed.
  • media type - Media type of the Linked Data will be returned.
  • result RDF - Result obtained of executing the query.


The Post Query operation is a service used to obtain Linked USDL data of the Repository GE by using a query without size restriction.The following parameters can be involved in the operation:

  • query - Query to be executed.
  • media type - Media type of the Linked Data will be returned.
  • result RDF - Result obtained of executing the query.

Basic Design Principles

Rationale

There are many proprietary solutions implementing repository functionality and also many standards for various types of repositories. Within FIWARE we try to abstract this functionality into a Generic Enabler.

Implementation agnostic

The API abstracts from the concrete implementation technology. Implementations using various kinds of databases should be possible. Although the main goal is to store services descriptions in a distributed environment, any implementation of the repository GE can be used as long as the technical interfaces comply with the GE operation protocol and can be mapped (mediated) to the FIWARE preferred REST-based reference implementation.

References

REP1 RDF Primer W3C Recommendation 10 February 2004 (http://www.w3.org/TR/2004/REC-rdf-primer-20040210/)
REP2 SPARQL 1.1 Overview, W3C Working Draft 01 May 2012 (http://www.w3.org/TR/2012/WD-sparql11-overview-20120501/9
Personal tools
Create a book