We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.ArchitectureDescription.Cloud.ObjectStorage V2 - FIWARE Forge Wiki

FIWARE.ArchitectureDescription.Cloud.ObjectStorage V2

From FIWARE Forge Wiki

(Difference between revisions)
Jump to: navigation, search
(Legal Notice)
 
Line 5: Line 5:
= Legal Notice =
= Legal Notice =
-
Please check the following [[FI-WARE Open Specifications Legal Notice (amended essential patents license) | Legal Notice]] to understand the rights to use these specifications.
+
Please check the following [[FI-WARE Open Specification Legal Notice (essential patents license) | Legal Notice (essential patents license)]] to understand the rights to use these specifications.
= Overview =
= Overview =

Latest revision as of 08:04, 13 August 2014

Contents


Copyright

The FI-WARE ObjectStorage Specification is Copyright © 2013 INTEL. Please note that this specification adopts the CDMI standard, which is published by and copyright SNIA[1].

Legal Notice

Please check the following Legal Notice (essential patents license) to understand the rights to use these specifications.

Overview

Object Storage is one of the Generic Enablers within FI-WARE. It offers persistent storage for digital objects, important cloud-based functionality that has been specifically requested by Use Cases within the project. Objects can be any binary content that needs to be archived. Typically objects correspond to files. Groups of objects can be stored in a common location known as a container. Containers can contain other containers as well as objects and so a hierarchy of containers can be created.

Containers and objects can have Metadata associated with them, capturing details about the nature of the container or the object. As with files in a traditional filesystem, objects in an Object Store belong to particular users (or accounts). Objects and containers are manipulated through a RESTful-style API. To promote interoperability and usability, the ISO-ratified Cloud Data Management Interface (CDMI) standard from the Storage Networking Industry Association (SNIA) has been employed as the API for this GE.

The following sections give more details on all of these topics.

When considering the architecture of Object Storage it is worth considering the perspectives of both the user and the provider of the object storage service.

  • User Perspective: The User will use the object storage service as a means to persist static content in locations remote to the application. Static content could include virtual machine images accessible only to the user. This is similar to the approach for storing customized virtual machines on Amazon EC2.
  • Provider Perspective: In terms of managing an object storage service the Provider must be able to accommodate very large deployments, and so scalability, redundancy and minimal maintenance are all of great importance. Functionality that must be considered includes
    • the automatic purging of stale data,
    • the automatic removal of deactivated accounts and their associated content,
    • the automatic replacement of corrupt data with a valid replica,
    • the automatic escalation of issues to an automated service that will attempt to resolve them. Should they not be resolved then notifications should be automatically sent to the Provider.
    • the automatic gathering of appropriate statistics, supporting both adhoc investigation and autonomous optimization of the system,
    • the ability to easily introduce additional storage capability and other hardware into the system, without interrupting the service.

The Provider, it should be stated, is also likely to assume the role of service user. Monitoring, Reporting and Auditing data are all very suitable for storing on an Object Storage system. The provider can provide access to this data to appropriate users. The Object Storage service could also be used as a virtual machine staging area. Customized virtual machines could be uploaded by end-users and stored in the object storage service.

The Provider will wish to distribute copies of the objects being stored to provide an appropriate level of redundancy. The distributed infrastructure employed means that the object storage service can become the backbone of a content distribution network.

Basic Concepts

Implementations of the Object Storage Generic Enabler (GE) should provide a highly available, distributed and eventually consistent object store. The object store is a collection of objects that are structured in a simple hierarchy. The object store presents itself as a service that has multi-tenant capabilities such that the service can be offered to many users and organizations with their data safely isolated from each other. The object storage service does not have a traditional POSIX-type file system. Instead, objects are stored in containers. Containers can be stored within containers, so simple hierarchies are supported. Access to the Object Storage GE is through a RESTful-style API. Largely for interoperability reasons the Cloud Data Management Interface (CDMI) specification has been adopted.

The key abstract entities identified that need to be considered by this GE are:

  • Object: opaque piece of data with associated meta-data. This directly maps to the concept of object within CDMI.
  • Container: collection of objects with associated meta-data. This directly maps to the concept of container within CDMI. A CDMI container can contain many other containers. A container cannot have more than one parent container.
  • Policy: meta data associated with an object or container that defines the management of the data by the object storage service provider. Policies will be expressed through the meta-data facilities offered by CDMI.
  • Account: a collection of containers assigned to a user. As the Object Storage GE uses the CDMI specification an Account maps directly to a top-level container.
  • User: the actor accessing and managing the above entities through the GE’s API. At a minimum the actors of end-user (human or external service) and administrator are considered. The security information related to User is managed by the Identity Management GE.

Access to and management of Object Storage entities is performed through the defined API. As standardized and open interfaces to GEs are a goal in FI-WARE GE specification, the Cloud Data Management Interface (CDMI) has been adopted as the API specification for the Object Storage GE.

The high-level internal architecture of the Object Storage GE is shown below:

The main elements in the functional block diagram are as follows:

  • Admin, User: These are Actors interacting with the service. Both might have different privileges associated to their accounts.
  • Identity Management GE: This is the entity that handles the privileges of the users.
  • API: this entity exposes the interface of the Object Storage GE and allows the administrators and users interact and manage their service instances.
  • Storage Management: this is the entity that manages the resources associated with a user’s service instance. Here entities such as containers, meta-data, objects and policies are managed.
  • Storage: these are the storage devices where the objects are physically persisted.

Cloud Data Management Interface (CDMI)

CDMI is the de-facto standard for manipulating data in the cloud. Developed by the Storage Networking Industry Association (SNIA), it has now been designated by the International Organization for Standardization (ISO) as a formal international standard.

CDMI considers the management of both cloud storage systems and those of enterprise systems. It implements the SNIA’s Storage Industry Resource Domain Model (SIRDM).

At the core of CDMI are the basic management operations of Create, Retrieve, Update and Delete. This functionality is exposed via a RESTful API that:

  • Enables clients to discover capabilities of the object storage offering
  • Manages containers and the objects that are placed within them
  • Assigns and manipulates metadata to containers and objects

Meta-data is core to enabling richer management of data within CDMI. Meta-data can be associated with both system-managed objects and of course user-managed objects. Indeed, according to the specification “metadata is interpreted by the cloud offering as data requirements that control the operation of underlying data services for that data.” This focus of CDMI makes it a very flexible and suitable specification for the Object Storage GE, especially as providing quality of service and experience is a core focus in FI-WARE. It is through CDMI’s meta-data facilities that various data handling policies and QoS parameters can be specified and how the abstract entity of Policy can be represented.

The core entities in the CDMI model are:

  • Capabilities: allow a client to discover features of the CDMI standard implemented by a provider. This is required for basic CDMI compliance.
  • Object: opaque piece of data with associated meta-data. This is required for basic CDMI compliance. This CDMI entity will allow for the representation of Object.
  • Container: collection of objects (and possibly containers) with associated meta-data. This is required for basic CDMI compliance. This CDMI entity will allow for the representation of Container and Account.
  • Domain: represents the concept of administrative ownership of stored data within a CDMI storage system. This is an optional entity and is not required to be implemented for basic CDMI compliance. For the first release of the Object Storage GE this feature will not be supported.
  • Queue: is a first-in, first-out (FIFO) access queue for storing and retrieving data. This is an optional entity and is not required to be implemented for basic CDMI compliance. For the first release of the Object Storage GE this feature will not be supported.

Main Interactions

For details on how a client interacts with the CDMI interface, please refer to the CDMI specification document. However below are two of the most basic but typical and perhaps important interactions with the interface.

Example: Create a Container

> PUT /mycontainer/ HTTP/1.1
> Host: os.fi-ware.eu

< HTTP/1.1 201 Created 

This creates a container named "mycontainer". The service responds with a HTTP 201, indicating success.

Example: Persisting an Object

> PUT /mycontainer/simpleobject.json HTTP/1.1  
> Host: os.fi-ware.eu  
> Content-Type: application/json  
> Content-Length: 29  
> 
> {"message": "Hello FIware!"}

< HTTP/1.1 201 Created 

This creates an object that contains JSON content (the supplied content is in the body of the HTTP PUT). The object is named "simpleobject.json" and is stored in the "mycontainer" container. The service responds with a HTTP 201, indicating success.

Please note CDMI is an evolving standard. At the time of writing the approved version is CDMI 1.0.2. The GE implementations define the precise version of CDMI that it supports.

References

  1. http://www.snia.org/cdmi
Personal tools
Create a book