We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.ArchitectureDescription.Cloud.ObjectStorage R3 CDMI - FIWARE Forge Wiki

FIWARE.ArchitectureDescription.Cloud.ObjectStorage R3 CDMI

From FIWARE Forge Wiki

Jump to: navigation, search

Contents

FIWARE WIKI editorial remark:
This page corresponds to Release 3 of FIWARE. The latest version associated to the latest Release is linked from FIWARE Architecture

Copyright

The FI-WARE ObjectStorage Specification is Copyright © 2013 INTEL. Please note that this specification adopts the CDMI standard, which is published by and copyright SNIA.

Legal Notice

Please check the following Legal Notice (essential patents license) to understand the rights to use these specifications.

Overview

Object Storage is one of the Generic Enablers within FI-WARE. It offers persistent storage for digital objects, important cloud-based functionality that has been specifically requested by Use Cases. Objects can be files, databases or other datasets which need to be archived. Objects are stored in named locations known as containers. Containers can be nested thus objects can be stored hierarchically.

Containers and objects can have Metadata associated with them, providing details of what the data represents. Similar to files in a traditional filesystem - objects in an Object store belong to a certain user (account).

The following sections provide more information on all of these topics and introduce the CDMI standard. Rather than develop an entirely new interface, this Generic Enabler is based on CDMI.

The users of the Object Storage Generic Enabler include both FI-WARE Cloud Instance Providers and FI-WARE Cloud Instance Users.

  • Provider usage: Cloud Instance Providers can both provide Object Storage as a service to Cloud Instance Users, and consume the Object Storage service themselves. In terms of providing the service, the Cloud Instance Providers will require a system that demands as little maintenance as is possible. This entails that any:
    • stale data be purged,
    • deactivated accounts be removed,
    • corrupt data is replaced with a valid replica,
    • Issues are escalated to an automated service that will attempt to resolve them (if they cannot be resolved then notifications to the Provider should be sent),
    • relevant statistics should be available to support inspection of the system and the User's utilisation of the system,
    • additional requirements for hardware (storage capability) can be easily added to the system without any drop in service. This will allow the storage capacity to grow over time.

In terms of consuming the service themselves, the Cloud Instance Providers will want to store certain types of data such as monitoring, reporting and auditing data to support their offering. This data can be made available to the Cloud Instance Users depending on requirements. The Object storage service can also be used as a virtual machine staging area. A Cloud Instance User may upload their custom virtual machine to the Object Store from which location the provider will make it available.

  • User usage: The User will use the object storage service as a means to distribute static content rather than incur the additional load of serving static content from an application. Taking this approach allows the Provider to optimize the distribution of those files. The Provider can also use this as a building block to offer further content distribution network capabilities. The User could also use the object storage service as a means to supply a customized virtual machine that only they have access to (the storage is isolated by user). This would operate in much the same way as how customized virtual machine images are supplied on services like Amazon EC2.

Basic Concepts

Implementations of the Object Storage Generic Enabler (GE) should provide a highly available, distributed and eventually consistent object store. The object store is a collection of objects that are structured in a simple hierarchy. The object store presents itself as a service that has multi-tenant capabilities such that the service can be offered to many users and organisations and that their data is safely partitioned. The object storage service does not have a traditional POSIX-type file system and as it has a simple hierarchical structure it really has little notion of true directory semantics. In fact, the Object Storage service allows for levels of hierarchy, where a container can reside within another container so allowing for more flexible organisation of data. Notably the Object Storage GE adopts the Cloud Data Management Interface (CDMI) specification.

The key abstract entities identified that need to be considered by this GE are:

  • Object: opaque piece of data with associated meta-data. This directly maps to the CDMI object.
  • Container: collection of objects with associated meta-data. This directly maps to the CDMI container. In CDMI, a container can contain many other containers. A container cannot have more than one parent container.
  • Policy: meta data associated with an object or container that dictates the use of the data by the object storage service provider. Policies will be expressed through the meta-data facilities offered by CDMI.
  • Account: a collection of containers assigned to a user. As the Object Storage GE uses the CDMI specification the concept of Account will map to the highest level of container.
  • User: the actor accessing and managing the above entities through the GE’s API. At a minimum the actors of end-user (human or external service) and administrator are considered. The security information related to User is managed by the Identity Management GE.

Access to and management of Object Storage entities is performed through the defined API. As standardised and open interfaces to GEs is an important aspect to consider in the specification of all GEs, the Cloud Data Management Interface (CDMI) has been adopted as the API specification for the Object Storage GE.

The overall internal architecture of the Object Storage GE is shown below:

[[

File:Objectstorage-fmc-block.png|center]]

The main elements in the functional block diagram are as follows:

  • Admin, User: These are Actors interacting with the service. Both might have different privileges associated to their accounts.
  • Identity Management GE: This is the entity which handles the privileges of the users.
  • API: this entity is what exposes the interface of the Object Storage GE and allows the administrator and user user-types interact and manage their service instances.
  • Storage Management: this is the entity that manages the resources associated with a user’s service instance. Here entities such as containers, meta-data, objects and policies are managed.
  • Storage: this is the storage device where the objects are physically persisted

Cloud Data Management Interface (CDMI)

CDMI is the de-facto standard for manipulating data in the cloud. Developed by the Storage Networking Industry Association (SNIA), it has now been designated by ISO as a formal international standard.

CDMI considers the management of both cloud storage systems and those of enterprise systems. It implements the SNIA’s Storage Industry Resource Domain Model (SIRDM).

At the core of CDMI are the basic management operations of Create, Retrieve, Update and Delete. This functionality is exposed via a RESTful API that:

  • Enables clients to discover capabilities of the object storage offering
  • Manage containers and the objects that are placed within them
  • Assigns and manipulates metadata to containers and objects

Meta-data is core to enabling richer management of data within CDMI. Meta-data can be associated with both system-managed objects and of course user-managed objects. Indeed, according to the specification “metadata is interpreted by the cloud offering as data requirements that control the operation of underlying data services for that data.” This focus of CDMI makes it as a very flexible and suitable specification for the Object Storage GE, especially as providing quality of service and experience is a core focus in FI-ware. It is through CDMI’s meta-data facilities that various data handling policies and QoS parameters can be specified and how the abstract entity of Policy can be represented.

The core entities in the CDMI model are:

  • Capabilities: allows a client to discover features of the CDMI standard implemented by a provider. This is required for basic CDMI compliance.
  • Object: opaque piece of data with associated meta-data. This is required for basic CDMI compliance. This CDMI entity will allow for the representation of Object.
  • Container: collection of objects with associated meta-data. This is required for basic CDMI compliance. This CDMI entity will allow for the representation of Container and Account.
  • Domain: represent the concept of administrative ownership of stored data within a CDMI storage system. This is an optional entity and is not required to be implemented for basic CDMI compliance. For the first release of the Object Storage GE this feature will not be supported.
  • Queue: is a first-in, first-out (FIFO) access when storing and retrieving data. This is an optional entity and is not required to be implemented for basic CDMI compliance. For the first release of the Object Storage GE this feature will not be supported.

Main Interactions

CDMI is a RESTful interface and all interactions are via well-known HTTP methods such as GET, PUT and DELETE. Some typical CDMI operations are introduced in this section. For complete details on how a client interacts with the CDMI interface, please refer to the CDMI specification document.

Creating a Container

The following request and response illustrate how a container named "mycontainer" can be created. The service responds with a HTTP 201, indicating the container was created successfully.

Request:

> PUT /mycontainer/HTTP/1.1
> Host: os.fi-ware.eu
> Accept: application/cdmi-container
> Content-Type: application/cdmi-container
> X-CDMI-Specification-Version: 1.0.1

Response:

< HTTP/1.1 201 Created
< Content-Type: application/cdmi-container
< X-CDMI-Specification-Version: 1.0.1

Persisting an Object

The following request and response illustrate how an object called "simpleobject.json" containing the string "Hello CDMI World!" can be stored in container named "mycontainer". The service responds with a HTTP 201, indicating the object was created successfully. Various object metadata is also returned.

Request:

> PUT /mycontainer/simpleobject.json HTTP/1.1
> Host: os.fi-ware.eu
> Accept: application/cdmi-object
> Content-Type: application/cdmi-object
> X-CDMI-Specification-Version: 1.0.1
> {
>     "mimetype" : "text/plain",
>     "metadata" : {
>     },
>     "value" : "Hello CDMI World!"
> }   

Response:

< HTTP/1.1 201 Created
< Content-Type: application/cdmi-object
< X-CDMI-Specification-Version: 1.0.1
< {
<     "objectType" : "application/cdmi-object",
<     "objectID" : "00007E7F0010BD1CB8FF1823CF05BEE4",
<     "objectName" : "simpleobject.json",
<     "parentURI" : "/mycontainer/",
<     "parentID" : "00007E7F00102E230ED82694DAA975D2",
<     "domainURI" : "/cdmi_domains/MyDomain/",
<     "capabilitiesURI" : "/cdmi_capabilities/dataobject/",
<     "completionStatus" : "Complete",
<     "mimetype" : "text/plain",
<     "metadata" : {
<         "cdmi_size" : "17"
<     }
< }   

Retrieving an Object

The following request and response illustrate how an object called "simpleobject.json" can be retrieved from a container named "mycontainer". The service responds with a HTTP 200, indicating the object was retrieved successfully. Various object metadata and the content of the file are returned in the HTTP response body.

Request:

> GET /mycontainer/simpleobject.json HTTP/1.1
> Host: os.fi-ware.eu
> Accept: application/cdmi-object
> X-CDMI-Specification-Version: 1.0.1

Response:

< HTTP/1.1 200 OK
< Content-Type: application/cdmi-object
< X-CDMI-Specification-Version: 1.0.1
< {
<     "objectType": "application/cdmi-object",
<     "objectID": "00007E7F0010BD1CB8FF1823CF05BEE4",
<     "objectName": "simpleobject.json",
<     "parentURI": "/mycontainer/",
<     "parentID" : "00007E7F00102E230ED82694DAA975D2",
<     "domainURI": "/cdmi_domains/MyDomain/",
<     "capabilitiesURI": "/cdmi_capabilities/dataobject/",
<     "completionStatus": "Complete",
<     "mimetype": "text/plain",
<     "metadata": { "cdmi_size": "17"
<     },
<     "valuetransferencoding": "utf-8",
<     "valuerange": "0-16",
<     "value": "Hello CDMI World!“
< }   
Personal tools
Create a book