We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Cloud.ObjectStorage R3 - FIWARE Forge Wiki

FIWARE.OpenSpecification.Cloud.ObjectStorage R3

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Cloud.ObjectStorage
Chapter Cloud,
Catalogue-Link to Implementation Object Storage
Owner IBM, Kalman Meth

Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.

FIWARE WIKI editorial remark:
This page corresponds to Release 3 of FIWARE. The latest version associated to the latest Release is linked from FIWARE Architecture

Overview

Object Storage is one of the Generic Enablers within FIWARE. It offers persistent storage for digital objects, important cloud-based functionality that has been specifically requested by Use Cases. Objects can be files, databases or other datasets which need to be archived. Objects are stored in named locations known as containers. Containers can be nested thus objects can be stored hierarchically.

Containers and objects can have Metadata associated with them, providing details of what the data represents. Similar to files in a traditional filesystem - objects in an Object store belong to a certain user (account).

The following sections provide more information on these topics. This Generic Enabler is based on OpenStack Swift.

The users of the Object Storage Generic Enabler include both FIWARE Cloud Instance Providers and FIWARE Cloud Instance Users.

  • Provider usage: Cloud Instance Providers can both provide Object Storage as a service to Cloud Instance Users, and consume the Object Storage service themselves. In terms of providing the service, the Cloud Instance Providers will require a system that demands as little maintenance as is possible. This entails that any:
    • stale data be purged,
    • deactivated accounts be removed,
    • corrupt data is replaced with a valid replica,
    • issues are escalated to an automated service that will attempt to resolve them (if they cannot be resolved then notifications to the Provider should be sent),
    • relevant statistics should be available to support inspection of the system and the User's utilisation of the system,
    • additional requirements for hardware (storage capability) can be easily added to the system without any drop in service. This will allow the storage capacity to grow over time.

In terms of consuming the service themselves, the Cloud Instance Providers will want to store certain types of data such as monitoring, reporting and auditing data to support their offering. This data can be made available to the Cloud Instance Users depending on requirements. The Object storage service can also be used as a virtual machine staging area. A Cloud Instance User may upload their custom virtual machine to the Object Store from which location the provider will make it available.

  • User usage: The User will use the object storage service as a means to distribute static content rather than incur the additional load of serving static content from an application. Taking this approach allows the Provider to optimize the distribution of those files. The Provider can also use this as a building block to offer further content distribution network capabilities. The User could also use the object storage service as a means to supply a customized virtual machine that only they have access to (the storage is isolated by user). This would operate in much the same way as how customized virtual machine images are supplied on services like Amazon EC2.

Basic Concepts

Implementations of the Object Storage Generic Enabler (GE) should provide a highly available, distributed and eventually consistent object store. The object store is a collection of objects that are structured in a simple hierarchy. The object store presents itself as a service that has multi-tenant capabilities such that the service can be offered to many users and organisations and that their data is safely partitioned. The object storage service does not have a traditional POSIX-type file system and as it has a simple hierarchical structure it really has little notion of true directory semantics.

The key abstract entities identified that need to be considered by this GE are:

  • Object: opaque piece of data with associated meta-data.
  • Container: collection of objects with associated meta-data.
  • Account: a collection of containers assigned to a tenant.
  • User: the actor accessing and managing the above entities through the GE’s API. At a minimum the actors of end-user (human or external service) and administrator are considered. The security information related to User is managed by the Identity Management GE.

Access to and management of Object Storage entities is performed through the defined API. As standardised and open interfaces to GEs is an important aspect to consider in the specification of all GEs, the OpenStack Swift API has been adopted as the API specification for the Object Storage GE.

OpenStack Swift Object Storage API overview

OpenStack Swift Object Storage API

Object Storage User and Programmers Guide

The overall internal architecture of the Object Storage GE is shown below:

[[

File:Objectstorage-fmc-block.png|center]]

The main elements in the functional block diagram are as follows:

  • Admin, User: These are Actors interacting with the service. Both might have different privileges associated to their accounts.
  • Identity Management GE: This is the entity which handles the privileges of the users.
  • API: this entity is what exposes the interface of the Object Storage GE and allows the administrator and user user-types interact and manage their service instances.
  • Storage Management: this is the entity that manages the resources associated with a user’s service instance. Here entities such as containers, meta-data, and objects are managed.
  • Storage: this is the storage device where the objects are physically persisted

Main Interactions

The OpenStack Swift API is a RESTful interface and all interactions are via well-known HTTP methods such as GET, PUT and DELETE. Some typical operations are introduced in this section.

For complete details, please refer to the OpenStack Swift Object Storage API.

Creating a Container

The following request and response illustrate how a container named "mycontainer" can be created. The service responds with a HTTP 201, indicating the container was created successfully.

Request:

> PUT $publicURL/mycontainer HTTP/1.1
> Host: os.fiware.eu
> X-Auth-Token: $token

Response:

< HTTP/1.1 201 Created

Persisting an Object

The following request and response illustrate how an object called "simpleobject" can be stored in container named "mycontainer". The service responds with a HTTP 201, indicating the object was created successfully. Various object metadata is also returned.

Request:

> PUT $publicURL/mycontainer/simpleobject HTTP/1.1
> Host: os.fiware.eu
> X-Auth-Token: $token  

Response:

< HTTP/1.1 201 Created

Retrieving an Object

The following request and response illustrate how an object called "simpleobject" can be retrieved from a container named "mycontainer". The service responds with a HTTP 200, indicating the object was retrieved successfully. Various object metadata and the content of the file are returned in the HTTP response body.

Request:

> GET $publicURL/mycontainer/simpleobject HTTP/1.1
> Host: os.fiware.eu
> X-Auth-Token: $token

Response:

< HTTP/1.1 200 OK

Storlets

Swift Storlets extend Swift with the capability to run computation near the data in a secure and isolated manner. With Swift Storlets a user can write code, package and deploy it as a Swift object, and then explicitly invoke it on data objects as if the code was part of the Swift pipeline.

Full details on Swift Storlets is available at https://github.com/openstack/storlets/blob/master/README.md.

Swift storlets are not yet available on the FIWARE testbed. The Swift Storlets package is available at https://github.com/openstack/storlets. An environment running storlets is set up on a machine made available by IBM for approved users. To request an account to experiment with storlets please contact: meth@il.ibm.com.


Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FIWARE level, please refer to FIWARE Global Terms and Definitions


  • Infrastructure as a Service (IaaS) -- a model of delivering general-purpose virtual machines (VMs) and associated resources (CPU, memory, disk space, network connectivity) on-demand, typically via a self-service interface and following a pay-per-use pricing model. The virtual machines can be directly accessed and used by the IaaS consumer (e.g., an application developer, an IT provider or a service provider), to easily deploy and manage arbitrary software stacks.
  • Platform as a Service (PaaS) -- an application delivery model in which the clients, typically application developers, follow a specific programming model to develop their applications and or application components and then deploy them in hosted runtime environments. This model enables fast development and deployment of new applications and components.
  • Project is a container of virtual infrastructure that has a set of virtual resources (e.g., computing capacities, storage capacities) to support the former. In other words, a VDC is a pool of virtual resources that supports the virtual infrastructure it contains.
  • Service Elasticity is the capability of the hosting infrastructure to scale a service up and down on demand. There are two types of elasticity -- vertical (typically of a single VM), implying the ability to add or remove resources to a running VM instance, and horizontal (typically of a clustered multi-VM service), implying the ability to add or remove instances to/from an application cluster, on-demand. Elasticity can be triggered manually by the user, or via an Auto-Scaling framework, providing the capability to define and enforce automated elasticity policies based on application-specific KPIs.
  • Service Level Agreement (SLA) is a legally binding contract between a service provider and a service consumer specifying terms and conditions of service provisioning and consumption. Specific SLA clauses, called Service Level Objectives (SLOs), define non-functional aspects of service provisioning such as performance, resiliency, high availability, security, maintenance, etc. SLA also specifies the agreed upon means for verifying SLA compliance, customer compensation plan that should be put in effect in case of SLA incompliance, and temporal framework that defines validity of the contract.
Personal tools
Create a book