We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Object Storage - User and Programmers Guide - FIWARE Forge Wiki

Object Storage - User and Programmers Guide

From FIWARE Forge Wiki

Jump to: navigation, search

Contents

Introduction

Welcome to the User and Programmer Guide for the Object Storage Generic Enabler. This generic enabler is built on an Open Source project, the OpenStack Swift API, and so where possible this guide points to the appropriate online content that has been created for this project. The online documents are being continuously updated and improved, and so will be the most appropriate place to get the most up to date information on installation and administration.

Background and Detail

This User and Programmers Guide relates to the Object Storage GE which is part of the Cloud Hosting chapter. Please find more information about this Generic Enabler in the following Open Specification.

User Guide - Accessing Object Storage via the Cloud Portal

Accessing the object storage via the cloud portal is obsolete.

Programmer Guide - Accessing Object Storage via the native API

Introduction

The Object Storage Generic Enabler exposes Object Storage functionality via a standard API: OpenStack Swift.

The complete OpenStack Swift Object Storage API specification is freely available online [1].

OpenStack Swift is a RESTful API and so just like OCCI it is built on well-known HTTP operations such as PUT, POST, GET and DELETE. For example, to retrieve a data object from a Swift container a simple HTTP GET command is used.

For example, the following request:

GET /MyContainer/MyDataObject.txt HTTP/1.1
Host: cloud.example.com
X-Auth-Token: 0cc2bab32f3246919c2d2cbea314d850

will generally return a response similar to:

HTTP/1.1 200 OK
{
    content of object
}

HTTP commands can be constructed and sent from all modern programming languages. They can also be invoked using a command line utility such as curl[2].

Authentication

A valid token is required to access an object store. This section describes how to get a valid token on the FI-LAB infrastructure. Note that the token retrieval procedure described is essentially OpenStack Keystone (v2 protocol). If the username, password and tenant details are known, only step 3 is required. If only the username and password are known then all three steps must be executed in order.

The examples in this section point to the identity management system on the FI-LAB infrastructure.

1. Authentication to get initial token

The response to the following command includes an initial token.

curl -d '{"auth": {"passwordCredentials": {"username":"email@company.com", "password":"mypassword"}}}' \
    -H 'Content-type: application/json' \
    http://cloud.lab.fiware.org:4730/v2.0/tokens \
    -vvv

The following bash script illustrates how the initial token can be retrieved and stored in a variable $token1.

username='email@company.com'
password='mypassword'
curl -d '{"auth": {"passwordCredentials": {"username":"'$username'", "password":"'$password'"}}}' \
    -H 'Content-type: application/json' \
    http://cloud.lab.fiware.org:4730/v2.0/tokens \
    > auth_token1.dat
token1=$(awk -F"[,:]" '{for(i=1;i<=NF;i++)
                       {if($i~/id\042/)
                         {print $(i+1)} 
                       }
                     }' auth_token1.dat | awk -F'"' '{print $2; exit}')

In python this can be achieved as follows.

   import httplib
   import json
   username='email@company.com'
   password='mypassword'
   conn = httplib.HTTPConnection('cloud.lab.fi-ware.org:4730')
   headers = {'Content-Type': 'application/json'}
   body = '{"auth": {"passwordCredentials":{"username": "'+username+'", "password": "'+password+'"}}}'
   conn.request("POST", "/v2.0/tokens", body, headers)
   response = conn.getresponse()
   data = response.read()
   datajson = json.loads(data)
   token1 = datajson['access']['token']['id']

2. Use initial token to get tenant

The response to the following command includes tenant details.

curl -H 'x-auth-token: '$token1 \
    http://cloud.lab.fiware.org:4730/v2.0/tenants \
    -vvv

The following bash script illustrates how the tenant details can be retrieved and stored in a variable $tenant. The script assumes a variable $token1 includes a valid token - as per step one.

curl -H 'x-auth-token: '$token1 \
      http://cloud.lab.fiware.org:4730/v2.0/tenants \
      > auth_tenant.dat
tenant=$(awk -F"[,:]" '{for(i=1;i<=NF;i++)
                       {if($i~/id\042/)
                         {print $(i+1)} 
                       }
                     }' auth_tenant.dat | awk -F'"' '{print $2; exit}')

In python this can be achieved as follows. The code assumes token1 includes a valid token - as per step one.

   headers = {'x-auth-token': token1}
   conn.request("GET", "/v2.0/tenants", None, headers)
   response = conn.getresponse()
   data = response.read()
   datajson = json.loads(data)
   tenant = datajson['tenants'][0]['id']

Instead of tenant Id, it is also possible to extract tenant Name from the "name" field of the returned data.

3. Authenticate tenant to get token for Object Storage

The response to the following command includes the token to access the object store, and the string necessary to refer to the user's object storage space.

curl -d '{"auth": {"passwordCredentials": {"username":"email@company.com", "password":"mypassword"},"tenantId":"00000000000000000000000000000150"}}' \
    -H 'Content-type: application/json' \
    http://cloud.lab.fiware.org:4730/v2.0/tokens \
    -vvv

Note that the above example uses the tenant id. In order to use the tenant name, use the following command:

curl -d '{"auth": {"passwordCredentials": {"username":"email@company.com", "password":"mypassword"},"tenantName":"mytenantname"}}' \
    -H 'Content-type: application/json' \
    http://cloud.lab.fiware.org:4730/v2.0/tokens \
    -vvv

The tenant name can be obtained from the cloud portal. Login to https://cloud.lab.fiware.org and follow the "Organizations" link. The organization name is the required tenant name. Typically it is the user name concatenated with the string " cloud". That is "<username> cloud".

The response body of the above request is of a json format having the following structure:

{ 
   "access": {
       "metadata": {
           <metadata>
       },
       "serviceCatalog": [
           {
               "name": "<service name>",
               "endpoints": [
                   {
                       "adminURL": "<service admin endpoint URL>",
                       "internalURL": "<service internal endpoint URL>",
                       "publicURL": "<service public endpoint URL>",
                       "id": "<endpoint id>",
                       "region": "<The region this endpoint belong to>"
                   },
                   ... (more endpoints here, each having a different region)
               ],
           },
           ... (more services here, each having another name)
       ],
       "token": {
           "expires": "<expiration date>",
           "id": "<token id>",
           ... (more token metadata)
       },
       "user": {
           ... (some metadata on the user) 
       }
   }
}

As you can see the token is made of 3 major parts: the "service catalog", the "token" and the "user". The service catalog lists all the regions for each service (and the endpoints of each region). For the sake of accessing the object store we are interested the service having the name "swift". Each region appearing in the "swift" service is essentially a different instance of a Swift cluster. It is important to understand that each such cluster is a different namespace having its own set of containers and objects and that there is no replication between clusters. Once picking a region use the PublicURL entry as the URL for the REST request to Swift. Each such rest request to Swift needs to be accompanied with the token id, this is the "id" field of the "token" part.

Note: Not all regions are active at all times, and some may be down. Thus, one needs to look for an active region. The python example below picks the first region in the list, which is not necessarily active.

The following bash script illustrates how the required token and auth string can be retrieved and stored in a variables $token and $auth. The script assumes variables $username, $password and $tenantName have been initialized as per step one and two.

You will also need to extract one of the fields that are labeled 'publicURL' under the 'swift' 'endpoints'. See python example.

curl -v \
    -d '{ "auth" : 
          { "passwordCredentials" : 
            { "username" : "'$username'" , "password" : "'$password'" },
          "tenantId" : "'$tenant'" } 
        }' \
    -H "Content-Type: application/json" \
     http://cloud.lab.fiware.org:4730/v2.0/tokens \
     > auth_token2.dat
token=$(awk -F"[,:]" '{for(i=1;i<=NF;i++)
                       {if($i~/id\042/)
                         {print $(i+1)} 
                       }
                     }' auth_token2.dat | awk -F'"' '{print $2; exit}')

In python this can be achieved as follows. The code assumes tenant is a valid tenant - as per step two.

   headers = {'Content-Type': 'application/json'}
   body = '{"auth": {"tenantId": "'+tenant+'", "passwordCredentials":{"username": "'+username+'", "password": "'+password+'"}}}'
   conn.request("POST", "/v2.0/tokens", body, headers)
   response = conn.getresponse()
   data = response.read()
   datajson = json.loads(data)
   token = datajson['access']['token']['id']
   for i in auth_reponse['access']['serviceCatalog']:
       if i['name'] == 'swift':
           auth_url = i['endpoints'][0]['publicURL']
           break

Example Python

The following python code illustrates some basic operations against object storage.

   #!/usr/bin/env python
   
   Example python to authenticate against an object store, create a new container, store text object, retrieve it, delete object, delete container.
   
   
   import httplib
   import json
   import sys
   import time
   import os
   import logging
   
   # Init a simple logger...
   logging.basicConfig(level=logging.INFO)
   console = logging.StreamHandler()
   console.setLevel(logging.DEBUG)
   logger = logging.getLogger()
   logger.addHandler(console)
   
   # hosts
   HOST_AUTH = 'cloud.lab.fiware.org:4730'
   
   TEST_CONTAINER_NAME = 'TestContainerPython'
   TEST_OBJECT_NAME = 'TestObjectPython.txt'
   TEST_TEXT = 'Hello SWIFT World'
   
   def authentication_request(username, password):
       
       Request authentication of user
       
       conn = httplib.HTTPConnection(HOST_AUTH)
   
       # retrieve initial token
       headers = {'Content-Type': 'application/json'}
       body = '{"auth": {"passwordCredentials":{"username": "'+username+'", "password": "'+password+'"}}}'
       conn.request("POST", "/v2.0/tokens", body, headers)
       response = conn.getresponse()
       data = response.read()
       datajson = json.loads(data)
       initialtoken = datajson['access']['token']['id']
   
       logger.info('Initial Token is: ' + initialtoken)
   
       # retrieve tenant
       headers = {'x-auth-token': initialtoken}
       conn.request("GET", "/v2.0/tenants", None, headers)
       response = conn.getresponse()
       data = response.read()
       datajson = json.loads(data)
       tenant = datajson['tenants'][0]['id']
   
       logger.info('Tenant is: ' + tenant)
   
       # retrieve authentication json
       headers = {'Content-Type': 'application/json'}
       body = '{"auth": {"tenantId": "'+tenant+'", "passwordCredentials":{"username": "'+username+'", "password": "'+password+'"}}}'
       conn.request("POST", "/v2.0/tokens", body, headers)
       response = conn.getresponse()
       data = response.read()
   
       return json.loads(data)
   
   
   def swift_request(verb, url, headers, body):
       
       Do a HTTP request defined by HTTP verb, a Url, a dict of headers and a body.
       
       logger.info('swift_request verb is: ' + verb)
       substring = url[url.find("//")+2:]
       marker = substring.find("/")
       host = substring[:marker]
       resource = substring[marker:]
       logger.info('host is: ' + host)
       logger.info('resource is: ' + resource)
       conn = httplib.HTTPConnection(host)
       conn.request(verb, resource, body, headers)
       response = conn.getresponse()
   
       if response.status not in [200, 201, 202, 204]:
           logger.error(response.reason)
           logger.warn(response.read())
           sys.exit(1)
   
       result = "Status: " + str(response.status) + ", Reason: " + response.reason + ", Body: " +  response.read()   
   
       conn.close()
   
       return result
   
   
   def create_container(token, auth, name):
       headers = {"X-Auth-Token": token}
       body = None
       url = auth + "/" + name
   
       return swift_request('PUT', url, headers, body)
   
   
   def list_container(token, auth, name):
       headers = {"X-Auth-Token": token}
       body = None
       url = auth + "/" + name
   
       return swift_request('GET', url, headers, body)
   
   
   def store_text(token, auth, container_name, object_name, object_text):
       headers = {"X-Auth-Token": token}
       body = object_text
       url = auth + "/" + container_name + "/" + object_name
   
       return swift_request('PUT', url, headers, body)
   
   
   def retrieve_text(token, auth, container_name, object_name):
       headers = {"X-Auth-Token": token}
       body = None
       url = auth + "/" + container_name + "/" + object_name
   
       return swift_request('GET', url, headers, body)
   
   
   def delete_object(token, auth, container_name, object_name):
       headers = {"X-Auth-Token": token}
       body = None
       url = auth + "/" + container_name + "/" + object_name
   
       return swift_request('DELETE', url, headers, body)
   
   
   def delete_container(token, auth, container_name):
       headers = {"X-Auth-Token": token}
       body = None
       url = auth + "/" + container_name
   
       return swift_request('DELETE', url, headers, body)
   
   
   if __name__ == '__main__':
       if len(sys.argv) < 3:
           print 'Usage: example.py <username> <password>'
           sys.exit(128)
   
       username = sys.argv[1]
       password = sys.argv[2]
       logger.info('username is: ' + username)
       logger.info('password is: ' + password)
   
       # display basic info
       logger.info('Authorisation host is: ' + HOST_AUTH)
   
       # get authentication response
       auth_response = authentication_request(username, password)
   
       # extract token
       token = auth_response['access']['token']['id']
       logger.info('Security token is: ' + token)
   
       # extract authentication string required for addressing users resources
       for i in auth_response['access']['serviceCatalog']:
       if i['name'] == 'swift':
                   # may take here any of the available entries that works
           auth_url = i['endpoints'][1]['publicURL']
            break
   
       logger.info('auth_url is: ' + auth_url)
   
       # perform some basic Object Store operations
   
       response = create_container(token, auth_url, TEST_CONTAINER_NAME)
       logger.info('Create Container Response: ' + response)
   
       response = list_container(token, auth_url, TEST_CONTAINER_NAME)
       logger.info('List Container Response: ' + response)
   
       response = store_text(token, auth_url, TEST_CONTAINER_NAME, TEST_OBJECT_NAME, TEST_TEXT)
       logger.info('Store Text Response: ' + response)
   
       response = list_container(token, auth_url, TEST_CONTAINER_NAME)
       logger.info('List Container Response: ' + response)
   
       response = retrieve_text(token, auth_url, TEST_CONTAINER_NAME, TEST_OBJECT_NAME)
       logger.info('Retrieve Text Response: ' + response)
   
       response = delete_object(token, auth_url, TEST_CONTAINER_NAME, TEST_OBJECT_NAME)
       logger.info('Delete Object Response: ' + response)
   
       response = list_container(token, auth_url, TEST_CONTAINER_NAME)
       logger.info('List Container Response: ' + response)
   
       response = delete_container(token, auth_url, TEST_CONTAINER_NAME)
       logger.info('Delete Container Response: ' + response)

Storlets

Swift Storlets extend Swift with the capability to run computation near the data in a secure and isolated manner. With Swift Storlets a user can write code, package and deploy it as a Swift object, and then explicitly invoke it on data objects as if the code was part of the Swift pipeline.

The User and Programmer Guide for Storlets is available at http://storlets.readthedocs.org/en/latest/writing_and_deploying_storlets.html.

Full details on Swift Storlets is available at https://github.com/openstack/storlets/blob/master/README.md.

Swift storlets are not yet available on the FIWARE testbed. The Swift Storlets package is available at https://github.com/openstack/storlets. An environment running storlets is set up on a machine made available by IBM for approved users. To request an account to experiment with storlets please contact: meth@il.ibm.com.

References

  1. http://developer.openstack.org/api-ref-objectstorage-v1.html
  2. http://curl.haxx.se/
Personal tools
Create a book