We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Cloud.Docker - FIWARE Forge Wiki

FIWARE.OpenSpecification.Cloud.Docker

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Cloud.Docker
Chapter Cloud,
Catalogue-Link to Implementation Docker GE - FIWARE Reference Implementation
Owner IBM, Kenneth Nagin

If you need to check out the Open Specs of the previous FIWARE Release (Release 4) you can go to FIWARE.OpenSpecification.Cloud.Docker R4

Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


Copyright

Copyright © 2015 by IBM

Legal notice

Please check the Legal Notice to understand the rights to use these specifications.

Overview

In order to support the strategic decision to adopt Docker as the standard application delivery mechanism in FIWARE, we introduce the Docker GE, providing multi-tenant hosting capabilities for Docker containers, networks, and volumes. Docker is the new de-facto standard for packaging, running and managing modern cloud applications, allowing to radically improve the developer experience (portability, lightweight runtime, DevOps enablement). While Docker is developed by a rapidly growing open source community and is triggering creation of its own commercial ecosystem, the technology itself is not fully mature yet. In order to accelerate its maturity and readiness for production use in FIWARE, we are investing effort to address requirements targeting commercial FIWARE providers who would need to efficiently operate large-scale multi-tenant Docker-based environments. In parallel, we are providing a simplified solution that can be used by developers already, on existing FIWARE Lab infrastructure (leveraging assets from the FI-Content 2 project).

While users can use the Cloud Portal GE to interact with their docker hosts, the Docker GE also ensures that they can remotely create and manage their docker hosts, clusters and containers from their local docker clients.

The Docker GE shows how developers can leverage Docker Machine to deploy and manage docker hosts on the FIWARE lab.

More importantly, the Docker GE has implemented and deployed a service that relieves users from managing their own Docker hosts. Instead it introduces the FIWARE Docker Container Service (FDCS) which allows users to focus solely on managing their containers in a secure environment. In addition to exposing the regular Docker capabilities FDCS provides authorization and authentication capabilities which allows it to support tenant isolation between different FIWARE accounts. The multi-tenant Docker support is integrated with FIWARE Keystone service and its existing accounts definitions. The service also provides support for name scoping between tenants such that tenants can use the same docker resource names without interfering with each other.

FDCS will simplify the usage of docker to develop FIWARE applications since tenant administrators will be relieved of the task of managing their docker hosts; the docker hosts will be managed by the FIWARE Lab infrastructure providers. Further more since the docker hosts and related resources will be shared by multiple tenants this will allow the FIWARE lab to achieve greater density of containers per host.

The main capabilities provided by FDCS are:

  • Multi-tenancy (support isolation between Contains of different accounts)
  • Name scoping (the same docker resource name can be used between tenants without interfering with each other )
  • Manage life cycle Docker Hosts
  • Manage network and storage attached Docker Hosts
  • Resource monitoring of Docker Hosts
  • Resiliency of the persistent data associated with Docker Hosts
  • Manage resource allocation
  • Secure access to the Docker Hosts
  • Secure access to the Docker Containers

FDCS is deployed as on the FIWARE Lab so users can get its benefits by simply applying for admission to the service. Alternatively, FDCS can be installed on-premises so a site can deploy docker containers on their private FDCS cluster.

Docker Main Components

Docker images: Docker images are the basis of Docker containers. A Docker image is an ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime. An image typically contains a union of layered filesystems stacked on top of each other. An image does not have state and it never changes. For example, IDM is the FIWARE Identity Manager. Its GUI is illustrated on the FIWARE Lab login page. The IDM Docker image is based on ubuntu:14.04 with lots of additional libraries and the IDM specific software installed. Images are used to create Docker containers. Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created. Docker clients generate docker images using the docker build command. The docker build command takes its input from a Dockerfile.

Dockerfile: A Dockerfile is a text document that contains all the commands you would normally execute manually in order to build a Docker image. Docker can build images automatically by reading the instructions from a Dockerfile. For an example see IDM's Dockerfile.

Docker registries: Docker registries hold images. These are public or private stores from which you upload or download images. The public Docker registry is provided with the Docker Hub. These can be images you build yourself or you can use images that others have previously created. Docker clients push images from their Docker Host to a Docker registry or pull them from a registry to a Docker Host. Docker clients can use the image to create and run Docker containers on a Docker Host; if the image does not already reside on the Docker Host it will automatically pull the image to itself.

Docker Hub: The Docker Hub is a centralized resource for working with Docker and its components. It provides the following services:

  • Docker image hosting
  • User authentication
  • Automated image builds and work-flow tools such as build triggers and web hooks
  • Integration with GitHub and Bitbucket

Docker Hub containers a huge collection of existing images for your use. Most of the FIWARE GEs have Docker Images in Docker Hub. These FIWARE GE images can be found at Docker Hub FIWARE. Of course, you can push your own docker images to Docker Hub to share with others.

Docker containers: A container is a runtime instance of a docker image. Containers running on a Docker Host share the same operating system kernel; they start instantly and use less RAM. Images are constructed from layered filesystems and share common files, making disk usage and image downloads much more efficient.

A Docker container holds everything that is needed for an application to run. Each container is created from a Docker image. Each container is an isolated and secure application platform. Docker clients manage their Docker containers on a Docker host. Examples of docker container operations are create, run, started, stop, start, delete, etc.

Docker Hosts: A Docker Host is a docker daemon process that is a lightweight runtime environment which manages docker resources, i.e. containers, networks, volumes, and images.

Docker clients: Docker clients allow users to manage their docker resources on a Docker Host. The clients interact with the Docker Host using the Docker Remote API. Examples of Docker clients are Docker CLI and Docker Compose.

Docker Swarm: Swarm is a native clustering tool for Docker. Swarm pools together several Docker hosts and exposes them as a single virtual Docker host. It serves the standard Docker API, so any tool that already works with Docker can now transparently scale up to multiple hosts. The FIWARE Docker Container Service (FDCS) is an enhanced version of Swarm that support Multi-Tenant Isolation and name scoping.


Docker’s architecture
Docker architecture

Docker Management Basic features

The main task you will perform on a Docker service is launching containers. This is done using the "run" command. Launching a container is simple as docker run + the image name you would like to run + the command to run within the container. If the image doesn’t exist on your local machine, Docker will attempt to fetch it from the public image registry. It’s important to note that containers are designed to stop once the command executed within them has exited. For example, if you ran /bin/echo hello world as your command, the container will start, print hello world and then stop:

docker run ubuntu /bin/echo hello world

However, docker containers typically run daemon services in the background. In this example we launch a container based on busybox. Upon entry busybox launches its http daemon, httpd, and enters its shell. httpd listens for connections on its port 80.

docker run -itd -p 2800:80 --name=httpd busybox sh -c "httpd,sh"

The -t and -i flags allocate a pseudo-tty and keep stdin open even if not attached. The -d flag runs the container in the background so it returns to the prompt. If we did not include the -d flag then we would enter the container's sh shell. The -p flag maps a host port to the container port, so in this example we could connect the container's httpd on the host port 2800. If we did not include the host port, i.e. -p 80, then docker auto assigns a port. The --name flag assigns a name to the container, i.e. httpd. busybox is the name of the Docker image. sh -c "httpd,sh" are the set of commands issued once the container is launched.

Some other basics Docker commands that you can use with this GE are:

  • docker ps: Lists containers.
  • docker port: displays s container's host port mapping
  • docker rm: delete a container
  • docker logs: Shows us the standard output of a container.
  • docker stop: Stops running containers.
  • docker start: Starts running containers.
  • docker attach: Attach to a running containers.

See (https://docs.docker.com/engine/reference/commandline Docker CLI) for complete list of commands.

Basic Concepts

Docker allows you to package an application with all of its dependencies into a standardized unit for software development. Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

As a user of the Docker GE you'll be able to leverage the unique Docker capabilities and benefits. Some of the typical use-cases that can take advantage of those capabilities are:

Code Pipeline Management: As the code travels from the developer’s machine to production, there are many different environments it has to go through to get there. Each of these may have minor differences along the way. Docker provides a consistent environment for the application from dev through production, easing the code development and deployment pipeline. The immutable nature of Docker images, and the ease with which they can be spun up, help you achieve zero change in application runtime environments across dev through production.

App Isolation: If for example, you need to run two REST API servers, but each of them uses a slightly different version of some library (e.g., flask) and other such dependencies. Running these API servers under different containers provides an easy way out through the “dependency hell.”

Server Consolidation: Similar to using VMs for consolidating multiple applications, the application isolation abilities of Docker allows consolidating multiple servers to save on cost. However, without the memory footprint of multiple OSes and the ability to share unused memory across the instances, Docker provides far denser server consolidation than you can get with VMs.

Rapid Deployment: We already know that VMs brought the deployment time down to minutes. Docker, by creating just a container for the process and not booting up an OS, brings it down to seconds and milli-seconds. Essentially, you can create and destroy resources in your data center without worrying about the cost of bringing it up again. Moreover, the immutable nature of Docker images gives you the peace of mind that things will work exactly the way they have been working and are supposed to work.

Main Interactions

The interaction with the Docker GE is done through the Docker Rest API. Docker API is a RESTful interface and all interactions are via well-known HTTP methods such as GET, PUT and DELETE. For complete details on how a client interacts with the Docker API, please refer to Docker Remote API

The Docker Remote API is used by docker clients to interact with the FIWARE Docker Container Service. The only difference is that two additional headers are included in the docker REST request, namely

    "X-Auth-Token": <keystone token id>, "X-Auth-TenantId": <keystone tenant id>

When the user's docker client is the docker cli, they should update the docker config.json to contain the headers:


    { "HttpHeaders":
      { "X-Auth-Token": <keystone token id>,
        "X-Auth-TenantId": <keystone tenant id>
      }
    }


When the Service receives a docker request it will contact FIWARE's Keystone Identity Manager to authorize the tenant's request based on the headers' token and tenant id. The figure below illustrates this process. In step 1 the user sends an Keystone Authenticate request to Keystone describing a FIWARE user which is a member of a tenant. In step 2,3 Keystone generates a token and returns it to the user. The user then updates their docker config.json file with the Keystone token and tenant id. The use then invokes a docker cli command which transparently sends docker REST requests (including the token and tenant id) to the Service. In step 6 the Service's multi-tenant swarm sends a Keystone list tenants request to Keystone which returns a list of tenants to which the token is authorized to access. In step 8 the Service's multi-tenant swarm checks whether the user's tenant id is in tenant list. Once the the tenant has been authorized swarm ensures that the tenants are isolated from each other by ensuring that each tenant can only manage and link to their own containers, volumes, and networks.


File:Multi-tenant_Swarm_with_Keystone_Authentication.png

Docker Container Service User Guide describes in more detail how the docker client may interact with the service.


Basic Design Principles

Docker implements a high-level API to provide lightweight containers that run processes in isolation. It's basic design principles are described succinctly in Docker's wikipedia page.

The Docker GE exposes the Docker API and thus allows any FIWARE Tenant with a docker client to remotely create and manage containers. In the illustration below we show a local Docker client remotely managing its Docker containers from a work station using the service. Most of the communication between the client and the service is through the Docker REST API. But the client must communicate with FIWARE’s Openstack Keystone Identity Management to get a valid token to interact with the service. Once a valid token is obtained the client can use the docker cli or docker compose to create and deploy complex Docker services.


In the illustration below we show a FIWARE Docker Container Service's cluster of docker hosts. Three FIWARE tenants are sharing the cluster's resources, but the service ensures that tenants are isolated from each other.


Re-utilised Technologies/Specifications

The Docker GE relies on the following specifications:

  • HTTP
  • TLS
  • JSON-RPC V2.0

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FIWARE level, please refer to FIWARE Global Terms and Definitions

  • Infrastructure as a Service (IaaS) -- a model of delivering general-purpose virtual machines (VMs) and associated resources (CPU, memory, disk space, network connectivity) on-demand, typically via a self-service interface and following a pay-per-use pricing model. The virtual machines can be directly accessed and used by the IaaS consumer (e.g., an application developer, an IT provider or a service provider), to easily deploy and manage arbitrary software stacks.
  • Platform as a Service (PaaS) -- an application delivery model in which the clients, typically application developers, follow a specific programming model to develop their applications and or application components and then deploy them in hosted runtime environments. This model enables fast development and deployment of new applications and components.
  • Project is a container of virtual infrastructure that has a set of virtual resources (e.g., computing capacities, storage capacities) to support the former. In other words, a VDC is a pool of virtual resources that supports the virtual infrastructure it contains.
  • Service Elasticity is the capability of the hosting infrastructure to scale a service up and down on demand. There are two types of elasticity -- vertical (typically of a single VM), implying the ability to add or remove resources to a running VM instance, and horizontal (typically of a clustered multi-VM service), implying the ability to add or remove instances to/from an application cluster, on-demand. Elasticity can be triggered manually by the user, or via an Auto-Scaling framework, providing the capability to define and enforce automated elasticity policies based on application-specific KPIs.
  • Service Level Agreement (SLA) is a legally binding contract between a service provider and a service consumer specifying terms and conditions of service provisioning and consumption. Specific SLA clauses, called Service Level Objectives (SLOs), define non-functional aspects of service provisioning such as performance, resiliency, high availability, security, maintenance, etc. SLA also specifies the agreed upon means for verifying SLA compliance, customer compensation plan that should be put in effect in case of SLA incompliance, and temporal framework that defines validity of the contract.
Personal tools
Create a book