We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.Cloud.DCRM - FIWARE Forge Wiki

FIWARE.OpenSpecification.Cloud.DCRM

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.Cloud.DCRM
Chapter Cloud,
Catalogue-Link to Implementation DCRM
Owner IBM, Alex Glikson, IBM


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.

FIWARE WIKI editorial remark:
This page corresponds to Release 3 of FIWARE. The latest version associated to the latest Release is linked from FIWARE Architecture

Copyright

Copyright © 2011-2013 by IBM and Intel Corporations

Legal notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

This specification describes the DataCenter Resource Management (DCRM) GE, which is a key enabler to build a cloud solution.

The DCRM GE provides the basic Virtual Machine (VM) hosting capabilities, as well as management of the corresponding resources within the DataCenter that hosts a particular FI-WARE Cloud Instance.

The baseline for the DCRM GE is OpenStack. Hence, DCRM offers all the capabilities that OpenStack natively provides to cloud hosting users and cloud hosting providers plus some unique extended capabilities.

The main capabilities provided for a cloud hosting user are:

  • Browse VM template catalogue and provision a VM with a specified virtual machine image
  • Manage life cycle of the provisioned VM
  • Manage network and storage of the VM
  • Resource monitoring of the VM
  • Resiliency of the persistent data associated with the VM
  • Manage resource allocation (with guarantees)
  • Secure access to the VM

For a cloud hosting provider, the following capabilities are provided:

  • Resource optimization and over-commit (aimed at increasing the utilization and decreasing the hardware cost)
  • Capacity management and admission control (allowing to easily monitor and control the capacity and the utilization of the infrastructure)
  • Multi-tenancy (support isolation between VMs of different accounts)
  • Automation of typical admin tasks (aimed at decreasing the admin cost)
  • Resiliency of the infrastructure and of the management stack (aimed at reducing outage due to hardware failures)



DCRM components

The following diagram shows the main components of the DCRM GE, as well as its main interactions.

DCRM Architecture
Main components of the DCRM Architecture

On the above diagram, the API component is the front-end of DCRM. It can be implemented either as a set of endpoints handling each of the resource types -- virtual servers, virtual disks, virtual networks and virtual images, or a single endpoint dispatching the incoming requests to the corresponding 'backend' runtime. At the back-end, different aspects of resource management are handled by corresponding internal services, such as policy service and placement service.

DCRM-specific features

With respect to the OpenStack baseline, DCRM provides in addition the following set of high-level advanced features:

  • Shared storage configuration enabling live VM migration and related scenarios
  • VM High Availability
  • Adaptive scheduling for optimized resource utilization
  • Support for QoS guarantees for workloads
  • Support for placement policies
  • Support of concurrent management and deployment workflows in a scalable consistent manner
  • Unified management of heterogeneous environments
  • Support for policy-based virtual network connectivity

Extended capabilities with respect to OpenStack

The high-level features listed above are enabled by the combination of a set of extended capabilities that can also be used in isolation. They are:

  • Host failure detection (Zookeeper): enables the automatic recovery of VMs from a failed host
  • Advanced Scheduler:
    • Flexible global resource optimization based on large and extensible set of metrics, including resource utilization: enables the implementation and configuration of several resource utilization optimization policies for infrastructure providers
    • Ongoing placement optimization using live migration and a solver: live migration of VMs across hosts allows continuous optimization of VMs placement according to configurable policies with no service disruption
    • Placement support for automated HA of VM instances: in an HA scenario, if a host fails, the placement logic will automatically provide correct placement for all recovered VMs
    • Host evacuation support: this capability simplifies infrastructure management tasks by automatically moving out all VMs from one host (host maintenance scenario)
    • Support for anti-affinity / placement policies: this capability allows cloud users to specify constraints and requirements with respect to VMs placement. This capability enables both service availability and VM proximity scenarios for cloud applications
    • HA-aware admission control: this mechanism is used to guarantee that at all times the system will have sufficient host spare capacity to recover from a single host failure
    • Unified support for heterogeneous performance of the underlying HW: a translation layer is used to assign correct VCPU nominal performance on heterogeneous hosts. This prevents wasting capacity and enables optimal resource utilization
    • Fine-grained compatibility verification: it prevents VMs from being deployed on incompatible HW
    • Flexible resource allocation policies and adaptive resource over-commit based on idleness detection: VM admission control adapts to the number of idle VMs in the system, maximizing resource utilization

Basic Concepts

The key concepts visible to the cloud user are:

  • Virtual server, a virtualized container that can host an arbitrary Operating System and arbitrary software stack on top, installed within the virtual server. Virtual servers are also referred as Virtual Machines (VMs) or (virtual) image instances (see definition of virtual image below). DCRM GE supports provisioning and life cycle management of Virtual Servers.
  • Virtual disk, representing a persistent virtual disk that can be potentially attached to an arbitrary virtual server. DCRM GE supports provisioning of virtual disks, as well as their attachment to virtual servers.
  • Virtual network, representing a logical network abstraction that would typically represent a network segment at layer 2 of the OSI model. DCRM GE supports provisioning of virtual networks, as well as attachment of virtual NICs of virtual servers to them.
  • Computing Flavor, a computing flavor is a hardware configuration that can be associated to a virtual server. Each flavor has a unique combination of CPU cores, memory capacity and disk space. An example of this combination could be 8 virtual CPUs, 16 Gb RAM, 10 Gb HDD and 160 Gb of ephemeral disk (a disk that is stored locally on the hypervisor host). Computing flavors must be registered and made available prior to their association to virtual servers.
  • Virtual image, an image is a collection of packaged files used to create or rebuild a virtual server. Basically, a virtual image is a snapshot of a virtual server from which you can create new virtual servers (i.e., image instances). Each virtual server derived from a virtual image hosts the Operating System and software stack associated to the virtual image and is assigned one among the set of available computing flavors, each of which maps to a configuration of computing resources (memory, CPU, etc.). A number of pre-built virtual images can be made available to cloud users but they may also create their own images using tools defined for that purpose. These custom images are useful for backup purposes or for producing "gold" server images if you plan to deploy a particular server configuration frequently. DCRM GE supports life cycle of virtual images, as well as provisioning of virtual servers based on virtual images.


Main Interactions

DCRM provides a wide variety of operations to provision and manage the life cycle of cloud resources. The most important ones are listed below.

Virtual Images

  • listVirtualImages -- Returns a list of all available virtual images (visible by the authenticated user)
  • queryVirtualImages -- Returns a list of available virtual images, filtered by given query criteria
  • getVirtualImageDetails -- Returns details of a virtual image (type, size, creation details, etc.)
  • uploadVirtualImage -- Uploads a new virtual image into the virtual image repository

Virtual Servers

Provisioning

  • createVirtualServer -- Provisions a new virtual server with the given properties (virtual hardware, policy parameters, access, etc). Returns unique ID of the virtual server.
  • destroyVirtualServer -- Removes a virtual server

Power Management

  • powerOnVirtualServer -- Powers on a virtual server
  • powerOffVirtualServer -- Powers off a virtual server
  • RestartVirtualServer -- Restarts a virtual server
  • ShutdownVirtualServer -- Shuts down a virtual server (note: the ability to perform this operation on the fly depends on the capabilities of the underlying virtualization platform)

Reconfiguration

  • resizeVirtualServer -- Changes the virtual hardware allocation for a virtual server, e.g., allocated RAM or number of CPUs (note: the types of resources for which the reconfiguration can be done on the fly depends on the capabilities of the underlying virtualization platform)

Inventory

  • getVirtualServerDetails -- Returns details of a virtual server (virtual hardware specification, state, associated policy parameters, access details, etc.)

Virtual Disks

Provisioning

  • createVirtualDisk -- Provisions a new virtual disk with the given properties (size, capabilities, etc.). Returns unique ID of the virtual disk.
  • destroyVirtualDisk -- Removes a virtual disk

Attachment

  • attachVirtualDisk -- Attaches a given virtual disk to a given virtual server (note: the ability to perform this operation on the fly depends on the capabilities of the underlying virtualization platform)
  • detachVirtualDisk -- Detaches a given virtual disk from a given virtual server (note: the ability to perform this operation on the fly depends on the capabilities of the underlying virtualization platform)

Inventory

  • getVirtualDiskDetails -- Returns details of a given virtual disk (size, capabilities, attachment details, etc.)

Virtual Networks

Provisioning

  • createVirtualNetwork -- Provisions a new virtual network with the given properties (e.g., VLAN ID, capabilities, etc.). Returns unique ID of the virtual network.
  • destroyVirtualNetwork -- Removes a virtual network

Attachment

  • attachVirtualServerToNetwork -- Attaches a virtual network interface of a given virtual server to a given virtual network
  • detachVirtualServerFromNetwork -- Detaches a virtual network interface of a given virtual server from a given virtual network

Inventory

  • getVirtualNetworkDetails -- Returns details of a given virtual network (ID, capabilities, attachment details, etc.)

Example Scenario

The following sequence of operations describes a typical (simple) scenario of provisioning of a virtual server hosted in the Cloud:

  • User authenticates with Identity Management GE, receives a token
  • User creates ssh key-pair, to be used to authenticate with the guest OS within the virtual server instances
  • User retrieves a list of available images and of virtual server flavors
  • User requests a new virtual server
  • User verifies that the virtual server creation has completed
  • User retrieves the IP address allocated for the virtual server
  • User connects to the virtual server using ssh

Basic Design Principles

When applied to DCRM, the general design principles outlined at Cloud Hosting Architecture can be translated into the following key design goals:

  • Fully-automated provisioning and life cycle of compute, storage and network resources, requested, managed and released via a standards-based REST API: The REST API allows management of the provisioned resources both through a Web-based user interface or direct API invocation. The API is designed to be abstract and "declarative": a tenant specifies "what" he needs, while the "how" of the provisioning is left to the infrastructural policies and goals. The goal is to provide a standard interface to consume the virtual resource service regardless of the underlying technology used to implement the provisioning infrastructure.
  • High resource utilization, while providing the necessary levels of isolation, availability and performance of provisioned resources: Improved utilization and automation of resources allow greater cost efficiencies for both infrastructure providers and tenants.
  • Ability to dynamically control the amount of allocated resources, as well as to monitor the actual resource usage: Dynamic control of resource provisioning is at the core of application elasticity, enabling the correct sizing of applications' components and operating costs to the varying load conditions.
  • High availability and scalability of the management stack: The infrastructure management components provide availability and scalability through the most advanced current design and development practices, including: fully-distributed shared-nothing architectures to naturally support horizontal scalability, asynchronous communication mechanisms, and extensive automated testing cycles for each contribution.
  • Non-disruptive, automated administrative tasks (e.g., infrastructure maintenance): when scale grows, partial hardware and software failures are the norm rather than the exception. Infrastructure providers require mechanisms to automate administrative tasks reducing the needed effort and preventing any disruption to the tenants' services and applications.
  • Avoid non-authorized access to resources and workloads: Role Based Access Control (RBAC), coupled with an Identity Management service, ensure security by user, role and project.


Detailed Specifications

Following is a list of Open Specifications linked to this Generic Enabler. Specifications labeled as "PRELIMINARY" are considered stable but subject to minor changes derived from lessons learned during last interactions of the development of a first reference implementation planned for the current Major Release of FIWARE. Specifications labeled as "DRAFT" are planned for future Major Releases of FIWARE but they are provided for the sake of future users.

Open API Specifications

The IaaS GE is in fact the OpenStack components: Nova, Glance, Swift, and Neutron. It's current deployment is the OpenStack Kilo Release and as such its API is OpenStack Kilo API:

Re-utilised Technologies/Specifications

The DCRM OpenStack API is a RESTful, resource-oriented API accessed via HTTP that uses XML-based and/or JSON-based representations for information interchange. It builds on top of the Compute, Identity, Images, and Network OpenStack API v2.0, but it extends the original specifications to provide support to additional DCRM-specific management features currently not available in OpenStack.


Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions

  • Infrastructure as a Service (IaaS) -- a model of delivering general-purpose virtual machines (VMs) and associated resources (CPU, memory, disk space, network connectivity) on-demand, typically via a self-service interface and following a pay-per-use pricing model. The virtual machines can be directly accessed and used by the IaaS consumer (e.g., an application developer, an IT provider or a service provider), to easily deploy and manage arbitrary software stacks.
Personal tools
Create a book