Cloud Hosting Architecture
From FIWARE Forge Wiki
The Cloud Chapter offers Generic Enablers that comprise the foundation for designing a modern cloud hosting infrastructure that can be used to develop, deploy and manage Future Internet applications and services.
The following diagram shows the main components (Generic Enablers) that comprise the fourth release of FI-WARE architecture.
FIWARE Cloud architecture is heavily based on OpenStack. It comprises the following services:
- FIWARE IaaS GE providing the foundational cloud infrastructure layer, aimed at provisioning and management of compute, storage and network reosurces. This GE comprises several components:
- FIWARE Cloud Compute Service based on OpenStack Nova providing the ability to provision and manage VM and Linux Containers, as well as associated resources and artifacts. This is the most foundational part of the FIWARE Cloud, allowing to host arbitrary applications -- including FIWARE GEs and 'custom' applications.
- FIWARE Cloud Image Service based on OpenStack Glance, providing the ability to manage pre-built images that can be used to provision VMs (with operating system and a software stack pre-installed in the image) or Linux containers (typically comprising only the files of the application itself and its dependencies, rather than a full operating system).
- FIWARE Cloud Volume Service based on OpenStack Cinder providing the ability to provision and manage block storage (which can be attached to VMs and Linux Containers to keep the persistent state of the applications hosted on FIWARE Cloud).
- FIWARE Cloud Network Service based on OpenStack Neutron providing the ability to provision and manage virtual networks (connecting VMs and Linux Containers between them and to the external network).
- FIWARE Cloud Orchestration Service based on OpenStack Heat, providing the ability to orchestrate provisioning and ongoing management (e.g., auto-scaling) of collections of basic resources (VMs, networks, etc), including inter-dependencies between them.
- FIWARE Cloud Object Storage Service based on OpenStack Swift providing a scalable, resilient and efficient facility to store and retrieve 'blob' objects and associated metadata, as well as to run computations (storlets) on objects.
- FIWARE Cloud Application Management Service based on OpenStack Murano providing the ability to manage provisioning and configuration of complex applications, including basic resources as well as software configuration within the VMs/containers.
- FIWARE Cloud Policy Service, providing the ability to define rules and apply actions in response to certain events associated with cloud resources and their state. The exact alignment with several related projects in the OpenStack ecosystem (e.g., Congress) is currently under evaluation.
- FIWARE Cloud Monitoring Service, providing the ability to collect and distribute resource metrics associated with VMs or hosts. The exact alignment with several related projects in the OpenStack ecosystem (Ceilomter, Monasca, etc) is currently under evaluation. Note: this service is being transitioned to the FIWARE Ops chapter.
The color-coding notation on the above diagram shows the different types of components in the FIWARE Cloud architecture. The components in Yellow (IdM GE, Context Broker GE, etc) indicate the GEs developed outside of the Cloud Chapter. The components in Green (Compute, Volume, Network, Image, Orchestration, Monitoring) indicate the native OpenStack services which are intended to be used in FIWARE Cloud as-is, without further modifications. On the other hand, the Blue components indicate the GEs which are either based on corresponding OpenStack service but with modifications enhancements (Object Storage, Application Management) , or are currently not based on OpenStack and are aimed at alignment and/or contribution to OpenStack in future releases (e.g., Policy). Notice that Application Management and Orchestration GE replaces the PaaS Management and Software Deployment and Configuration (SDC) GEs (which are now deprecated).
Moreover, FIWARE Cloud architecture comprises the Docker GE, providing Docker container hosting capabilities, by leveraging the Docker open source project, with enhancements developed in FI-Core.
FIWARE Cloud capabilities can be invoked either directly through REST APIs, or via a command-line Toolkit, or via a Web Portal In particular, the most common way to get started with FIWARE Cloud in FIWARE Lab is by using the Cloud Portal, providing an easy and intuitive access to the cloud capabilities. Moreover, FIWARE Ops tools (developed in the corresponding FIWARE chapter) enable cloud administrators to implement federation of multiple cloud deployments, including Cloud portal, image management, (federated) monitoring, user management, networking and identity management as well as the ability to check the health status of the FIWARE Lab nodes.
Inter-Dependencies and Interaction between GEs
While each GE comprises a set of functions and capabilities that can be used stand-alone, in a typical cloud deployment the different GEs will interact with each other, to provide a complete end-to-end solution. For example:
- FIWARE Cloud Orchestration Service GE is likely to invoke APIs of the GEs responsible for provisioning and management of individual resources -- Compute, Image, Volume, and Network. Moreover, it is likely to use the FIWARE Cloud Monitoring Service GE to collect metrics of the underlying resources which comprise the service, to drive application elasticity.
- FIWARE Cloud Application Management Service GE is likely to invoke the Orchestration Service GE in order to orchestrate the (virtualized) resources comprising an application. It is also likely to leverage Monitoring Service GE and Policy Service GE to apply advanced autonomous management scenarios such as auto-scaling.
- FIWARE Cloud Image Service GE is likely to leverage the FIWARE Cloud Object Storage GE for resilient and scalable storage of VM images and associated artifacts.
- FIWARE Cloud Compute Service GE is likely to leverage the Image Service GE, the Volume Service GE and the Network Service GE in order to support VMs with richer set of capabilities and associated resources.
- All the FIWARE Cloud GEs use FIWARE Cloud Identity Service GE for authentication and authorization purposes.
- FIWARE Cloud Monitoring Service GE may interact with the Context Broker GE to collect metrics from other clouds in a federated environment.
All the above interactions are implemented in the FIWARE Lab environment.
Guiding Design Principles
There has been quite an amount of work carried out in defining what cloud computing is and what design principles it should adhere to. There are a number of general principles that have been collated by the Future Internet Architecture Group and some of these are especially relevant. One of the canonical definitions of cloud computing, more so from a technology perspective, is that of the National Institute for Standards and Technologies (NIST).
A consumer can unilaterally provision computing capabilities, such as server time and network storage, automatically as needed without requiring manual human interaction with each service provider.
Broad Network Access
Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin- or thick-client platforms (e.g., mobile phones, tablets, laptops, and workstations).
The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center). Examples of resources include storage, processing, memory and network bandwidth.
Capabilities can be quickly elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time.
Measured Service / Pay-As-You-Go
Cloud systems automatically control and optimize use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported. This provides transparency for both the provider and consumer of the utilized service. This finally allows for metered services pricing, also known as pay-as-you-go.
To be dependable, a service must exhibit/implement the following attributes as defined by Avizienis:
- Availability: readiness for correct service
- Reliability: continuity of correct service
- Safety: absence of catastrophic consequences on the user(s) and the environment
- Integrity: absence of improper system alteration
- Maintainability: ability for a process to undergo modifications and repairs
- Confidentiality: absence of unauthorised disclosure of information
This definition was formed in 2004, however in the age of the Internet of services, where end-users are service- and not product-oriented, this list needs to be updated to better reflect today’s needs. The definition should cover aspects of transparency. Transparency in this context is the ability to inspect and introspect a service so that the delivered and guaranteed quality of the service agreement can be verified and observed. For us in Cloud Hosting this means that cloud providers should provide means to access performance information on one’s provisionings so that, not only can one see what one has but one can build more useful services atop. This finding is reflected in the EU Future Internet Architecture Working Group findings.
Architecture Description of GEs
The following FIWARE Cloud Services are adopted 'as is' from OpenStack (comprising together the IaaS GE):
The following GEs are deprecated, and have been replaced by the Application Management GE:
- PaaS (DEPRECATED)
- SDC (DEPRECATED)
The following GEs are part of FIWARE Cloud architecture, but are maintained under different technical chapters:
- FIWARE.ArchitectureDescription.Security.Identity Management Generic Enabler (Security)
- FIWARE.ArchitectureDescription.Cloud.Monitoring (FIWARE Ops)
Interfaces to Other Chapters
All the FIWARE GEs are designed to be hosted on the FIWARE Cloud, including automated provisioning and life cycle management by leveraging the Cloud GEs. This includes packaging of each GE reference implementation as a virtual machine image compatible with FIWARE Cloud (and OpenStack), and/or by designing corresponding Application Blueprints comprising the orchestration and software configuration artifacts required to provision and manage the life cycle of the corresponding GEs.
Data and Context Management
Several GEs in the Data and context Management Chapter have critical scalability requirements. As such, they leverage the advanced capabilities of the Cloud chapter to automate their provisioning and life cycle, including provisioning of individual resources, orchestration of groups of resources, software configuration, auto-scaling, etc.
The FIWARE Cloud architecture provides an inherent support for multi-tenancy. In particular, access to FIWARE Cloud GEs, both via the Cloud Portal and via APIs, is secured by the FIWARE Security Identity Management GE, which provided a unified OAuth-based identity management system across chapters and GEs, based on OpenStack Identity Service (Keystone).
FIWARE Ops chapter provides tools to provision and manage FIWARE Cloud deployments (e.g., across the different FIWARE Lab nodes), as well as to federate several FIWARE Clouds together.