We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.WebUI.Synchronization R5 - FIWARE Forge Wiki

FIWARE.OpenSpecification.WebUI.Synchronization R5

From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.MiWi.Synchronization
Chapter Advanced Web-based User Interfaces,
Catalogue-Link to Implementation Synchronization
Owner Adminotech Ltd., Lasse Öörni


Contents

Preface

Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


Copyright

Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.

Overview

Introduction

Interactive, multi-user Web applications such as virtual worlds often have the common requirements of storing the "world" or scene on a server, and communicating changes in it to participating clients. Depending on the application, the kinds of objects stored in the scene and the operations performed on them may differ. To allow for arbitrary use-cases a generic scene model is required, as well as open communication protocols. A scene should at the same time be easily accessible and editable from various external applications, but also allow high-performance synchronization for clients which are observing the scene and the changes in its content (such as objects moving due to a physics simulation taking place) in real-time.

In large-scale (3D) applications the scene data will need to be distributed across several server instances for example based on spatial partitioning.

This GE presents a lightweight server architecture and two communication protocols to interact with a synchronized dynamic scene hosted on a server: SceneAPI, a RESTful HTTP API for non-realtime querying and modification of the scene, and a WebSocket-based bidirectional protocol for connected Web clients to receive continuous real-time scene updates, and to post their real-time changes to the scene.

Basic Concepts

This GE is centered on the concept of a scene, which is defined as generically as possible. It is not only limited to 3D virtual worlds, but could just as well represent for example a 2D spreadsheet, or an even more abstract collection of data.

Scene

A hierarchical collection of objects with arbitrary attribute values, which can be modified. The scene can be represented in one of the two following ways:

  • A HTML DOM document, which is augmented with the elements from the XML3D [1] extension for 3D objects.
  • The Entity-Component-Attribute structure as described in Scene and EC model. Entities are empty containers identified by unique integer IDs, with no functionality on their own, to which typed Components (for example a 3D Transform or a 3D Mesh) can be created. Entities can also contain child entities. The actual data of the Components exists in form of named and typed Attributes.

The mapping between the DOM and Entity-Component-Attribute scene representations is defined by the 3D-UI GE, which is concerned with actually rendering the scene in 3D. The synchronization mechanism itself does not need to make distinction between renderable and non-renderable scene data, or understand the mapping.

The basic data types supported for Attributes are:

  • boolean
  • integer
  • float
  • string

Compound and specialized attributes can be formed from these basic types, for example a 3D position vector from 3 float values, or a quaternion representing a rotation (or an RGBA color value) from 4 float values.

Server

A program hosting a synchronized scene, and which implements the communication protocols described below.

Client

A program which wishes to either observe or modify the scene hosted by a server. The client can use either, or both of the communication protocols defined below. A typical example would be a JavaScript-based client running in a Web browser.

SceneAPI

A RESTful HTTP API provided by the server, which allows for query and modify operations into the scene.

Real-time Sync Protocol

A real-time, connection-oriented protocol that clients use to connect (login) into an interactive scene to observe the scene content, to observe changes, and to push their own changes, as well as to send and receive Entity Actions, a form of Remote Procedure Call. It is implemented using the WebSocket protocol.

Because of realtime processing performance and bandwidth demands, it is estimated that it is more efficient to operate using the Entity-Component-Attribute data model, than using a DOM scene representation. This allows to effectively encode attribute values in binary according to their type (for example integer values can be variable-length encoded according to their magnitude). From this it follows that the internal scene model of the server should be ECA-based.

Entity Action

A real-time command sent either by the client or the server to the other end, which does not directly modify the scene, but is instead interpreted by a script or plugin running on the client or the server. It can include a number of parameters, similar to a Component's Attributes. An example would be to transmit keyboard and mouse controls from the client to steer a moving object which is being simulated on the server.

Access control

The server may choose to limit access to scenes for both observing (read access) and modifying (write access). The actual authentication mechanisms are beyond the scope of this GE, but both the SceneAPI and the Real-time Sync Protocol provide for sending arbitrary credentials or access tokens: the SceneAPI in each operation, and the Sync Protocol during login to scene. The server implementation should provide hooks for authentication and access control.

Local objects

An important concept for the scene objects is to be able to opt-out of the synchronization, ie. be either server-local or client-local objects. This avoids wasting bandwidth for objects which are for example used as special effects only (ie. particle effects in a game.) These can be defined and queried with the SceneAPI, but will not be synchronized with the Real-time Sync Protocol.

Distributed scene

A scene that is distributed into smaller scene regions hosted on separate server instances. The servers need to be aware of neighbor servers to relay a client to the appropriate neighbor region(s) when the client is approaching the border of the server's own region.

Application-specific data

To contain arbitrary application-specific data, new combinations of Attributes can be dynamically introduced during runtime by either using the DynamicComponent mechanism, which allows adding Attributes to a component on the fly, or by registering a new static-structured custom component type by giving its type name and a list of the Attributes it should contain.

Generic Architecture

The server needs to hold an internal model of the scene, which is manipulated both by the SceneAPI and the Real-Time Sync Protocol. Both protocols can be implemented as plugins, which access the internal scene model.

In addition of a reference server implementation, a JavaScript library for making connections using the Real-Time Sync Protocol will be provided.

The diagram below describes the overall server architecture and examples of usage: a Web client making a real-time connection, and an external editor application performing REST calls to modify the scene.

File:SynchronizationServerArch.png

Main Interactions

Interactions with the SceneAPI protocol and the Real-Time Sync Protocol are described separately. The SceneAPI operations are stateless, while a real-time server connection requires state to be maintained on both the server and the client.

SceneAPI

The SceneAPI operations can be divided into scene queries and scene modification.

A scene query specifies the criteria for objects and how to return the results, ie. just an object list, or the full attribute content of the objects. Query criteria is either:

  • The value of some attribute or a combination of them. For string attributes wildcards and regexes can be supported, to enable for example searching for objects named in a certain manner.
  • A spatial query, for example all 3D objects contained within a sphere of certain center and radius.

Scene modification consists of creating, modifying and removing objects. An object can either be created empty or with its full content already be specified on creation. Object hierarchies ("scene fragments") can also be created in one operation. The place in the hierarchy can be specified (eg. "create as child of this object")

In terms of the Entity-Component-Attribute model, object modification is the addition and removal of Components, and modifying Attribute values inside the Components.

The whole scene can also be created or queried at once.

The following sequence diagram illustrates basic interactions (query and modification) between the client and the server.

Real-Time Sync Protocol

The real-time connection into a server scene operates in phases:

  • Login into the scene, including access credentials
  • Server sends initial scene state
  • Server sends additional scene updates, and the client may push its modifications to the scene, which the server will distribute to other connected clients. The server and client can also exchange RPC-like Entity Action commands
  • Disconnection (leaving the scene)

After login, the scene updates consist of the following operations, which for the most part mirror the SceneAPI modification operations:

  • Create an Entity (can be empty or include full Component / Attribute content)
  • Create a Component into an Entity
  • Modify an Attribute value in a Component
  • Delete a Component from an Entity
  • Delete an Entity
  • Send an Entity Action
  • Introduce a new static-structured custom component type: send its type name and the list of Attributes it should contain

The scene modification operations sent by a client may fail if the client does not have permission: in this case an error reply will be returned and the client should roll back its local scene state.

In its basic form the synchronization protocol sends all updates from the scene to all clients without regard for optimization. Several optimization strategies can be devised to reduce the bandwidth use and CPU load of both the server and the client: these are commonly called interest management. For example a metric based on object bounding box size and distance from observer can be used to determine whether an object is important enough to be synchronized to the client. This kind of interest management requires the client to stream its observer position to the server with sufficient frequency. In non-spatial scenes other interest management strategies would be required.

An example session between a single client and server is depicted in the following sequence diagram:

Basic Design Principles

  • The two communication protocols are independent, but manipulate the same internal scene data.


Detailed Specifications

Re-utilised Technologies/Specifications


Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FIWARE level, please refer to FIWARE Global Terms and Definitions

Annotations
Annotations refer to non-functional descriptions that are added to declaration of native types, to IDL interface definition, or through global annotations at deployment time. The can be used to express security requirements (e.g. "this string is a password and should be handled according the security policy defined for password"), QoS parameters (e.g. max. latency), or others.
AR
AR → Augmented Reality
Augmented Reality (AR)
Augmented Reality (AR) refers to the real-time enhancement of images of the real world with additional information. This can reach from the rough placement of 2D labels in the image to the perfectly registered display of virtual objects in a scene that are photo-realistically rendered in the context of the real scene (e.g. with respect to lighting and camera noise).
IDL
IDL → Interface Definition Language
Interface Definition Language
Interface Definition Language refers to the specification of interfaces or services. They contain the description of types and function interfaces that use these types for input and output parameters as well as return types. Different types of IDL are being used including CORBA IDL, Thrift IDL, Web Service Description Language (WSDL, for Web Services using SOAP), Web Application Description Language (WADL, for RESTful services), and others.
Middleware
Middleware is a software library that (ideally) handles all network related functionality for an application. This includes the setup of connection between peers, transformation of user data into a common network format and back, handling of security and QoS requirements.
PoI
PoI → Point of Interest
Point of Interest (PoI)
Point of Interest refers to the description of a certain point or 2D/3D region in space. It defines its location, attaches meta data to it, and defines a coordinate system relative to which additional coordinate systems, AR marker, or 3D objects can be placed.
Quality of Service (QoS)
Quality of Service refers to property of a communication channel that are non-functional, such a robustness, guaranteed bandwidth, maximum latency, jitter, and many more.
Real-Virtual Interaction
Real-Virtual Interaction refers to Augmented Reality setup that additionally allow users to interact with real-world objects through virtual proxies in the scene that monitor and visualize the state in the real-world and that can use services to change the state of the real world (e.g. switch lights on an off via a virtual button the the 3D scene).
Scene
A Scene refers to a collection of objects, which are be identified by type (e.g. a 3D mesh object, a physics simulation rigid body, or a script object.) These objects contain typed and named data values (composed of basic types such as integers, floating point numbers and strings) which are referred to as attributes. Scene objects can form a hierarchic (parent-child) structure. A HTML DOM document is one way to represent and store a scene.
Security
Security is a property of an IT system that ensures confidentiality, integrity, and availability of data within the system or during communication over networks. In the context of middleware, it refers to the ability of the middleware to guarantee such properties for the communication channel according to suitably expressed requirements needed and guarantees offer by an application.
Security Policy
Security Policy refers to rules that need to be fulfilled before a network connection is established or for data to be transferred. It can for example express statements about the identity of communication partners, properties assigned to them, the confidentiality measures to be applied to data elements of a communication channel, and others.
Synchronization
Synchronization is the act of transmitting over a network protocol the changes in a scene to participants so that they share a common, real-time perception of the scene. This is crucial to implementing multi-user virtual worlds.
Type Description
Type Description in the context of the AMi middleware refers to the internal description of native data types or the interfaces described by an IDL. It contains data such as the name of a variable, its data type, the hierarchical relations between types (e.g. structs and arrays), its memory offset and alignment within another data type, and others. Type Description are used to generate the mapping of native data types to the data that needs to be transmitted by the middleware.
Virtual Character
Virtual Character is a 3D object, typically composed of triangle mesh geometry, that can be moved and animated and can represent a user's presence (avatar) in a virtual world. Typically supported forms of animation include skeletal animation (where a hierarchy of "bones" or "joints" controls the deformation of the triangle mesh object) and vertex morph animation, where the vertices of the triangle mesh are directly manipulated. Virtual character systems may support composing the character from several mesh parts, for example separate upper body, lower body and head parts, to allow better customization possibilities.
WebGL
WebGL → (Web Graphics Library) is a JavaScript API for rendering 3D and 2D computer graphics in web browser.
Personal tools
Create a book