We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
FIWARE.OpenSpecification.WebUI.RealVirtualInteraction - FIWARE Forge Wiki


From FIWARE Forge Wiki

Jump to: navigation, search
Name FIWARE.OpenSpecification.WebUI.RealVirtualInteraction
Chapter Advanced Web-based User Interfaces,
Catalogue-Link to Implementation Real Virtual Interaction



Within this document you find a self-contained open specification of a FIWARE generic enabler, please consult as well the FIWARE Product Vision, the website on http://www.fiware.org and similar pages in order to understand the complete context of the FIWARE platform.


Legal Notice

Please check the following Legal Notice to understand the rights to use these specifications.


A key aspect of Augmented Reality is that virtual content is not just presented embedded within the context of the real world, but that it should also allow users to interact actively with real objects and the objects to provide input to the user. An example is a real object that visualizes its internal (non-visible) state overlaid on the real object using AR techniques. The user can then affect the real object by interacting with the virtual object using multi-touch gestures, which are relayed to the real object using services. Many of the necessary tools are available by the other topics in this GE (e.g. presentation of overlaid content, services to modify the state of the UI, services calls initiated from UI components). This higher-level GE deals with defining the aspects necessary to put them together and provide any necessary missing components and glue them together.

Basic Concepts

Internet of Things (IoT) concept

The Internet of Things refers to uniquely identifiable objects and their virtual representations in an Internet-like structure. According to some estimates more than 30 billion devices will be wirelessly connected to the Internet of Things [1]. Equipping all objects in the world with minuscule identifying devices or machine-readable identifiers could transform daily life. For instance, business may no longer run out of stock or generate waste products, as involved parties would know which products are required and consumed. One's ability to interact with objects could be altered remotely based on immediate or present needs, in accordance with existing end-user agreements. IoT concept is being researched in EU by Internet of Things Europe [2].

Augmented Reality

Augmented reality is a concept where images from a camera are being augmented with virtual 3D models of objects or characters based on location and use of specially designed markers.


In this GE request/response offers other service developers to send HTTP POST/GET queries to real virtual interaction backend service to access and obtain sensor and actuator information for instance based on their geographic location.


WebSocket protocol provides a full-duplex connection between a browser run web application and real virtual interaction backend service[3].Websocket has been defined in IETF's [4]. In this GE WebSocket acts as an enabler for end-users to publish and subscribe to real time data flow with sensors and actuators.

RESTful Data Format

A JavaScript Object Notation (JSON) type of document that consist of parent-child relationships, and uses JSON Objects or JSON Arrays as building blocks.

Generic Architecture

Figure 1: Deployment diagram representing the interaction between real world devices, real virtual interaction service, and 3rd party services and end-user client applications.

Device Entity

Represents end-point IoT devices in figure 1. In this GE the decision was to implement the connection between end-point devices and the back-end server using uniform datagram packets (UDP). UDP is often used in traffic where delivery is not guaranteed. Although it is possible to implement higher level logic to make sure that a sensor event was delivered using call back events. In default though sensor event delivery is not verified. UDP is also suitable in situations where there are a sensor events published in high frequency intervals, on such situation it is not critical to deliver all packets similar to video streams for instance. In this GE the device was implemented as an Android software application. Since there can exist upper limits for UDP packet size we have also included data compression technique to allow more information to be send in one packet. The device in this GE can consist of one or more sensors or actuators. The device itself contains communication logic and operating system.

Real Virtual Interaction Server

The back-end server consist of connection interfaces for clients and third party services. The interface in this contexts differs from device-to-server communication in a way that the server offers means of conversation between clients and other services and the server. The device-to-server is only a socket offering means of connection. By this we mean that it is up to the sensors to handle HTTP POST query for instance and it is up to the server to broker sensor events and parse the information and store it in a data base. The data base contains sensor and actuator information containing history of events. The information is being categorized in relational manner using UUIDs as data base keys to differentiate devices. The message handler component is the core server component responsible of maintaining subscription lists and brokering sensor events and actuator commands accordingly.

Third Party Services

With third party services this GE refer to cloud type of services that apply big data to create contexts suitable for their clients. A practical example in this GE is a POI service that connects sensors and actuators to points of interest based on geographic location of end-user using the POI service. This connection is handled using request/response type of asynchronous queries. A third party server may define a spatial query of a real world geographic area and the real virtual interaction server provides found sensors and actuators to the given search criteria. Alternatively third party services may broker commands to actuators from end-users connected to their service or offer direct access to the real virtual interaction server socket.

End-user Application

End-user application refers to users using some form web application or web service through a desktop or hand held device application. In this GE we provide sample client using Web Socket to subscribe to sensor events in real time.

Main Interactions

End-point Devices

With fairly recent advances in sensors and actuators technologies have made it possible for such devices to be applied in everyday usage due to lowered size, power consumption and mainly cost. Although most of the sensors apply their "awareness" on a personal area network (PAN) level around their location of installation, there exists a vision of IoT where such devices would be equipped with ability to connect to resources across Internet. It is worthwhile to be aware that often these devices contain lightweight microprocessors Bluetooth or similar connectivity thus not able to directly connect to Internet resources. In such cases they need a gateway and Mesh networking protocol to relay messages across their local domain to Internet resources. Also to conserve power many wireless sensors only publish information either on request basis or in specific intervals. Without existence of a server and a data base this information could be lost[6]. Also further down the topology FIWARE.OpenSpecification.IoT.Gateway.DataHandling should be implemented to gateway device to store information in temporary buffers in case connections are lost to either back-end or to IoT devices and resend when connection is established. This component could also do some traffic managing in case the amount of sensor data blocks the network connections.

Since majority of devices containing raw sensors and actuators offer vendor specific and proprietary software solutions, in this GE we chose Android devices as a basis for testing implementation. Android devices contain both real sensors and actuators and ability to connect to Internet resources in variety of ways. Also Android devices are widely available and do not require purchasing a specific device for testing.

Real Virtual Interaction Back-end Service

Middle-ware IoT service is the core component of this GE. The purpose of this middle-ware service is to gather together devices containing sensors and actuators. The middle-ware service offers a data base component for storing necessary connection information and history or latest data published by a sensor. Since IP addresses often change the real virtual interaction server needs to maintain the information based on UUIDs [7]. The middle-ware service also provides a layer of security that allows publishing sensor data publicly in the Internet, but also to shield them from direct access. FIWARE.OpenSpecification.IoT.Backend.IoTBroker GE states that The IoT Broker is stateless in the sense that no context information is stored by it. Its role in the Internet of Things is not to serve as a central data repository, but as a central point which retrieves and aggregates data from various sources on behalf of applications. In Real Virtual Interaction GE this is being extended as virtual environments are highly context-defined and thus we are likely managing a particular set of IoT devices tied together by their type or geographic location for instance. Thus mere broker in this case is not enough to act between an end-user client and an IoT device. In this GE the main purpose is provide RESTful means for connecting devices with very limited logical capabilities for cloud type of services and directly to end-users. How this information is being applied by augmented reality software developers is not being tackled in this GE, we merely provide means for accessing actuators and receiving data from sensors.

Services and Applications Utilizing Device Data

Similar to FIWARE.OpenSpecification.I2ND.CDI GE, the aim Real Virtual Interaction GE is to provide access to devices that may contain multiple sensors and actuators or directly to an individual sensor. Changing state of a sensor may be resolved by sending a HTTP POST call request brokered by a middleware service to the particular sensor. This is the case in FIWARE.OpenSpecification.Data.PubSub GE where it presents a way for hand held devices to receive in a specific area by subscribing through Bluetooth devices and receiving information based on users location within indoor spaces. Considering that the end-user may be interested to see live feed from a sensor or be able to control a device in a way that requires real-time control of the device through remote interface, we will need to implement different method in respect to HTTP. Web Sockets offer this ability. In architectural representation shown in figure 1 suggestion is that a third party middle-ware service such as a point-of-interest (POI) type of service will request and parse the information to the end-user client interested about a specific POI. The third party middle-ware can then provide connection details to a web socket to a particular sensor or set of sensors.

RESTful Data Format

There needs to exist a format how devices publish information regarding sensors and actuators. Considering that the end-user client applications or third party services may not be aware of specifics regarding a sensor or an actuator, there needs to exist a format that contains necessary elements so that the data can be dynamically applied. Also there needs to be way to also publish what kind of call backs are available to verify that an actuator command has been implemented for instance. Also there needs to be a way to embed a parent context in situations where the device itself is not able to connect to Internet, but rather uses a gateway device for the connection. In this case the gateway device should embed its signature in parent-child relationship. Whether the RESTful format is embedded with gateway context there needs to be a "dataformat_version" key among the parent fields. The FIWARE release is designated with version "1.0". The backend will drop packets that have different version or lack the field. The data format version number will not be stored nor passed to connected clients applications. The designed data format has been derived from FIWARE NGSI-10 Open RESTful API Specification in to a modified JavaScript Object Notation (JSON) format.

Figure 2: Class diagram depicting the structure of the data format which devices containing sensors and actuators can use to publish information.

Entity in this context represents a physical device that may contain zero or more sensors and zero or more actuators. It is expected that a device would at least contain either one sensor or one actuator. Entity device could be either thought as a independent device that has capability to send messages to Internet resources (real virtual interaction server) or at least transmit messages forward in local area network(LAN) or personal area network(PAN) by using RJ-45, Wi-Fi, Bluetooth or similar connection methods. In this particular case an another device that has an Internet connection and receives this Wi-Fi or Bluetooth message should wrap the RESTful data format with its context. This enables real virtual interaction server or similar to be aware that the device itself requires a broker gateway device in order to transmit or received messages. Entities, sensors and actuators all can have attributes. These attributes are sorted in key-value pairs. For instance en entity may have two attributes: {"name" : "TI CC2540", "gps" : [65.1003, 25.3321]}. Although the API specification examples will propose few suggestion regarding what attributes should be included, the design itself supports dynamic key-value pairs. The "key" must be of String primitive data type. Value may be any primitive data type supported by JSON schema. Sensors and actuators may have configuration parameters. Parameter refers to a configurable variable. For instance, a sensor may have configurable variable for interval of sensor events in milliseconds. A parameter should contain the description of the configurable variable and what value options are available and SI unit of the values.

Each sensor may publish one or more value entries in a single RESTful data format string. Each value should at least have the following attributes: SI unit for the values, time stamp for the measurement, primitive data type of the values and one or more entries with same primitive data type. Value may contain additional key-value pairs. It is expected that a sensor only publishes one type of value, addition value entries are similar in type but may have different entries and time stamp. Actuators can have one or more Actions. In this way the actuators can publish to clients how to the action can be triggered. Action should publish name or type of the action, what values are available for the action and what SI unit they are in. Also an action should indicate the current state of the actuator. Callbacks are for verifying the send message is being handled. For instance a callback events could be simple ACK,CON, RST type of message to indicate new status or to confirm that the message was received.

Basic Design Principles

  • Modular open source back-end implementation that offers a RESTful API for application developers using JSON.
  • Enable both asynchronous and synchronous subscription/publish methods for application developers by enabling HTTP/WEBSOCKET options.
  • Work on Session, Presentation and Application layers of OSI model [8].


Detailed Specifications

Real Virtual Interaction detailed API

Re-utilised Technologies/Specifications

  • FIWARE.OpenSpecification.IoT.Backend.IoTBroker
  • FIWARE.OpenSpecification.IoT.Backend.DeviceManagement
  • FIWARE.OpenSpecification.I2ND.CDI
  • FIWARE.OpenSpecification.Data.PubSub

Terms and definitions

This section comprises a summary of terms and definitions introduced during the previous sections. It intends to establish a vocabulary that will be help to carry out discussions internally and with third parties (e.g., Use Case projects in the EU FP7 Future Internet PPP). For a summary of terms and definitions managed at overall FI-WARE level, please refer to FIWARE Global Terms and Definitions

Annotations refer to non-functional descriptions that are added to declaration of native types, to IDL interface definition, or through global annotations at deployment time. The can be used to express security requirements (e.g. "this string is a password and should be handled according the security policy defined for password"), QoS parameters (e.g. max. latency), or others.
AR → Augmented Reality
Augmented Reality (AR)
Augmented Reality (AR) refers to the real-time enhancement of images of the real world with additional information. This can reach from the rough placement of 2D labels in the image to the perfectly registered display of virtual objects in a scene that are photo-realistically rendered in the context of the real scene (e.g. with respect to lighting and camera noise).
IDL → Interface Definition Language
Interface Definition Language
Interface Definition Language refers to the specification of interfaces or services. They contain the description of types and function interfaces that use these types for input and output parameters as well as return types. Different types of IDL are being used including CORBA IDL, Thrift IDL, Web Service Description Language (WSDL, for Web Services using SOAP), Web Application Description Language (WADL, for RESTful services), and others.
Middleware is a software library that (ideally) handles all network related functionality for an application. This includes the setup of connection between peers, transformation of user data into a common network format and back, handling of security and QoS requirements.
PoI → Point of Interest
Point of Interest (PoI)
Point of Interest refers to the description of a certain point or 2D/3D region in space. It defines its location, attaches meta data to it, and defines a coordinate system relative to which additional coordinate systems, AR marker, or 3D objects can be placed.
Quality of Service (QoS)
Quality of Service refers to property of a communication channel that are non-functional, such a robustness, guaranteed bandwidth, maximum latency, jitter, and many more.
Real-Virtual Interaction
Real-Virtual Interaction refers to Augmented Reality setup that additionally allow users to interact with real-world objects through virtual proxies in the scene that monitor and visualize the state in the real-world and that can use services to change the state of the real world (e.g. switch lights on an off via a virtual button the the 3D scene).
A Scene refers to a collection of objects, which are be identified by type (e.g. a 3D mesh object, a physics simulation rigid body, or a script object.) These objects contain typed and named data values (composed of basic types such as integers, floating point numbers and strings) which are referred to as attributes. Scene objects can form a hierarchic (parent-child) structure. A HTML DOM document is one way to represent and store a scene.
Security is a property of an IT system that ensures confidentiality, integrity, and availability of data within the system or during communication over networks. In the context of middleware, it refers to the ability of the middleware to guarantee such properties for the communication channel according to suitably expressed requirements needed and guarantees offer by an application.
Security Policy
Security Policy refers to rules that need to be fulfilled before a network connection is established or for data to be transferred. It can for example express statements about the identity of communication partners, properties assigned to them, the confidentiality measures to be applied to data elements of a communication channel, and others.
Synchronization is the act of transmitting over a network protocol the changes in a scene to participants so that they share a common, real-time perception of the scene. This is crucial to implementing multi-user virtual worlds.
Type Description
Type Description in the context of the AMi middleware refers to the internal description of native data types or the interfaces described by an IDL. It contains data such as the name of a variable, its data type, the hierarchical relations between types (e.g. structs and arrays), its memory offset and alignment within another data type, and others. Type Description are used to generate the mapping of native data types to the data that needs to be transmitted by the middleware.
Virtual Character
Virtual Character is a 3D object, typically composed of triangle mesh geometry, that can be moved and animated and can represent a user's presence (avatar) in a virtual world. Typically supported forms of animation include skeletal animation (where a hierarchy of "bones" or "joints" controls the deformation of the triangle mesh object) and vertex morph animation, where the vertices of the triangle mesh are directly manipulated. Virtual character systems may support composing the character from several mesh parts, for example separate upper body, lower body and head parts, to allow better customization possibilities.
WebGL → (Web Graphics Library) is a JavaScript API for rendering 3D and 2D computer graphics in web browser.
Personal tools
Create a book