We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Roadmap of Data/Context Management R5 - FIWARE Forge Wiki

Roadmap of Data/Context Management R5

From FIWARE Forge Wiki

Jump to: navigation, search

Contents

Introduction

The Data/Context chapter in FIWARE provides key assets enabling high performance Data, Context and Events acquisition, processing, analysis and exploitation.

You can learn more about the Data/Context Chapter by reading the FI-WARE Architecture and Open Specifications. Following is a description of the Technical Roadmap planned for the chapter, which will be developed through subsequent releases of the FIWARE Platform. Please also check the Releases and Sprints numbering, with mapping to calendar dates.

Fifth Major Release

The planned date for the fifth major release of the project is September 2016. The main features and epics from the different GEs in the Data chapter, that are planned to be available in this release can be found below, as part of this section. This list of features and epics is based on the latest roadmap analysis that has been performed for each of the GEs in the chapter. This roadmap analysis effort has also been reflected in our development activities tracking system (Jira), and might be subject to minor changes if new requirements are identified until the planned date for this release.

The list of the main planned features and epics for each of the GEs that are part of the Data/Context chapter can be found below:

Publish / Subscribe Broker (Orion)

  • Extend the set of FIWARE NGSIv2 APIs and their flexibility, in particular:
    • Accept empty attribute values in UpdateContext operation: this will allow to update the attribute metadata, without having to update the attribute value itself.
    • Enable the subscription to multiple entities at a time, based on filtering and/or the FIWARE service/subservice they belong to.
    • Allow subscribing to an entity leaving the conditions empty, therefore being notified on any change to the entity's parameters, including those that were not present at the time of subscription.
    • Enable the creation of subscriptions without expiration date.
    • List available registrations, entities, entity attributes, subscriptions, subscription filters, filtered entities and supported operations in the Context Broker.
    • Enable all single entity related NGSIv2 operations, based on their id and type as unique identifiers.
    • Add the fiware service/subservice concept in the entity query parameters and subscription notifications.
    • Enable listing the available FIWARE service/subservice tree.
    • Enable changing the FIWARE service/subservice in entities and registrations.
    • Enable massive entities deletion.
  • Support websockets in the Context Broker.
  • Allow retries in the subscription notifications.
  • Support CORS in the Context Broker APIs.
  • Support location based subscriptions.
  • Improve robustness (error management) and efficiency.
  • Add optional parameters for: avoiding initial notification at subscription creation time, returning entity/attribute creation and update timestamp, validating empty subscriptions, deactivation for not listening notification endpoints.


Complex Event Processing (Proton)

The goal of this release is to increase the usability of the CEP. This includes:

  • Enhance existing pattern usability to ease the implementation of advanced scenarios. Thos additions are based on user requests. In particular:
    • Enhance the TREND pattern, to detect a trend pattern only if a given ratio over the detected trend was preserved.
    • Allow the user to refer to the attributes of the pattern's participant events, even in patterns that usually do not keep the participant events in memory (such as trend and aggregation).
  • Support running the CEP on historical events, by injecting those historical events to the engine and detecting patterns as if the events occurred in present time, while reducing the processing time in comparison to the actual historical event time duration. This can also be useful for simulation purposes.
  • Implement new patterns according to analysis of scenarios from various domains. Currently we plan to implement the Relative N pattern that allows to detect N events in a given time window that have the highest or lowest attribute values. Implementation of a new pattern requires both engine support and UI support.


BigData Analysis (Cosmos)

  • Hadoop clusters as a service (continuing the work from 4th release)
    • Deploy a global instance of Sahara in FIWARE Lab
  • Spark clusters as a service (continuing the work from 4th release)
    • Deploy a shared cluster in FIWARE Lab
    • Deploy the Spark plugin in FIWARE Lab’s Sahara
  • NGSI-like repositories as source of input data for Hadoop (continuing the work from 4th release)
    • Extend the Hadoop API with a new NGSI input format
  • Swift as source of input data for Hadoop (continuing the work from 4th release)
    • Extend the Hadoop API with a new Swift input format
  • Sinfonier integration (continuing the work from 4th release)
    • Deploy a global instance of Sinfonier in FIWARE Lab as a Storm studio
    • Create a catalogue of NGSI-like Storm bolts
  • NGSI integration (continuing the work from 4th release)
    • Binary version of HDFS sink
    • Batch processing on the sink side
    • New sink for DynamoDB
  • Cosmos GUI
    • Calendar-like visualization tool showing the available analysis slots in terms of infrastructure and time
    • Administration page
    • File system browser
    • MapReduce uploading and execution page


Stream Oriented (Kurento)

  • Extend the enabler to support latest trends on media technologies
    • Media renegotiation
    • Track grouping for synchronization and other purposes
    • DataChannels consumption on filters
    • Filter communication of synchronized meta data
    • Multi-domain filters acting as sensors publishing events throughd ata channels.
  • Improve the interoperability and the transport protocols of the enabler
    • Support for legacy SDES key negotiation
    • Support for configurable port ranges
    • Support for latest trends on Interactive Connecitity Establishment
  • Improve the efficiency of the enabler
    • Profile the enabler under different load conditions
    • Find performance bottlenecks
    • Optimize code on the appropriate bottlenecks


Short Term Historic Open Data Repository (CKAN)

  • Service integrations
    • CKAN to support DCAT-AP (v1.1)
    • OASC support
  • Platform updates
    • Migrate away from the Pylons web framework to something more modern
    • Make it easy to install, add and configure CKAN extensions from within CKAN
  • Releases
    • Ensure that the FIWARE project is up to date and compatible with with the latest CKAN (versions 2.5 and 2.6) including:
    • Assessment of FIWARE extensions
    • GE update
    • Lab update



Incubated GEs in the chapter

The following three GEs listed below are currently classified as Incubated GEs in the chapter, and may become a FIWARE GE upon decision of the FIWARE Technical Committee in the future:

Social Data Aggregator (SDA)

  • Social Data aggregator consumer module for Sentiment Analysis (both batch and near real time version):
    • Add the capability to detect if a tweet contains positive, negative or neutral opinion
    • Add the capability to detect emotional states on tweets (hungry,sad,happy)
  • Social Data Aggregator Controller module:
    • Capability to add or remove topics by a user through a Graphical User Interface or programmatically by defining criterias
    • Batch analytics management in therm of ease and cluster resources


Social Semantic Enricher (SSE)

  • Enricher Module to enrich extracted concepts with external media
    • Enrichment with Images from different sources
    • Enrichment with video from different sources
    • Enrichment with news from different sources
  • Enricher API module to get the enrichment functionalities via API
    • API for images enrichment
    • API for video enrichment
    • API for news enrichment
  • Api Controller Module to work as "Orchestrator" for other modules API
    • Handler for SSE Core API
    • Handler for SSE Enrichment API
    • Handler for SSE Extratction API
  • Graphical User Interface Modules to demonstrate the enabler functionalities
    • File Analysis to allow file as input
    • Text Analysis to allow using text as input
    • URL Analysis to allow using URLs as input


Cloud Messaging (AEON)

Summary list of supported Features and Epics per GE in the chapter

For a more detailed description of features to be provided, you can refer to the following table which helps to identify which Backlog Features have already been specified and planned for the fifth Major Release of FIWARE.


FIWARE GE Supported Features and Epics Epics under analysis

Publish/Subscribe Broker -Orion

Release 5.1:
FIWARE.Feature.Data.OrionContextBroker.AttributeManagement
FIWARE.Epic.Data.OrionContextBroker.NGSIv2API
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.UpdateContext
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.SubscribeContext
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.ListingContextBrokerInfo
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.QueryContext
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.SingleEntityOperations
FIWARE.Feature.Data.OrionContextBroker.NGSIv2API.EntitiesDeletion

Release 5.2:
FIWARE.Feature.Data.OrionContextBroker.Websockets
FIWARE.Feature.Data.OrionContextBroker.DataManagementOptimizations
FIWARE.Feature.Data.OrionContextBroker.ErrorManagementOptimizations
FIWARE.Feature.Data.OrionContextBroker.CORS

Release 5.3:
FIWARE.Feature.Data.OrionContextBroker.Retries
FIWARE.Feature.Data.OrionContextBroker.LocationBasedSubscriptions
FIWARE.Feature.Data.OrionContextBroker.ServiceSubserviceManagement
FIWARE.Feature.Data.OrionContextBroker.OptionalParameters

Release 5.4:
FIWARE.Feature.Data.OrionContextBroker.NGSI9OneTimeSubscription
FIWARE.Feature.Data.OrionContextBroker.ImproveNGSIOpsFormat
FIWARE.Feature.Data.OrionContextBroker.RegisterWithRegExp


Complex Event Processing - Proton

Release 5.1:
FIWARE.Epic.Data.CEP.PatternUsability
FIWARE.Feature.Data.CEP.PatternUsability.TrendRatio


Release 5.2:
FIWARE.Feature.Data.CEP.PatternUsability.UseParticipantEvents
FIWARE.Epic.Data.CEP.EventPatternDetection
FIWARE.Feature.Data.CEP.EventPatternDetection.HistoricalEvents

Release 5.3:
FIWARE.Epic.Data.CEP.NewPattern
FIWARE.Feature.Data.CEP.NewPattern.RelativeN

Release 5.4:
FIWARE.Feature.Data.CEP.NewPattern.RelativeNUI


BigData Analysis -Cosmos

Release 5.1:
FIWARE.Epic.Data.BigData-Analysis.HadoopAsAService
FIWARE.Feature.Data.BigData-Analysis.HadoopAsAService.Sahara
FIWARE.Epic.Data.BigData-Analysis.NGSIIntegration
FIWARE.Feature.Data.BigData-Analysis.NGSIIntegration.OrionHDFSSinkBinary
FIWARE.Feature.Data.BigData-Analysis.NGSIIntegration.OrionSinkBatchProcessing
FIWARE.Feature.Data.BigData-Analysis.NGSIIntegration.OrionDynamoDBSink

Release 5.2:
FIWARE.Epic.Data.BigData-Analysis.GUI
FIWARE.Feature.Data.BigData-Analysis.GUI.Calendar
FIWARE.Feature.Data.BigData-Analysis.GUI.FileBrowser
FIWARE.Feature.Data.BigData-Analysis.GUI.AdminPage
FIWARE.Feature.Data.BigData-Analysis.GUI.ExecutionPage

Release 5.3:
FIWARE.Epic.Data.BigData-Analysis.NGSIAsInput
FIWARE.Feature.Data.BigData-Analysis.NGSIAsInput.HadoopAPIExtension
FIWARE.Epic.Data.BigData-Analysis.SwiftAsInput
FIWARE.Feature.Data.BigData-Analysis.SwiftAsInput.HadoopAPIExtension


Release 5.4:
FIWARE.Epic.Data.BigData-Analysis.SparkAsAService
FIWARE.Feature.Data.BigData-Analysis.SparkAsAService.SharedCluster
FIWARE.Feature.Data.BigData-Analysis.SparkAsAService.SaharaPlugin
FIWARE.Epic.Data.BigData-Analysis.SinfonierAsStormStudio
FIWARE.Feature.Data.BigData-Analysis.SinfonierAsStormStudio.DeploySinfonierSharedInstance
FIWARE.Feature.Data.BigData-Analysis.SinfonierAsStormStudio.NGSIStormSpoutsCatalogue
FIWARE.Feature.Data.BigData-Analysis.SinfonierAsStormStudio.NGSIStormBoltsCatalogue

Stream Oriented -Kurento

Release 5.1:
FIWARE.Feature.Data.Stream-oriented.AdvancedMedia.AdvancedSdpNegotiation
FIWARE.Feature.Data.Stream-oriented.AdvancedMedia.DataChannelFilter

Release 5.2:
FIWARE.Feature.Data.Stream-oriented.RtcProtocols.SDES
FIWARE.Feature.Data.Stream-oriented.AdvancedMedia.FilterMetadata

Release 5.3:
FIWARE.Feature.Data.Stream-oriented.RtcProtocols.RtpPortRange
FIWARE.Feature.Data.Stream-oriented.Cloud.KmsProvider

Release 5.4:
FIWARE.Feature.Data.Stream-oriented.AdvancedMedia.MultiDomainMedia
FIWARE.Feature.Data.Stream-oriented.RtcProtocols.IceUpdate
FIWARE.Epic.Data.Stream-oriented.AdvancedMedia
FIWARE.Epic.Data.Stream-oriented.RtcProtocols


CKAN

Release 5.1:
FIWARE.Epic.Data.CKAN.Release
FIWARE.Feature.Data.CKAN.GenericEnabler.Catalogue
FIWARE.Feature.Data.CKAN.Release23
FIWARE.Feature.Data.CKAN.MultilingualSupport.DCAT
FIWARE.Feature.Data.CKAN.DOI.ManualRegistration
FIWARE.Feature.Data.CKAN.DOI.AutomaticRegistration
FIWARE.Feature.Data.CKAN.MultilingualSupport.TranslationInterface
FIWARE.Feature.Data.CKAN.MultilingualSupport.MultilingualMetadata
FIWARE.Feature.Data.CKAN.DCAT.Harvest



Release 5.2:
FIWARE.Feature.Data.CKAN.Harvesting.Sevilla
FIWARE.Feature.Data.CKAN.DCAT.DCATAP11
FIWARE.Epic.Data.CKAN.PylonsMigration
FIWARE.Feature.Data.CKAN.Release.25
FIWARE.Feature.Data.CKAN.EasyInstallAndConfiguration.ConfigurableSettings
FIWARE.Feature.Data.CKAN.EasyExtensionInstallAndConfiguration.ConfigurableSettings
FIWARE.Epic.Data.CKAN.DCAT
FIWARE.Feature.Data.CKAN.PylonsMigration.Audit

Release 5.3:
FIWARE.Feature.Data.CKAN.PylonsMigration.ObjectCompatability
FIWARE.Feature.Data.CKAN.PylonsMigration.FlaskMiddleware

Release 5.4:
FIWARE.Feature.Data.CKAN.DCAT.OASCSupport
FIWARE.Feature.Data.CKAN.PylonsMigration.FlaskMigrate


The table below shows the detailed information on the release 5 planned epics and features for the Incubated GEs in the chapter:


FIWARE GE Supported Features and Epics Epics under analysis

Social Data Aggregator

Release 5.1:
No new epics or features planned for this release.

Release 5.2:
FIWARE.Epic.Data.SocialDataAggregator.SentimentAnalysisAlgorithm
FIWARE.Feature.Data.SocialDataAggregator.SentimentAnalysisAlgorithm.ClassifyPolarity

Release 5.3:
FIWARE.Epic.Data.SocialDataAggregator.SentimentAnalysisAlgorithm
FIWARE.Feature.Data.SocialDataAggregator.SentimentAnalysisAlgorithm.ClassifyEmotionalStates

Release 5.4:
FIWARE.Epic.Data.SocialDataAggregator.Controller
FIWARE.Feature.Data.SocialDataAggregator.Controller.AddRemoveTopics
FIWARE.Feature.Data.SocialDataAggregator.Controller.BatchModules


Social Semantic Enricher

Release 5.1:
FIWARE.Epic.Data.SocialSemanticEnricher.Enricher
FIWARE.Feature.Data.SocialSemanticEnricher.Enricher.Image
FIWARE.Feature.Data.SocialSemanticEnricher.Enricher.News
FIWARE.Feature.Data.SocialSemanticEnricher.Enricher.Video

Release 5.2:
FIWARE.Epic.Data.SocialSemanticEnricher.API
FIWARE.Feature.Data.SocialSemanticEnricher.API.Image
FIWARE.Feature.Data.SocialSemanticEnricher.API.News
FIWARE.Feature.Data.SocialSemanticEnricher.API.Video



Release 5.3:
FIWARE.Epic.Data.SocialSemanticEnricher.APIController
FIWARE.Feature.Data.SocialSemanticEnricher.APIController.HandleCoreAPI
FIWARE.Feature.Data.SocialSemanticEnricher.APIController.HandleEnricherAPI
FIWARE.Feature.Data.SocialSemanticEnricher.APIController.HandleExtractionAPI

Release 5.4:
FIWARE.Epic.Data.SocialSemanticEnricher.GUI
FIWARE.Feature.Data.SocialSemanticEnricher.GUI.FileAnalysis
FIWARE.Feature.Data.SocialSemanticEnricher.GUI.TextAnalysis
FIWARE.Feature.Data.SocialSemanticEnricher.GUI.URLAnalysis

Cloud Messaging -AEON

Release 5.1:


Release 5.2:


Release 5.3:


Release 5.4:


Future releases

The following features and epics correspond to features that are in the roadmap of the respective Generic Enablers, but are not going to be implemented as part of FIWARE project activities. Some of them may continue under the FIWARE Community activities, some others will not.


FIWARE GE


Future activities


Publish/Subscribe Broker - Orion Context Broker


Complex Event Processing


BigData Analysis

  • Sinfonier as a batch data studio
    • Library of generic MapReduce bolts for Storm
  • CKAN plugin for Open Big Data
    • Implement Open Big Data uploading capabilities through CKAN
    • Implement Open Big Data visualization sampling through CKAN
    • Implement Open Big Data authorized usage through CKAN

Stream Oriented

CKAN


You can check out the previous releases of this FIWARE chapter on Roadmap of Data/Context Management(previous releases)

Personal tools
Create a book