We use proprietary and third party's cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Security-Monitoring: Service Level SIEM Open API Specification - FIWARE Forge Wiki

Security-Monitoring: Service Level SIEM Open API Specification

From FIWARE Forge Wiki

Jump to: navigation, search


Introduction to the Service Level SIEM Open Specifications

Please check the Legal Notice to understand the rights to use FI-WARE Open Specifications.

This API is deprecated.

Service Level SIEM API Core

A Security Information and Event Management (SIEM) solution is a technology that provides real-time analysis of security events, aggregating data from many sources and providing the ability to consolidate and correlated monitored data to generate reports and alerts. OSSIM (Open Source Security Information Management - http://www.ossim.net), developed and maintained by AlientVault (http://www.alientvault.com), is one of the most widely used Open Source SIEM. The Service Level SIEM (SLS) component included in the Security Monitoring GE is build on top of the open source OSSIM SIEM v4.0. But to overcome its performance limitations and to allow the processing of more complex rules and correlate events at different layers, the Service Level SIEM integrate that OSSIM core engine so the events already normalized go to a Storm cluster where several processes has been included in a running topology for correlation in a parallel and distributed manner. The following picture summarizes the Service Level SIEM architecture:

Service Level SIEM Architecture

This page provides a description of the Service Level SIEM specifications with the collection functionality (collection methods and plugins) heritated from OSSIM and the processes included in the Storm topology that makes it possible the high-performance correlation engine.

Intended Audience

This specification is addressed for both software developers and service providers that will need advanced monitoring features in their environments.

The Security Monitoring GE will include a Service Level SIEM component based on the open source OSSIM SIEM (Security Information and Event management) that will overcome its limitations with a high performance correlation engine.

API Change History

The most recent changes are described in the table below:

Revision Date Changes Summary
April 2012
  • Initial version
September 2012
  • Reviewed initial version
April 2013
  • Second version

How to Read This Document

The following list summarizes these special notations.

  • A bold, mono-spaced font is used to represent code or logical entities, e.g., HTTP method (GET, PUT, POST, DELETE).
  • An italic font is used to represent document titles or some other kind of special text, e.g., URI.
  • The variables are represented between brackets, e.g. {id} and in italic font. When the reader find it, can change it by any value.

For a description of some terms used along this document, see the Architecture Description document.

Additional Resources

Additional information about OSSIM SIEM open source solution can be found on the official OSSIM - AlienVault Technical Documentation Web Page:


Additional information about Storm distributed and fault-tolerant realtime computation can be found on: http://storm-project.net/

Additional information about how to create your own complex rules and plugins to detect attacks can be found in Service Level SIEM User and Programmers Guide.

General Service Level SIEM Specification Information

Event Collection

Collection is one of the first things required in a SIEM. Data collection is firstly done as a result of the Agents installed in Sensors. Each server can receive data from different sources, but data collection is only possible from:

  • Agents: These are the main sources for incoming events.
  • Other Server: This is only possible in multi-level architecture.

In the FI-WARE context, the OSSIM SIEM core will receive events coming from the Heterogeneous event normalization service component included in the Security Monitoring GE. Consequently. Our main goal here is to describe the specification of the OSSIM agents which collect the incoming events to be proccessed by the SIEM.


The OSSIM agents are the components responsible for collecting all the data sent by the various devices existing on the network, in order to subsequently send it to the OSSIM server in a standardized way.

The agents are installed on the sensor machines, normally one per machine although it is possible to install more than one if necessary. This will normally only occur in multi-level environments, where one machine with several agents can be sending information to various different servers, each from different devices.

The way in which the agent receives the data (which will then be converted into events for OSSIM) that it is going to send to the server is by means of reading a log file in most cases. The ports to which the agent is connected are:

Port number Use

Normally the port of the OSSIM server to which they are connected


DB port to which it is connected for monitor requests


Port where the AgentOSSIM_Spout process in the Storm cluster will receive normalized ossim events for its correlation

These ports can be configured in the file /etc/ossim/agent/config.cfg. The port that is the input to the Storm cluster must be the same configured in the {Storm_install_dir}/conf/ServiceLevelSIEM.conf

Each of the events received by the OSSIM server has always been processed beforehand by an agent in order to standardize them. The point of standardizing events prior to sending them to the server is so that the latter can deal with these events equally and so that storage and processing is simpler and more coherent.

For any device from which one wishes to collect data a plugin has to have been created beforehand so that OSSIM is capable of processing it. This is achieved thanks to the creation of a plugin which basically consists of a series of regular expressions and a list that allows the event type being produced to be unambiguously identified, including Reliability evaluations.


Plugins are each of the elements defined in the Agent to analyze and standardize the information from a device. Once this has been standardized it is passed to the remaining functionalities of the Agent in order to be sent to the OSSIM server in the form of an event.

In OSSIM there are two types of plugins:

  • Detectors: Their job is to read from the logs that store the devices and to standardize them so that the Agent can send them to the OSSIM server. Detector plugins passively read a file, socket or process and send events upon pattern-matching lines.
  • Monitors: These plugins will receive a question from the OSSIM server and send it to the corresponding tool; then as they obtain the reply let the server know whether it agrees or not with what it has asked. Sample monitor plugins would be:
  • Nmap: It receives a monitor request, launches an nmap scan against a specific host:port pair and returns a message stating the open/filtered/closed status of the requested pair.
  • OSSIM C/A: After receiving a Compromise or Attack status request the agent watches for those values inside the OSSIM database, returning and event after having reached it or after the timeout expires.
  • Tcptrack: OSSIM Server asks these agents for specific TCP session information such as duration and bytes sent/received.

In general, each of the plugins can read and send data from a specific device identified by its plugin_id and each event type belonging to that plugin is identified as its plugin_sid.

The plugins consist of two basic files, one with its configuration, and another with the information that the OSSIM server needs in order to correlate the events subsequently. In order to create a new plugin, it will only be necessary to create these two files as specified in the documentation. One of the most important parts is to create the regular expression which must correspond to the one in the log file of the device for which we are creating the plugin. Both the server and the plugin have to agree on what each plugin_id and plugin_sid of each event means; both files are therefore inseparable and it is essential to have both in order for the plugin to work.

In the Service Level SIEM a new plugin called "fiware.cfg" has been created to normalize events coming from FI-WARE Generic Enablers. More information about this plugin can be found in the Service Level SIEM User and Programmers Guide.

Collection Methods

There are several ways of collecting information in OSSIM and it is important to know which ones will be used in order to configure the agents and the plugins required to proccess the incoming data. The most common ways are:

  • Syslog
The device from which the logs wish to be extracted can inject information directly into the syslog of an OSSIM sensor. An agent will be active in this sensor to read from this syslog, and will standardize the events so that they can be sent to the server on which it depends.
  • SNMP
An agent can receive events in SNMP format. Anyway, in order to receive them from any device, it will be necessary to install in the sensor which is going to receive the data, additional software to establish the connection and make the sensor be able to understand this protocol. This software is available on SNMP Sourceforge web site.
  • Log Files
In the same way as with Syslog and SNMP, an agent can be configured in order to read from any log file once a dedicated plugin has been configured for this purpose.
  • Osiris: Unix HIDS
Osiris is a Host Integrity Monitoring System that periodically monitors one or more UNIX hosts for change. It maintains detailed logs of changes to the file system, user and group lists, resident kernel modules, etc.
It is possible to define both an agent and a plugin to extract information from UNIX machines by accessing Osiris stored information.
  • Snare: Collecting from Windows
Snare is the method OSSIM uses to extract information from a windows box. Each Windows host with snare agent installed must be able to send UDP port 514 data towards an OSSIM sensor. Then Windows events are normalized into the OSSIM nomenclature and sent to the ossim-server.
  • FW1LogGrabber: Collecting from Checkpoint FW-1
It is also possible, by installing in the sensor machine some additional software (available as part of Checkpoints OPSEC API) to download the logs from the Checkpoint FW-1. These logs, once downloaded and stored in the sensor hard drive, will be read from a plugin, exactly equal as the other plugins in the Agent.

In the Service Level SIEM, the Syslog collection method is used to receive events coming from FI-WARE Generic Enablers in order to be processed and correlated.

SIEM Event Description

OSSIM defines four types of events that are recognized by the server:

  • Normalized event
  • Mac event
  • OS event
  • Service event

The events received will be treated in a different way depending on the type of data. Plugins should parse events from different sources to these standardized ones, typically to the first of them as the other three are dedicated for special situations. Consequently, any developer who wants to implement a new plugin in compliance with the SIEM provided by the FI-WARE Security Monitoring GE should take into consideration this event description.

In the Service Level SIEM, only OSSIM normalized event are considered for its parsing and later correlation to generate alarms. The fields of which the standardized event consists are detailed in the table below:

Field name Description

Type of event: detector or monitor


Date on which the event is received from the device


IP address of the sensor generating the event




Identifier of the type of event generated


Class of event within the type specified in plugin_id




Three types of protocol are permitted: TCP, UDP, ICMP


IP which the device generating the original event identifies as the source of this event


Source port


Ip which the device generating the original event identifies as the destination of this event


Destination port


Event data that the specific plugin considers as part of the log and which is not accommodated in the other fields. Due to the Userdata fields, it is used increasingly less


Normally stores the event payload, although the plugin may use this field for anything else


User who has generated the event or user with whom it is identifying mainly used in HIDS events


Password used in an event (HIDS events)


File used in an event, mainly used in HIDS

Userdata 1 to 9

These fields can be defined by the user from the plugin. They can contain any alphanumeric information, and on choosing one or another, the type of display they have in the event viewer will change. Up to 9 fields can be defined for each plugin

High-performance Event Processing

Once the events have been collected and normalized by the OSSIM Agents they must be filtered and processed. In order to overcome OSSIM limitations in performance, the Service Level SIEM developed in FI-WARE integrate the filter and correlation processes in a Storm Cluster.

Storm (http://storm-project.net/) is a free and open source distributed realtime computation system that allows processing the events in a scalable, distributed and fault-tolerant way. It is based on two main concepts: Storm cluster and Storm topologies.

Storm cluster

A Storm cluster is basically a set of nodes (hosts) where the processing tasks are distributed according to a predefined role. There are two types of nodes:

  • Master node: it runs a daemon called Nimbus. This process is responsible for distributing code around the cluster, assigning tasks to machines and monitoring for failures.
  • Worker nodes: each worker node runs a daemon called Supervisor. This process listens for work assigned to its machine and starts and stops worker processes as necessary based on the tasks the master node has assigned to it.

The Nimbus and Supervisor daemons are fail-fast and stateless and for that reason the coordination between them and the state information is maintained through a Zookeeper Cluster (http://zookeeper.apache.org/). So, in case any of them is shutdown, it is automatically started maintaining the cluster stable. Furthermore, the Zookeeper servers will be executed under a supervisory process such as daemontools (http://cr.yp.to/daemontools) to ensure that if the process does exit abnormally, it will automatically be restarted and will quickly rejoin to the cluster.

Each worker executes a subset of what is called topology. A topology in Storm's terminology is a graph of computation. It consists of a set of data sources (spout) and data operations (bolts) connected with stream grouping. Each node (spout/bolt) in a topology contains processing logic and links between nodes (stream grouping) indicate how data should be passed around between nodes. Consequently, there are two types of nodes or abstractions in a topology:

  • Spout: it is a source of streams in the topology (for example, reading data from a file or listening for incoming data in port). It provides the interface to make this process reliable, ensuring to resend a tuple (which is an ordered list of data items) in case Storm fails to process it.
  • Bolt: it is responsible for all the data processing (from filtering to joins, aggregations, reading and writing to files or databases, etc) that takes place in the topology. A bolt consumes any number of input streams (from a spout or other bolt) and produce new output streams (emit tuples to another bolt).

A stream in Storm is an unbounded sequence of tuples. Spouts and bolts have interfaces that must be implemented to run an application specific logic. The Spouts are the processes that generate the input tuples (for example listening in a port for incoming events or reading lines from a file) and the Bolts are processes that transform those streams and generate other ones depending on the specific logic of the running application. Depending the complexity of the stream transformation, it can be necessary to have multiples spouts and bolts and the network of them will be packaged into a specific "topology" to be run in the Storm cluster.

A stream grouping tells a topology how to send tuples among sets of bolt's task. It is important to remark that spouts and bolts will be executed in parallel as many tasks across the cluster. Consequently, the flow of streams between them must be correctly configured to guarantee fully message processing according the application logic. There are several built-in stream groupings included in Storm but the two main used in the Service Level SIEM are:

  • Shuffle grouping: in this case, tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
  • Fields grouping: in this case, the stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id" may go to different tasks.

Finally, we could summarize the following advantages of running the Service Level SIEM as a topology in a Storm cluster:

  • Guaranteed message processing: each tuple coming off the OSSIM agents will be fully processed in a distributed manner. Storm provides the capability to track the tree of messages that a tuple triggers in an efficient way so in case of any failure, the tuple can be resent from the spout.
  • Robust process management with the use of Storm, Zookeeper and Supervisor running together.
  • Fault detection and automatic reassignment: tasks in a running topology heartbeat to Nimbus to indicate that they are running smoothly. Nimbus monitors heartbeats and will reassign tasks that have timed out. Additionally, all the tasks throughout the cluster that were sending messages to the failed tasks quickly reconnect to the new location of the tasks.
  • Efficient message passing: messages are passed directly between tasks using ZeroMQ without intermediate queuing.

More information about Storm can be found in its official page http://storm-project.net/

Service Level SIEM Topology

In order to run the Service Level SIEM in a Storm cluster to have high-performance and be able of processing and correlating pattern of events from a more complex and business perspective, the following topology has been defined:

Service Level SIEM Topology
  • AgentOSSIMSpout

This spout is listening in a predefined port (41000/tcp by default) for events coming from the OSSIM Agents. These events will be already normalized to the ossim event format. This Spout emits tuples with only a field called 'ossimEvent' that includes the event sent by the ossim-agent. The listening port can be configured in the conf/ServiceLevelSIEM.conf file.

  • GE_FilterBolt

This bolt process will make a first event preprocessing detecting the FI-WARE GE that generated them. With this purpose, the userdata1 field in the ossim event must include the name of the device that generated it. This process receives the output from the AgentOSSIMSpout and emits tuples with two fields: "FIWARE_GE" and "event".

  • PRRS_FilterBolt

At this step of the topology it is possible to have different bolt processes depending on the event processing or filtering required by each FI-WARE GE that sent the events. Like an example, the PRRS_FilterBolt is implemented to process events coming from the Context-based Security & Compliance GE. This bolt process included in the Storm cluster (with parallelism configurable by file) uses an eventSchema file to create output tuples only with the relevant fields for the correlation. The conf/schemas/eventSchema file includes a line that starts with the name of the Generic Enabler (for example "PRRS:") and then a list separated by commas of pairs: <ossim_field>=<output_field>. The ossim fields are recovered from the "event" field received in the incoming tuples and the output fields are the one emitted by this process to the following one in the topology. This bolt is connected to the previous GE_FilterBolt with a "fields grouping" through the first field "FIWARE_GE" so all the events coming from a same Generic Enabler can be correlated.

  • CorrelationBolt

This bolt is the process included in the Service Level SIEM topology that will perform the correlation of the tuples emitted by the previous filter bolt that contains only the relevant fields for the specific FI-WARE GE. In the case of the events coming from the Context-based Security & Compliance GE, they arrive with a field grouping based on the field "securitySpecs", but this is configurable depending on the event source. The language chosen for the correlation that will arise alarms from a business perspective in this Service Level SIEM topology is the Event Processing Language (EPL). For this reason, the Esper (http://esper.codehaus.org/) libraries are used in this correlation bolt process.

  • DBWriterBolt

This bolt process included in the Service Level SIEM topology is the one in charge of writing the alarms received from the correlation bolt into the OSSIM Database. The database information is read from the conf/ServiceLevelSIEM.conf file. In this case, a Shuffle Grouping is used because each tuple received from whatever correlation bolt represents an alarm to be written immediately in the database to be available for the Visualization Framework. In the current implementation, these alarms are stored in the OSSIM database in a table called sls_alarm with the following fields:

Field name Description

It is the primary key that identifies the alarm generated


It identifies the type of alarm. All available type of alarms are stored in a table called sls_rule


Short description of the alarm


Timestamp when the alarm was stored in the database


Identifier of the first event included in the pattern that generated the alarm. The information about this event (for example timestamp, source_ip, destination_ip, etc.) is stored in the table Event with all the other events received by the OSSIM Agents.


Identifier of the last event included in the pattern that generated the alarm. The information about this event (for example timestamp, source_ip, destination_ip, etc.) is stored in the table Event with all the other events received by the OSSIM Agents.

In order to make it more flexible, an alarmScheme file is used to define the list of fields to be recovered from the tuples received from the Correlation Bolts. This file includes the list of those fields separated by commas. For example:

AlarmID, AlarmMSG, FirstEvent, LastEvent

Besides, the file sqlsAlarmsDB.conf is used to define the list of SQL commands to interact with the database. Currently it includes the following one:

SQLInsertSLSAlarm = INSERT INTO sls_alarm (rule_id, msg, timestamp, firstEvent_id, lastEvent_id) values ({0},''{1}'',now(),0x{2},0x{3});

Service Level Correlation Rules

The language chosen for the correlation rules that will arise alarms from a business perspective in this Service Level SIEM topology is the Event Processing Language (EPL). This language is quite simple because it is similar to the well-known SQL in the use of the select and where clause. But it also allows expressing rich event conditions or patterns, correlation and time windows. The EPL statements, variables and the correlation rules that will trigger alarms are defined through the following configuration files that will be used by the correlation bolt process included in the topology:

  • EPLVariables.conf

This file is used to define variables to be used in the EPL statements. They depends on the correlation to be performed to arise alarms from a service level or bussiness perspective. Each line represents a variable and it is composed by three parts separated by comas: <variable_name, variable_type, variable_value> For example:

AlarmMSG,string,FI-WARE additional security level compromised in client environment
  • EPLStatements.conf

This file is used to define statements in EPL language to be used in the correlation process. It is important to remark that these statements will not generate alarms in the correlation process. They will be used for example to define the objects (events) to be used in a pattern of events in the correlation rule. They can be also used to define variables (instead the EPLVariables.conf file) or any other EPL statement required for the correlation. For example:

insert into PRRSEventA select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "non-compliance")
insert into PRRSEventB select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "undeployed")
insert into PRRSEventC select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "not-found")
insert into PRRSEventD select * from filterBolt_default where (FIWARE_GE = "prrs") and (status = "deployed")
  • ServiceLevelSIEM.conf

The Service Level SIEM configuration file includes a parameter to defined the correlation rule that will generate the alarm. This statement will have asocciated a listener in the correlation process what means that when the condition included in it happens, a tuple will be emitted to the following process in the Service Level SIEM topology to write that alarm in the database.

correlation_rule = select AlarmID, AlarmMSG, a.event_id as FirstEvent, c.event_id as LastEvent from pattern [every (a=PRRSEventA) -> b=PRRSEventB(client_ip=a.client_ip and sec_enabler=a.sec_
enabler) -> c=PRRSEventC(client_ip=a.client_ip and securitySpecs=a.securitySpecs) where timer:within(var_timeout sec)]

In our example this correlation rule is a pattern of events but it could have any other condition depending on the client and the alarm to be generated. It will use the variables and statements defined in files indicated in the EPLVariables and EPLStatements parameters, respectively.

The parameters alarmMSGToEmit, alarmIDToEmit, eventToEmit1 and eventToEmit2 are the name of the fields that will be emitted by the correlation process. Those names can be modified depending on the correlation rule defined but they must be the same included in the alarmSchema file used by the DBWriteBolt process.

More information about the EPL language can be found in: http://docs.oracle.com/cd/E13157_01/wlevs/docs30/epl_guide/overview.html

Personal tools
Create a book