Fluentd

[ 1 Install the Fluentd plugin ] [ 2 Customize the Fluentd configuration file ] [ 3 Set up rules on your Devo relay ]

Fluentd is an open source data collector that you can use to collect and forward data to your Devo relay. Complete documentation for using Fluentd can be found on the project's web page.

If you're already familiar with Fluentd, you'll know that the Fluentd configuration file needs to contain a series of directives that identify the data to collect, how to process it, and where to send it. Several different types of plugins may be used to identify and configure inputs, outputs, and processing tasks. This article assumes a competency with Fluentd and will address only the parts of the configuration file needed to send data to the Devo Relay where you will create the rules that apply the necessary Devo tags.

Install the Fluentd plugin

First you need to obtain a plugin that outputs your data in syslog format which, as you know, is the standard that Devo uses. We recommend using the remote_syslog plugin. Click here to download the plugin, then follow the installation instructions you find in the GitHub readme file.

Although Devo has tested and confirmed this plugin, it was built and is maintained by a third party.

Customize the Fluentd configuration file

The following excerpt from a sample Fluentd configuration file contains a source directive and a match directive.

In the source directive you specify what files to read and how to read them. Of particular importance is the tag parameter. Not to be confused with the syslog tag, this parameter simply identifies the match directive to use for forwarding the events collected.

In the match directive, the @type parameter needs to call the remote_syslog plugin you already installed. The host and port parameters should be customized for your Devo relay. We recommend using tcp as the protocol and automatically assigning the hostname as indicated in this sample file.

Referencing the remote_syslog plugin in the Match directive - Fluentd .conf

<source>
  @type tail
  path /logs/apache/*.log
  pos_file /logs-positions/source/apache-access.log.pos
  tag devo_relay
  refresh_interval 10
  read_from_head true
  enable_stat_watcher false
  encoding "utf-8"
  <parse>
    @type apache2
  </parse>
</source>


...


<match devo_relay>
  @type remote_syslog
  host devo_relay_ip
  port 13013
  protocol tcp
  hostname "#{Socket.gethostname}"
</match>

Keep in mind

The tag parameter in the source directive should name a specific match directive that forwards events to a port of the Devo relay. In the example, this is "devo_relay".
If appropriate for the data you're sending, you may create several match directives, one for each port you're sending to on your Devo relay.
Plan how you will build your relay rules before you edit the Fluentd configuration file.

Set up rules on your Devo relay

With the configuration file prepared, you need to establish rules for the port (or ports) on the Devo relay that will receive the events.

Pay close attention to the port(s) you assign in the Fluentd configuration file and be sure they correspond to the rule(s) you create.

How you set up your rules will vary depending on the type of data you are sending. If you only have to apply a single Devo tag to all events you are sending to the relay, check out relay rule scenario 1. However, if you need to apply multiple tags depending on content present the syslog message, then check out relay rule scenario 4.

To learn all about relay rules, start here.