Logstash
- Juan Tomás Alonso Nieto (Deactivated)
Logstash is an open source tool for collecting, parsing and storing logs for future use. It ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite repository (in this case, Devo).
The procedures in the article assume a general working knowledge of this tool. To resolve questions about using Logstash, please visit the Logstash Reference Guide on the Elastic website.
In this article you will learn about:
About Logstash configuration files
For every data source that sends events to Logstash, there must be a corresponding pipeline configuration (.conf) file in the /etc/logstash/conf.d directory.
In order to forward a source's events from Logstash onward to a Devo Relay or to Devo directly, you will have to edit its corresponding pipeline configuration file. Specifically, by creating a syslog-type output section in the configuration file. This requires the logstash-output-syslog plugin. Make sure that this Logstash plugin is installed before you use these instructions.
Sending from Logstash to a Devo relay
To forward events to a Devo Relay, Logstash must be installed on the same machine as the relay.
To set up the forwarding, simply create a syslog plugin in the output section of the configuration file. Here, we not only specify the routing information but we also apply the Devo tag to the events. Here's an excerpt of a sample configuration file showing the output section.
input {...}
output {
syslog {
facility => "local7"
severity => "informational"
host => "localhost"
port => 13000 ### forwarding port 13000
sourcehost => "syslogHostname" ### syslog message hostname
appname => "av.mcafee.epo.events" ### Devo tag
protocol => "tcp"
}
}
You can get details about each of the parameters contained in this example in the Logstash Reference Guide, but note that:
- The port is 13000. This is because the events will be delivered to the relay already tagged and no further processing is required. More about default relay rules here.
- The appname parameter is where we specify the Devo tag to apply to these events.
Once you've edited the .conf file, reactivate the configuration with:
./bin/logstash -f <filename>.conf
Sending from Logstash to the Devo Cloud
In this case, we will be sending data over the internet so it is necessary to establish a secure channel using the Devo domain's SSL certificates. Download the certificate files from Administration → Relays and save them to a directory on the machine where Logstash is installed.
The configuration of the syslog plugin directs the events to the Devo Cloud (for your region, in this case Europe), applies the Devo tag, and references the SSL certificates.
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => "443"
appname => "my.app.logstash.test" #SPECIFY THE DEVO TAG HERE
protocol => "ssl-tcp"
ssl_cert => "domain.crt"
ssl_key => "domain.key"
ssl_cacert => "chain.crt"
}
}
Again, you can read more about each of the parameters contained in this example in the Logstash Reference Guide, but note that:
- The host specifies the address of the Devo Cloud for the region you are using. It should be one of:
- USA: collector-us.devo.io
- Europe: collector-eu.devo.io
- Spain: collector-es.devo.io
- The port is 443 because this is the inbound port used for sending to the Devo Cloud.
- The appname parameter is where we specify the Devo tag to apply to these events.
Once you've edited the .conf file, reactivate the configuration with:
./bin/logstash -f <filename>.conf
Some configuration examples
Below we provide some sample configuration files for some cases where Logstash can be used to forward events from different kinds of data sources.
Forwarding from a database with JDBC
This configuration will send an event to Devo each time a new record appears in the users table of the specified database.
In this example, the input section uses the jdbc plugin to collect input:
- from a database specified by the jdbc_connection_string parameter
- with a frequency defined by the schedule parameter
- using the SQL statement set in the statement parameter
Read more about the jdbc input parameters here.
The output section forwards all new records as events to the Devo Cloud.
input {
jdbc {
jdbc_driver_library => "/Users/Bob/logstash/ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@192.168.1.33:1521:test"
jdbc_user => "system"
jdbc_password => "password"
schedule => "* * * * *"
tracking_column => id
use_column_value => true
statement => "select * from users where id > :sql_last_value"
}
}
output {
syslog {
facility => "local7"
host => "ecollector-eu.devo.io"
port => "443"
severity => "informational"
appname => "my.app.oracle.test"
protocol => "ssl-tcp"
ssl_cert => "/Users/Bob/logstash/ca/domain.crt"
ssl_key => "/Users/Bob/logstash/ca/domain.key"
ssl_cacert => "/Users/Bob/logstash/ca/chain.crt"
codec => line {
format => "%{name},%{id}"
}
}
}
Forwarding keyboard input
This configuration will send an event to Devo each time a user types something using the keyboard and then presses Enter.
In this example, the input section uses the stdin plugin to collect input from the keyboard. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates.
input {
stdin { }
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => 443
protocol => "ssl-tcp"
sourcehost => "macbook_121"
appname => "my.app.test.logstash"
ssl_cacert => "chain.crt"
ssl_cert => "domain.crt"
ssl_key => "domain.key"
}
}
Forwarding from a file
This configuration will send events to Devo each time a file is updated with new information.
In this example, the input section uses the file plugin to collect input from a file. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates.
input {
file {
path => "/Users/Ramon/logstash/logs/*"
start_position => "beginning"
sincedb_path => "/Users/Ramon/logstash/dbfile"
}
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-eu.devo.io"
port => "443"
appname => "my.app.ramon.logstash"
protocol => "ssl-tcp"
ssl_cert => "/Users/Ramon/logstash/ca/domain.crt"
ssl_key => "/Users/Ramon/logstash/ca/domain.key"
ssl_cacert => "/Users/Ramon/logstash/ca/chain.crt"
}
}
Forwarding from an Apache Kafka topic
This configuration will send events to Devo that are read from an Apache Kafka topic.
In this example, the input section uses the kafka plugin to collect input from a Kafka topic. The output section sends this input to the Devo Cloud using SSL-TCP. Security is enabled through the use of authentication certificates. It also sends events to standard output.
input {
kafka {
group_id => "test-kafka"
topics => ["devotest"]
bootstrap_servers => "localhost:9092"
}
}
output {
syslog {
facility => "local7"
severity => "informational"
host => "collector-us.devo.io"
port => "443"
appname => "my.app.kafka.topic"
protocol => "ssl-tcp"
ssl_cert => "/home/devo/domain.crt"
ssl_key => "/home/devo/domain.key"
ssl_cacert => "/home/devo/chain.crt"
}
stdout{codec => rubydebug }
}