Document toolboxDocument toolbox

Monitoring files using syslog-ng

Owned by Juan Tomás Alonso Nieto (Deactivated)
May 23, 2022
3 min read

The instructions for setting up connections to a Devo Relay or Cloud involve the editing of the syslog-ng.conf and are centered on the sending of all system log events to a Devo endpoint.

In this article, we explain how to include other log as sources in the syslog-ng.conf file and configure their sending to Devo endpoint destinations. There are a few simple steps:

Edit the syslog-ng.conf file

This configuration file is saved in /etc/syslog-ng/, and consists of a series of source, destination and log object definitions. These are represented in this generic and anonymous sample syslog-ng.conf file. 

source s_myfile {
    file("/path/to/file.log" follow_freq(1) flags(no-parse));};
 
destination d_devo_myfile {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST my.devo.tag: $MESSAGE\n"));};

log { source(s_myfile); destination(d_devo_myfile); };

  • In the source object, you would replace s_myfile with a custom identifier and "/path/to/file.log" with the path including the file name of the file you want syslog-ng to monitor.

  • In the destination object, you would replace d_devo_myfile with a custom identifier, DEVO-RELAY with the relay IP address and PORT with the relay port. Go to Administration → Relays in Devo to see a list of available relays. In the template definition, replace my.devo.tag with the Devo tag to apply to all events coming from the source file. Find the correct tag in the List of Devo parsers.

Here's another sample file, this time showing how you can configure the monitoring of multiple log files - in this case the logs for an Apache Server.

# Apache access log
source s_apache_access {
    file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_access); destination(d_devo_apache_access); };
 
# Apache SSL access log
source s_apache_ssl_access {
    file("/var/log/apache2/ssl_access.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_access {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_access); destination(d_devo_apache_ssl_access); };
 
# Apache error log
source s_apache_error {
    file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar.www1: $MESSAGE\n"));};
log { source(s_apache_error); destination(d_devo_apache_error); };
 
# Apache SSL error log
source s_apache_ssl_error {
    file("/var/log/apache2/ssl_error.log" follow_freq(1) flags(no-parse));
};
destination d_devo_apache_ssl_error {tcp("DEVO-RELAY" port(PORT)
                            template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar-ssl.www1: $MESSAGE\n"));};
log { source(s_apache_ssl_error); destination(d_devo_apache_ssl_error); };

Ensure the required permissions are enabled

Ensure that both the source file and the directory where it resides can be read by the user running syslog-ng (usually this is syslog). Here's an example of how you might change the owner:

chown :syslog /var/log/apache2 /var/log/apache2/*.log

If the source files are part of a logrotate policy and logrotate create option is being used, ensure that the syslog-ng user will have permissions over the new file. Here we have an excerpt of a logrotate config file for Apache logs. The create option is used to grant permissions to the user root in the group syslog.

Configuration file /etc/logrotate.d/apache2 extract
/var/log/apache2/*.log {
        ...
        create 640 root syslog
        ...

Restart syslog-ng

Once you have edited the syslog-ng.conf file, restart syslog-ng to activate the new configuration:

/etc/init.d/syslog-ng restart

With the configuration activated, syslog-ng should begin to forward log events to your Devo Cloud.

A note about log rotation

Here is an example of truncated logrotate configuration file.

/var/log/file.log
{
        rotate 12
        weekly
        copytruncate
        missingok
        notifempty
        compress
}

This is an example of a logrotate command that applies to several log files.

/var/logs/file.out
/var/logs/file.log
/var/logs/localhost.log
/var/logs/localhost_access_log.txt
{
        rotate 10
        daily
        copytruncate
        missingok
        notifempty
        compress
        lastaction
                service syslog-ng reload
        endscript
}

The last action directive reloads syslog-ng once all of the log files have been rotated.