Monitoring files using syslog-ng
The instructions for setting up connections to a Devo Relay or Cloud involve the editing of the syslog-ng.conf and are centered on the sending of all system log events to a Devo endpoint.
In this article, we explain how to include other log as sources in the syslog-ng.conf file and configure their sending to Devo endpoint destinations. There are a few simple steps:
Edit the syslog-ng.conf file
This configuration file is saved in /etc/syslog-ng/, and consists of a series of source, destination and log object definitions. These are represented in this generic and anonymous sample syslog-ng.conf file.
source s_myfile { file("/path/to/file.log" follow_freq(1) flags(no-parse));}; destination d_devo_myfile {tcp("DEVO-RELAY" port(PORT) template("<$PRI>$DATE $HOST my.devo.tag: $MESSAGE\n"));}; log { source(s_myfile); destination(d_devo_myfile); };
In the source object, you would replace
s_myfile
with a custom identifier and"/path/to/file.log"
with the path including the file name of the file you want syslog-ng to monitor.In the destination object, you would replace
d_devo_myfile
with a custom identifier,DEVO-RELAY
with the relay IP address andPORT
with the relay port. Go to Administration → Relays in Devo to see a list of available relays. In the template definition, replacemy.devo.tag
with the Devo tag to apply to all events coming from the source file. Find the correct tag in the 2021-07-08_07-50-26_List of Devo parsers.
Here's another sample file, this time showing how you can configure the monitoring of multiple log files - in this case the logs for an Apache Server.
# Apache access log source s_apache_access { file("/var/log/apache2/access.log" follow_freq(1) flags(no-parse)); }; destination d_devo_apache_access {tcp("DEVO-RELAY" port(PORT) template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar.www1: $MESSAGE\n"));}; log { source(s_apache_access); destination(d_devo_apache_access); }; # Apache SSL access log source s_apache_ssl_access { file("/var/log/apache2/ssl_access.log" follow_freq(1) flags(no-parse)); }; destination d_devo_apache_ssl_access {tcp("DEVO-RELAY" port(PORT) template("<$PRI>$DATE $HOST web.apache.access-combined.pro.webFoobar-ssl.www1: $MESSAGE\n"));}; log { source(s_apache_ssl_access); destination(d_devo_apache_ssl_access); }; # Apache error log source s_apache_error { file("/var/log/apache2/error.log" follow_freq(1) flags(no-parse)); }; destination d_devo_apache_error {tcp("DEVO-RELAY" port(PORT) template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar.www1: $MESSAGE\n"));}; log { source(s_apache_error); destination(d_devo_apache_error); }; # Apache SSL error log source s_apache_ssl_error { file("/var/log/apache2/ssl_error.log" follow_freq(1) flags(no-parse)); }; destination d_devo_apache_ssl_error {tcp("DEVO-RELAY" port(PORT) template("<$PRI>$DATE $HOST web.apache.error.pro.webFoobar-ssl.www1: $MESSAGE\n"));}; log { source(s_apache_ssl_error); destination(d_devo_apache_ssl_error); };
Ensure the required permissions are enabled
Ensure that both the source file and the directory where it resides can be read by the user running syslog-ng (usually this is syslog). Here's an example of how you might change the owner:
chown :syslog /var/log/apache2 /var/log/apache2/*.log
If the source files are part of a logrotate policy and logrotate create option is being used, ensure that the syslog-ng user will have permissions over the new file. Here we have an excerpt of a logrotate config file for Apache logs. The create option is used to grant permissions to the user root in the group syslog.
/var/log/apache2/*.log { ... create 640 root syslog ...
Restart syslog-ng
Once you have edited the syslog-ng.conf file, restart syslog-ng to activate the new configuration:
/etc/init.d/syslog-ng restart
With the configuration activated, syslog-ng should begin to forward log events to your Devo Cloud.
A note about log rotation
Here is an example of truncated logrotate configuration file.
/var/log/file.log { rotate 12 weekly copytruncate missingok notifempty compress }
This is an example of a logrotate command that applies to several log files.
/var/logs/file.out /var/logs/file.log /var/logs/localhost.log /var/logs/localhost_access_log.txt { rotate 10 daily copytruncate missingok notifempty compress lastaction service syslog-ng reload endscript }
The lastaction directive reloads syslog-ng once all of the log files have been rotated.