Document toolboxDocument toolbox

Monitoring files using rsyslog (Obsolete legacy format)

Owned by Juan Tomás Alonso Nieto (Deactivated)
Jul 06, 2022

The instructions for setting up connections to a Devo Relay or Cloud require the creation of the 00-devo.conf and 49-devo.conf files and cover the sending of all system log events to the Devo endpoint.

For an example of the 00-devo.conf and 49-devo.conf files for sending events to a Devo relay, see Simple sending using rsyslog (Obsolete legacy format) article. If you are planning to send the monitored files directly to the Devo Cloud, check out the sample 49-devo.conf file in Secure sending using rsyslog (Obsolete legacy format).

This article explains how to use an additional /etc/rsyslog.d/*.conf file to send events from other logs on the host to the same Devo endpoint. You may create multiple .conf files depending on the variety of log files you need to send. Each file can include up to 100 different input files to monitor.

Create the configuration file

For each log file that you want to monitor on the host, you will include a dedicated set of input parameters in the configuration file. These parameters are part of the imfile module loaded in 00-devo.conf.  The sample below contains placeholder values:  

/etc/rsyslog.d/45-filemonitor.conf file monitoring example
$template myFileMonitorTemplate,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"

# File access
$InputFileName /path/to/file.log      # the file being monitored
$InputFileTag my.devo.tag:        # tag to assign to messages from the InputFileName
$InputFileStateFile stat-file1-myFileMonitor   # defines name of state files
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

if $syslogtag contains 'my.devo.tag' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;myFileMonitorTemplate
:syslogtag, contains, "my.devo.tag" ~

These are the parameters we recommend using to configure each input file.

1$InputFileNameThe path and filename of the file to monitor.
2$InputFileTagThe tag to assign to messages that come from the monitored file. 
3$InputFileStateFileThe name of the state file that will be saved in the $WorkingDirectory (specified in 00-devo.conf). The state file keeps a history of the messages that were more recently processed. In the case of a rsyslog crash or reboot, this file will be checked in order to know where to resume message processing.
4$InputFileSeverity / $InputFileFacilityThe syslog severity and facility to assign to messages from the input file.
5$InputFilePollIntervalThe interval in seconds between checking the file for new messages.
6$InputFilePersistStateIntervalThe number of events after which the processing status should be updated. This should be set to 1.
7If $syslogtag contains 'my.devo.tag' …This line instructs rsyslog to send all incoming events with the ‘my.devo.tag’ tag via TCP to ‘DEVO-RELAY:PORT’ using the log format defined in ‘myFileMonitorTemplate’ in the configuration file header.  
8:syslogtag, contains, 'my.devo.tag.' ~This line removes messages with the my.devo.tag from the rsyslog processing chain so that they will not be re-processed by any other configuration files.

The following configuration file shows how to monitor multiple Apache server log files and send them to a Devo Relay. Note that this file is not set up with SSL/TLS security.

File /etc/rsyslog.d/45-apache.conf monitoring example
$template apache,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"

# Define the input of access.log
$InputFileName /var/log/apache2/access.log
$InputFileTag web.apache.access-combined.pro.webFoobar.www1:
$InputFileStateFile stat-file1-ApacheAccess
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# Define the input of ssl_access.log
$InputFileName /var/log/apache2/ssl_access.log
$InputFileTag web.apache.access-combined.pro.webFoobar-ssl.www1:
$InputFileStateFile stat-file2-ApacheAccess
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# Define the input of error.log
$InputFileName /var/log/apache2/error.log
$InputFileTag web.apache.error.pro.webFoobar.www1:  
$InputFileStateFile stat-file1-ApacheError
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

# Define the input of ssl_error.log
$InputFileName /var/log/apache2/ssl_error.log
$InputFileTag web.apache.error.pro.webFoobar-ssl.www1:  
$InputFileStateFile stat-file2-ApacheError
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor

if $syslogtag contains 'web.apache.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;apache
:syslogtag, contains, "web.apache." ~

Ensure the required permissions are enabled

The user running rsyslog, usually syslog, must have permissions over all directories and files that you want to monitor. Here's an example of how you might change the owner:

chown :syslog /var/log/apache2 /var/log/apache2/*.log

If the input files are part of a logrotate policy and logrotate create mode is being used, ensure that syslog will still have permissions over the new file. For example, in Apache: 

/etc/logrotate.d/apache2 configuration file extract
/var/log/apache2/*.log {
        ...
        create 640 root syslog
        ...

Restart rsyslog

Close the configuration file, then restart rsyslog.

/etc/init.d/rsyslog restart

Once restarted, go to Data Search in the Devo web application and look for the table that contains the events with the tag you just set up.