proxy.squid

[ 1 Tag structure ] [ 2 Squid access.log configuration ] [ 3 Sending to the Devo endpoint ] [ 4 Other ways to forward events ]

The tags beginning with proxy.squid identify log events generated by the Squid caching proxy.

Devo recommends using rsyslog as a means to forward Squid log events to a Devo endpoint. This enables you to correctly tag the events and forward them to either a Devo Relay or directly to the Devo Cloud.

Tag structure

The full tag must have at least four levels. The first two are fixed as proxy.squid. The third level identifies the event's log type and must be one of access-clf, access-combine, access-lt, access-squid, access-squid-mime, or cache. The fourth level of the tag is required and should be the hostname of the Squid proxy server that generated the event.

Technology	Brand	Log format/type	Subtype

Technology	Brand	Log format/type	Subtype
proxy	squid	access-clf access-combine access-lt access-squid access-squid-mime cache	required - Squid server hostname

Therefore, the valid tags include:

proxy.squid.access-clf.serverHostname
proxy.squid.access-combined.serverHostname
proxy.squid.access-lt.serverHostname
proxy.squid.access-squid.serverHostname
proxy.squid.access-squid-mime.serverHostname
proxy.squid.cache.serverHostname

For more information, read more about Devo tags.

Squid access.log configuration

Squid offers several default formats for log events that you can generate (or not) using the logformat and access_log directives. For more information about default formats, see the Squid documentation.

To these formats, Devo proposes a more detailed log format called access-lt that is expressed like this:

Devo Access

logformat access-lt %{%F:%T%z}tl %>a:%>p %ui %un %<A "%rm %ru HTTP/%rv" "%{Referer}>h" "%{User-Agent}>h" "%{Cookie}>h" %>Hs %>st %tr %<st %mt %Ss:%Sh

Sending to the Devo endpoint

We suggest you confirm the location of the Squid log files as dictated in the squid.conf file. Then you can use rsyslog to forward the log events to a Devo endpoint, either a Devo relay or directly to the Devo Cloud.

Below are examples of how the access_log directive is used in this file to indicate the location for each log file.

/etc/squid/squid.conf file extract

cache_log /var/log/squid3/cache.log
cache_store_log /var/log/squid3/store.log
 
#Choose one or more formats for the access log
access_log /var/log/squid3/accessSquid.log squid
access_log /var/log/squid3/accessMime.log squidmime
access_log /var/log/squid3/accessClf.log common
access_log /var/log/squid3/accessCombined.log combined
access_log /var/log/squid3/access-lt.log access-lt

Below is an example of how to monitor these log files using a rsyslog configuration file:

/etc/rsyslog.d/45-squid.conf file

$template squid,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
 
# File access
$InputFileName /var/log/squid3/cache.log
$InputFileTag proxy.squid.cache.myProxyName:  
$InputFileStateFile stat-file1-SquidCache
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-lt.log
$InputFileTag proxy.squid.accesslt.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessLt
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-clf.log
$InputFileTag proxy.squid.accessclf.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessClf
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-combined.log
$InputFileTag proxy.squid.accesscombined.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessCombined
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-squid.log
$InputFileTag proxy.squid.accesssquid.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquid
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/accessMime.log
$InputFileTag proxy.squid.access-squid-mime.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquidMime
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# SSL config for sending to Devo Cloud
#$DefaultNetstreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
#$ActionSendStreamDriverMode 1 # require TLS for the connection
#$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer collector
 
if $syslogtag contains 'proxy.squid.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;squid
:syslogtag, contains, "proxy.squid." ~

Sending to a Devo Relay: Replace DEVO-RELAY and PORT with the IP address and the port of your Devo In-house Relay.
Sending to the Devo Cloud: Uncomment the SSL config section and be sure the security certificate files are correctly referenced. Replace DEVO-RELAY and PORT with the hostname of the Devo Cloud and port 514.

Other ways to forward events

A log destination, besides being a file, can also be an external process such as local syslog machine or a ydp/tcp server. To send to an external process, you should include program name and full path. Squid uses a minimal protocol in all the messages sent to the process and controls the logs processing by using the first byte of each event. The external process, like logger, should be capable of sending to a remote syslog server.

Forwarding to an external process

Another mechanism is to send to the machine's local syslog daemon, by specifying the destination as syslog:facility.priority. In this case, you should apply a rsyslog or syslog-ng filter for the given facility.priority to tag all the events coming through it as proxy.squid.access-lt.myProxyName and send them to the Devo endpoint.

Forwarding to local syslog

Although Squid offers a TCP Receiver module for sending log lines to an external server, because the data is sent in text format, not syslog, this is not supported by Devo.