proxy.squid

[ 1 Introduction ] [ 2 Valid tags and data tables ] [ 3 Table structure ] [ 4 How is the data sent to Devo? ]

Introduction

The tags beginning with proxy.squid identify log events generated by the Squid caching proxy belonging to Squid.org.

Valid tags and data tables

The full tag must have 4 levels. The first two are fixed as proxy.squid. The third level identifies the type of events sent. The fourth level indicates the event subtype.

These are the valid tags and corresponding data tables that will receive the parsers' data:

Product / Service	Tags	Data tables

Product / Service	Tags	Data tables
Squid caching proxy	`proxy.squid.access-clf.turrican`	`proxy.squid.accessClf`
	`proxy.squid.access-combined.turrican`	`proxy.squid.accessCombined`
	`pproxy.squid.access-lt.turrican`	`proxy.squid.accessLt`
	`proxy.squid.access-squid.turrican`	`proxy.squid.accessSquid`
	`proxy.squid.accessSquidMime`	`proxy.squid.accessSquidMime`
	`proxy.squid.cache.turrican`	`proxy.squid.cache`

For more information, read more about Devo tags.

Access logs

Squid offers several default formats for log events that you can generate (or not) using the logformat and access_log directives. For more information about default formats, see the Squid documentation.

To these formats, Devo proposes a more detailed log format called access-lt that is expressed like this:

Devo Access:

logformat access-lt %{%F:%T%z}tl %>a:%>p %ui %un %<A "%rm %ru HTTP/%rv" "%{Referer}>h" "%{User-Agent}>h" "%{Cookie}>h" %>Hs %>st %tr %<st %mt %Ss:%Sh

Table structure

These are the fields displayed in these tables:

How is the data sent to Devo?

Devo recommends using rsyslog as a means to forward Squid log events to a Devo endpoint. This enables you to correctly tag the events and forward them to either a Devo Relay or directly to the Devo Cloud.

Sending to the Devo endpoint

We suggest you confirm the location of the Squid log files as dictated in the squid.conf file. Then you can use rsyslog to forward the log events to a Devo endpoint, either a Devo relay or directly to the Devo Cloud.

/etc/squid/squid.conf file extract: Below are examples of how the access_log directive is used in this file to indicate the location for each log file.

cache_log /var/log/squid3/cache.log
cache_store_log /var/log/squid3/store.log
 
#Choose one or more formats for the access log
access_log /var/log/squid3/accessSquid.log squid
access_log /var/log/squid3/accessMime.log squidmime
access_log /var/log/squid3/accessClf.log common
access_log /var/log/squid3/accessCombined.log combined
access_log /var/log/squid3/access-lt.log access-lt

/etc/rsyslog.d/45-squid.conf file: Below is an example of how to monitor these log files using a rsyslog configuration file:

$template squid,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
 
# File access
$InputFileName /var/log/squid3/cache.log
$InputFileTag proxy.squid.cache.myProxyName:  
$InputFileStateFile stat-file1-SquidCache
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-lt.log
$InputFileTag proxy.squid.accesslt.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessLt
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-clf.log
$InputFileTag proxy.squid.accessclf.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessClf
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-combined.log
$InputFileTag proxy.squid.accesscombined.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessCombined
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-squid.log
$InputFileTag proxy.squid.accesssquid.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquid
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/accessMime.log
$InputFileTag proxy.squid.access-squid-mime.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquidMime
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# SSL config for sending to Devo Cloud
#$DefaultNetstreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
#$ActionSendStreamDriverMode 1 # require TLS for the connection
#$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer collector
 
if $syslogtag contains 'proxy.squid.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;squid
:syslogtag, contains, "proxy.squid." ~

Sending to a Devo Relay: Replace DEVO-RELAY and PORT with the IP address and the port of your Devo In-house Relay.
Sending to the Devo Cloud: Uncomment the SSL config section and be sure the security certificate files are correctly referenced. Replace DEVO-RELAY and PORT with the hostname of the Devo Cloud and port 514.

Other ways to forward events

A log destination, besides being a file, can also be an external process such as local syslog machine or a ydp/tcp server. To send to an external process, you should include program name and full path. Squid uses a minimal protocol in all the messages sent to the process and controls the logs processing by using the first byte of each event. The external process, like logger, should be capable of sending to a remote syslog server.

Forwarding to an external process

Another mechanism is to send to the machine's local syslog daemon, by specifying the destination as syslog:facility.priority. In this case, you should apply a rsyslog or syslog-ng filter for the given facility.priority to tag all the events coming through it as proxy.squid.access-lt.myProxyName and send them to the Devo endpoint.

Forwarding to local syslog

Although Squid offers a TCP Receiver module for sending log lines to an external server, because the data is sent in text format, not syslog, this is not supported by Devo.