Page Comparison

Introduction

The tags beginning with proxy.squid squid identify log events generated by the Squid caching proxy .

Devo recommends using rsyslog as a means to forward Squid log events to a Devo endpoint. This enables you to correctly tag the events and forward them to either a Devo Relay or directly to the Devo Cloud.

...

belonging to Squid.org.

Valid tags and data tables 

The full tag must have at least four 4 levels. The first two are fixed as asproxy.squid. The  The third level identifies the event's log type and must be one of access-clf, access-combine, access-lt, access-squid, access-squid-mime, or cachetype of events sent. The fourth level of the tag is required and should be the hostname of the Squid proxy server that generated the event. 

...

Technology

...

Brand

...

Log format/type

...

Subtype

...

proxy

...

squid

...

• access-clf

• access-combine

• access-lt

• access-squid

• access-squid-mime

• cache

...

required - Squid server hostname

Therefore, the valid tags include:

...

indicates the event subtype.

These are the valid tags and corresponding data tables that will receive the parsers' data:

...

...

...

...

...

...

...

For more information, read more about Devo tags.

...

logformat access-lt %{%F:%T%z}tl %>a:%>p %ui %un %<A "%rm %ru HTTP/%rv" "%{Referer}>h" "%{User-Agent}>h" "%{Cookie}>h" %>Hs %>st %tr %<st %mt %Ss:%Sh

Table structure

These are the fields displayed in these tables:

How is the data sent to Devo?

Devo recommends using rsyslog as a means to forward Squid log events to a Devo endpoint. This enables you to correctly tag the events and forward them to either a Devo Relay or directly to the Devo Cloud.

Sending to the Devo endpoint

We suggest you confirm the location of the Squid log files as dictated in the squid.conf file. Then you can use rsyslog to forward the log events to a Devo endpoint, either a Devo relay or directly to the Devo Cloud.

/etc/squid/squid.conf file extract: Below are examples of how the access_log directive is used in this file to indicate the location for each log file.

/etc/squid/squid.conf file extract

cache_log /var/log/squid3/cache.log
cache_store_log /var/log/squid3/store.log
 
#Choose one or more formats for the access log
access_log /var/log/squid3/accessSquid.log squid
access_log /var/log/squid3/accessMime.log squidmime
access_log /var/log/squid3/accessClf.log common
access_log /var/log/squid3/accessCombined.log combined
access_log /var/log/squid3/access-lt.log access-lt

/etc/rsyslog.d/45-squid.conf file: Below is an example of how to monitor these log files using a rsyslog configuration file:

...

$template squid,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
 
# File access
$InputFileName /var/log/squid3/cache.log
$InputFileTag proxy.squid.cache.myProxyName:  
$InputFileStateFile stat-file1-SquidCache
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-lt.log
$InputFileTag proxy.squid.accesslt.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessLt
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-clf.log
$InputFileTag proxy.squid.accessclf.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessClf
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-combined.log
$InputFileTag proxy.squid.accesscombined.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessCombined
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/access-squid.log
$InputFileTag proxy.squid.accesssquid.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquid
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# File access
$InputFileName /var/log/squid3/accessMime.log
$InputFileTag proxy.squid.access-squid-mime.myProxyName:  
$InputFileStateFile stat-file1-SquidAccessSquidMime
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# SSL config for sending to Devo Cloud
#$DefaultNetstreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
#$ActionSendStreamDriverMode 1 # require TLS for the connection
#$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer collector
 
if $syslogtag contains 'proxy.squid.' and $syslogfacility-text == 'local7' then @@DEVO-RELAY:PORT;squid
:syslogtag, contains, "proxy.squid." ~

• Sending to a Devo Relay: Replace DEVO-RELAY and PORT with the IP address and the port of your Devo In-house Relay. 

• Sending to the Devo Cloud: Uncomment the SSL config section and be sure the security certificate files are correctly referenced.  Replace DEVO-RELAY and PORT with the hostname of the Devo Cloud and port 514. 

Other ways to forward events

A log destination, besides being a file, can also be an external process such as local syslog machine or a ydp/tcp server. To send to an external process, you should include program name and full path. Squid uses a minimal protocol in all the messages sent to the process and controls the logs processing by using the first byte of each event. The external process, like logger, should be capable of sending to a remote syslog server.

Forwarding to an external process

logfile_daemon /path/to/helper_process
access_log daemon:/var/log/squid/access.log access-lt

Another mechanism is to send to the machine's local syslog daemon, by specifying the destination as syslog:facility.priority. In this case, you should apply a rsyslog or syslog-ng filter for the given facility.priority to tag all the events coming through it as proxy.squid.access-lt.myProxyName and send them to the Devo endpoint.

Forwarding to local syslog

access_log syslog:local4.info access-lt

Although Squid offers a TCP Receiver module for sending log lines to an external server, because the data is sent in text format, not syslog, this is not supported by Devo. 

Related articles

...

Squid access_log directive

...

SquidLogs

...

Squid Log Modules

...