Versions Compared

Old Version 2

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Borja Moro Moreno

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The tags beginning with web.tomcat identify log events generated by the Apache Tomcat web application server and the access logs of the applications the server hosts. 

This article describes the tag naming structure, a little about the main types of Tomcat server logs, and how to send log events from the Tomcat server to Devo.

Table of Contents
maxLevel2

Tag structure

The full tag must have at least six levels. The first two are fixed as web.tomcat. The third level identifies the log type/format and currently must be one of access-clf, access-combined, access-lt, app, app-lt, catalina, catalina-lt, gc or out.

The fourth, fifth and sixth levels are required and should identify the environment type, web application, and instance respectively. 

  • environment - Describes the environment in when the event occurred. For example, development, testing, or production.
  • web application - The name of the web application.
  • clon - This is the instance that generated the event. Depending on your network, this can be a machine name, or the virtual name of a Tomcat process.

The values of these levels should be guided by the structure we propose because they will be saved in the events when saved in Devo. When you open the resulting data table, these will appear in the environment, site and clon columns.

...

Technology

...

Brand

...

Log type/format

...

Environment

...

Web application

...

Clon

...

  • access-clf
  • access-combined
  • access-lt
  • app
  • app-lt
  • catalina
  • catalina-lt
  • gc
  • out

free but required

...

Therefore, the valid tags include:

  • web.tomcat.access-clf.env.app.clon
  • web.tomcat.access-combined.env.app.clon
  • web.tomcat.access-lt.env.app.clon
  • web.tomcat.app.env.app.clon
  • web.tomcat.app-lt.env.app.clon
  • web.tomcat.catalina.env.app.clon
  • web.tomcat.catalina-lt.env.app.clon
  • web.tomcat.gc.env.app.clon
  • web.tomcat.out.env.app.clon

For more information, read more about Devo tags.

Logging in Tomcat

Tomcat uses Apache Common Logging to generate the log events. The log messages that arrive through ServletContex log methods are also routed to this log library and associated to the following category:

org.apache.catalina.core.ContainerBase.[engine].[host].[context]

  • Usually, the engine is Catalina and the host is localhost.
  • The context name it is used as context, but there are some exceptions. For example, an application whose context path is /cava would have the category org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/cava] associated.

In Tomcat default distribution, Apache Common Logging is associated to the JUL framework (java.util.logging). Hence, all events, including the ones generated by ServletContext are managed by JUL.

Tomcat configures JUL with the file specified in the LOGGING_CONFIG environment variable. In case it is missing, the installation file CATALINA_BASE/conf/logging.properties is used.

catalina.yyyy-MM-dd.log

This log contains the messages generated by Tomcat, usually associated with the server and applications life cycle. Tags starting with web.tomcat.catalina should be associated with events from this log file.

The definition is found in CATALINA_BASE/ conf/logging.properties. The logs coming from the container are configured to be stored in this file. This is done with the help of a handler built to write in catalina.yyy-MM-dd.log and associated with the root logger.

...

Table of Contents
minLevel1
maxLevel2
outlinefalse
typeflat
printablefalse

Introduction

The tags beginning with web.tomcat identify events generated by the Apache Tomcat web application server belonging to The Apache Software Foundation. 

Valid tags and data tables

The full tag must have at least 3 levels. The first two are fixed as web.nginx. The third level identifies the type of events sent and the rest of them indicate the event subtypes.

Product/Service

Tag

Data table

Apache Tomcat web application server

web.tomcat.accessClf

web.tomcat.accessClf

web.tomcat.accessCombined

web.tomcat.accessCombined

web.tomcat.access-lt.des.lucca.zkoxgiaud155m0j

web.tomcat.accessLt

web.tomcat.app

web.tomcat.app

web.tomcat.app-lt.des.lucca.zkoxgiaud155m0j

web.tomcat.appLt

web.tomcat.catalina

web.tomcat.catalina

web.tomcat.catalina-lt.des.lucca.zkoxgiaud155m0j

web.tomcat.catalinaLt

web.tomcat.tomcat_gc

web.tomcat.tomcat_gc

web.tomcat.out.pro.app-logtrust.aws1

web.tomcat.out

For more information, read more about Devo tags.

Expand
titleEvent formats

Tomcat uses Apache Common Logging to generate the log events. The log messages that arrive through ServletContex log methods are also routed to this log library and associated to the following category:

org.apache.catalina.core.ContainerBase.[engine].[host].[context]

Usually, the engine is Catalina and the host is localhost. The context name it is used as [context], but there are some exceptions. For example, an application whose context path is /cava would have the following category associated org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/cava].

In Tomcat default distribution, Apache Common Logging is associated to the JUL framework (java.util.logging). Hence, all events, including the ones generated by ServletContext are managed by JUL.

Tomcat configures JUL with the file specified in the LOGGING_CONFIG environment variable. In case it is missing, the installation file CATALINA_BASE/conf/logging.properties is used.

  • Access log (localhost_access_log.yyyy-MM-dd.log):
    This file contains an access log similar to web server logs. In the log, there is an event for each petition processed by the server. You can control the event content (format and fields) in detail.
    The access log is configured as a Tomcat Valve. The definition is found in CATALINA_BASE/conf/server.xml. 

    Code Block
    <Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs"
    prefix="localhost_access_log." suffix=".txt"
    pattern="%h %l %u %t &quot;%r&quot; %s %b" />

    Devo supports three formats for access logs; two of them are based upon standard formats and the other is defined by Devo to offer a format that contains more details. Each of these formats corresponds with the third level of an access log tag.

    • The log format that corresponds to the web.tomcat.access-clf tag is based on the Common Log Format (CLF). The specification of this format is:

      Code Block
      pattern="%h %l %u %t &quot;%r&quot; %s %b"
pattern="common"

    • The log format that corresponds to the web.tomcat.access-combined tag is based on the NCSA Combined log format. The specification of this format is:

      Code Block
      pattern="%h %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
pattern="combined"

    • The log format that corresponds to the web.tomcat.access-lt tag is a custom format defined by Devo to offer more detailed logging. The specification of this format that includes cookie names is:

      Code Block
      pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{cookieName1}c:%{cookieName2}c&quot; %s %D %B %I"

      However, if you prefer not to include cookie names in your events, use this specification instead. 

      Code Block
      pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;&quot; %s %D %B %I"

  • localhost.yyyy-MM-dd.log:
    This log contains the messages written by the applications with JUL (java.util.logging). Tags starting with web.tomcat.app should be associated with events from this log file.
    The definition is found in CATALINA_BASE/conf/logging.properties. The logs coming from the applications are configured to be stored in this file. This is done with the help of a handler built to write in localhost.yyy-MM-dd.log and associated with ServletContext log.

    Code Block
    2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina
  • 2localhost.org.apache.juli.FileHandler.prefix = localhost.
org.apache.catalina
. 
  • .core.ContainerBase.[Catalina].[localhost].handlers =
1catalina
  • 2localhost.org.apache.juli.FileHandler

...

  • catalina.yyyy-MM-dd.log:
    This log contains the messages

...

  • generated by Tomcat, usually associated with the

...

  • server and applications life cycle. Tags starting with web.tomcat.

...

  • catalina should be associated with events from this log file.
    The definition is found

...

  • in CATALINA_BASE/conf/logging.properties. The logs coming from the

...

  • container are configured to be stored in this file. This is done with the help of a handler built to write in 

...

  • catalina.yyy-MM-dd.

...

  • log and associated with the root logger.

    Code Block
2localhost
  • 1catalina.org.apache.juli.FileHandler.level = FINE
2localhost
  • 1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost
  • 1catalina.org.apache.juli.FileHandler.prefix =
localhost
  • catalina.  
.handlers = 1catalina.org.apache.
catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler

catalina.out

  • juli.FileHandler

  • catalina.out:
    This log contains the information sent by the applications to stdout and stderr. It is generated by redirecting to this files standard and error output at operating system level. Since Tomcat is not directly responsible of its creation, it doesn't rotate. Tags starting with web.tomcat.out should be associated with events from this log file.

...

Access log (localhost_access_log.yyyy-MM-dd.log)

This file contains an access log similar to web server logs. In the log, there is an event for each petition processed by the server. You can control the event content (format and fields) in detail.

The access log is configured as a Tomcat Valve. The definition is found in CATALINA_BASE/conf/server.xml. 

Code Block
<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs"
    prefix="localhost_access_log." suffix=".txt"
    pattern="%h %l %u %t &quot;%r&quot; %s %b" />

Devo supports three formats for access logs; two of them are based upon standard formats and the other is defined by Devo to offer a format that contains more details. Each of these formats corresponds with the third level of an access log tag.

The log format that corresponds to the web.tomcat.access-clf tag is based on the Common Log Format (CLF). The specification of this format is:

Code Block
pattern="%h %l %u %t &quot;%r&quot; %s %b"
pattern="common"

The log format that corresponds to the web.tomcat.access-combined tag is based on the NCSA Combined log format. The specification of this format is:

Code Block
pattern="%h %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
pattern="combined"

The log format that corresponds to the web.tomcat.access-lt tag is a custom format defined by Devo to offer more detailed logging. The specification of this format that includes cookie names is:

Code Block
pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;%{cookieName1}c:%{cookieName2}c&quot; %s %D %B %I"

 However, if you prefer not to include cookie names in your events, use this specification instead. 

Code Block
pattern="%t %a %l %u %S %v:%p %m &quot;%U%q&quot; %H &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot; &quot;&quot; %s %D %B %I"

Sending to Devo using file monitoring

Table structure

These are the fields displayed in these tables:

Rw ui tabs macro
Rw tab
title1-3

web.tomcat.accessClf

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

serverdate

timestamp

 

srcIp

ip4

 

user

str

 

method

str

 

url

str

 

protocol

str

 

statusCode

int4

 

responseLength

int4

 

srcIdentd

str

 

userAgent

str

 

hostchain

str

 

✓

tag

str

 

✓

rawMessage

str

 

✓

web.tomcat.accessCombined

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

serverdate

timestamp

 

srcIp

ip4

 

user

str

 

method

str

 

url

str

 

protocol

str

 

statusCode

int4

 

responseLength

int4

 

referer

str

 

userAgent

str

 

srcIdentd

str

 

hostchain

str

 

✓

tag

str

 

✓

rawMessage

str

 

✓

web.tomcat.accessLt

Field

Type

Source field name

Extra Label

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

serverdate

timestamp

 

srcIp

ip4

 

serverName

str

 

serverPort

int4

 

user

str

 

sessionId

str

 

method

str

 

url

str

 

protocol

str

 

statusCode

int4

 

referer

str

 

userAgent

str

 

cookies

str

 

responseTime

int4

 

requestLength

int4

 

threadName

str

 

srcIdentd

str

 

hostchain

str

 

✓

tag

str

 

✓

rawMessage

str

 

✓

Rw tab
title4-6

web.tomcat.app

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

message

str

 

rawMessage

str

message

✓

hostchain

str

 

✓

tag

str

 

✓

web.tomcat.appLt

Field

Type

Source field name

Extra Label

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

serverdate

timestamp

 

sequence

int4

 

level

str

 

thread

int4

 

class

str

 

method

str

 

message

str

 

exceptionClass

str

 

exceptionMessage

str

 

stackTtrace

str

 

hostchain

str

 

✓

tag

str

 

✓

rawMessage

str

 

✓

web.tomcat.catalina

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

message

str

 

rawMessage

str

message

✓

hostchain

str

 

✓

tag

str

 

✓

Rw tab
title7-9

web.tomcat.catalinaLt

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

serverdate

timestamp

 

logger

str

 

sequence

int4

 

level

str

 

thread

int4

 

class

str

 

method

str

 

message

str

 

exceptionClass

str

 

exceptionMessage

str

 

stackTtrace

str

 

hostchain

str

 

✓

tag

str

 

✓

web.tomcat.tomcat_gc

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

rawMessage

str

rawSource

✓

message

str

 

hostchain

str

 

✓

tag

str

 

✓

web.tomcat.out

Field

Type

Source field name

Extra fields

eventdate

timestamp

 

environment

str

venv

application

str

vapp

clon

str

vclon

message

str

 

hostchain

str

 

✓

tag

str

 

✓

How is the data sent to Devo?

To send data to Devo, you con use file monitor as shown below.

File monitor in UNIX environments

The standard method is to configure Tomcat to write the logs to file and rely on another tool (like rsyslog or syslog-ng) to send the events. To know more about this topic, visit Monitoring files using rsyslog and Monitoring files using rsyslog.

Rw ui steps macro
Rw step

Since rsyslog cannot read dynamic files and the Tomcat log files are rotated daily, you need to turn log rotation off. This means the log files will have static names that rsyslog can reference. To disable rotation, set the rotatable attribute to false in the files where these logs are configured.

Disabling log rotation for catalina and localhost logs: For localhost-yyyy-MM-dd.log and catalina-yyyy-MM-dd.log you have to modify CATALINA_BASE/conf/logging.properties:

...

Disabling log rotation for catalina and localhost logsDisabling log rotation for access logicon
Code Block
title

Code Block
1catalina.org.apache.juli.FileHandler.level = FINE 
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs 
1catalina.org.apache.juli.FileHandler.prefix = catalina 
1catalina.org.apache.juli.FileHandler.rotatable = false 


2localhost.org.apache.juli.FileHandler.level = FINE 
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs 
2localhost.org.apache.juli.FileHandler.prefix = localhost 
2localhost.org.apache.juli.FileHandler.rotatable = false
Disabling log rotation for access log: For the access log you have to modify CATALINA_BASE/conf/server.xml:
 Code Block
title
<Valve className="org.apache.catalina.valves.AccessLogValve"
    directory="logs"
    prefix="localhost_access_log" suffix=".txt"
    pattern="%h %l %u %t &quot;%r&quot; %s %b"
    rotatable="false" />


 Note
false
Note that the prefixes do not end with a dot. Since the date will no longer be appended to the file name daily, it's not necessary to include a separator after the final character of the prefix.
 Rw step
/etc/rsyslog.d/45-tomcat.conf (sample): Here we offer a sample rsyslog configuration file that is set up to monitor the catalina.log, catalina.out, localhost.log, and localhost_access_log.txt files, and forward them to a 
...
/etc/rsyslog.d/45-tomcat.conf (sample)/etc/logrotate.d/tomcat
 Code Block
title
Devo Relay.
 Code Block
$template tomcat,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag% %msg%"
  
# File catalina.log
$InputFileName CATALINA_BASE/logs/catalina.log
$InputFileTag web.tomcat.catalina.ENV.APP.CLON:
$InputFileStateFile stat-Tomcat-catalina
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
  
# File localhost.log
$InputFileName CATALINA_BASE/logs/localhost.log
$InputFileTag web.tomcat.app.ENV.APP.CLON:
$InputFileStateFile stat-Tomcat-localhost
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
  
# File catalina.out
$InputFileName CATALINA_BASE/logs/catalina.out
$InputFileTag web.tomcat.out.ENV.APP.CLON:
$InputFileStateFile stat-Tomcat-out
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
  
# File access log
$InputFileName CATALINA_BASE/logs/localhost_access_log.txt
$InputFileTag web.tomcat.access-clf.ENV.APP.CLON:
$InputFileStateFile stat-Tomcat-accessClf
$InputFileSeverity info
$InputFileFacility local7
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
  
# SSL config for Devo secure relay
#$DefaultNetstreamDriver gtls # use gtls netstream driver
#$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.crt
#$DefaultNetstreamDriverCertFile /etc/rsyslog.d/user.crt
#$DefaultNetstreamDriverKeyFile /etc/rsyslog.d/user.key
#$ActionSendStreamDriverMode 1 # require TLS for the connection
#$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer collector
  
if $syslogtag contains 'web.tomcat.' and $syslogfacility-text == 'local7' then @@LOGTRUST-RELAY:PORT;tomcat
:syslogtag, contains, "web.tomcat." ~
You should personalize the rsyslog configuration file as follows:
  • Replace CATALINA_BASE by the absolute path where Tomcat logs reside.
  • Replace ENV.APP.CLON with the values that represent the source environment, application, and instance.
  • In this example, the access log uses the access-clf tag. You should specify the tag that corresponds to the format you have elected to use.
  • Replace DEVO-RELAY and PORT with the IP address and port of your Devo Relay. 
  • You can uncomment the SSL section of the file to send the events directly to the Devo Cloud. In this case, you should replace DEVO-RELAY and PORT with the hostname of your Devo domain and port 443. 
  • Make sure that the file you are sending and the directory where resides can be read by the user running rsyslog. If not, you should assign the necessary permissions by changing the file group to syslog.
 Rw step
/etc/logrotate.d/tomcat: To configure file rotation, you can use the logrotate utility. The following in an example of a tomcat.conf might be set up.
 Code Block
title
CATALINA_BASE/logs/catalina.out
CATALINA_BASE/logs/catalina.log
CATALINA_BASE/logs/localhost.log
CATALINA_BASE/logs/localhost_access_log.txt
{
    rotate 14
    daily
    copytruncate
    missingok

    notifempty
    delaycompress
    compress
    sharedscripts
 

  
lastaction
  notifempty
    delaycompress
 
service
 
rsyslog
 
stop
 compress
    sharedscripts
  
rm
 
-f /var/spool/rsyslog/stat-Tomcat-{catalina,localhost,out,accessClf}
 lastaction
        service rsyslog 
start
stop
    
endscript
 
}

When you have made all the configuration changes necessary, don't forget to restart the rsyslog process:
 Code Block
/etc/init.d/rsyslog restart
Windows environments
When running Tomcat on Windows, Devo recommends using the MagicLog component of the 2021-06-15_09-43-16_Devo Agent for Windows to forward the log events to Devo. You can also use the third-party tool, Snare Epilog. 
In both cases:
  • Make sure the logs are written in text files.
  • Have the complete paths to the log files on hand when setting up the sending.
...
   rm -f /var/spool/rsyslog/stat-Tomcat-{catalina,localhost,out,accessClf}
        service rsyslog start
    endscript
}
 Rw step
When you have made all the configuration changes necessary, don't forget to restart the rsyslog process:
 Code Block
/etc/init.d/rsyslog restart