Relay troubleshooting tips (v2.X.X)
Introduction
Devo Relay 2.X.X is based on a Java process that runs on a Linux machine. The Java process internally uses a syslog engine called Scoja.
The relay Java process runs as a service in the Linux OS and it is managed by the system manager systemd
. Once running, the relay listens for TCP/UDP connections on ports and, depending on the port, modifies the syslog events and forwards them to the Devo central collector / ELB. The main role of the relay is to tag events based on a defined rule, then forward the tagged events to Devo.
The relay comes with a few ports that are preconfigured with rules for managing specific types of events. For example, port 13000 is preconfigustartured to forward the events received directly to Devo without any other processing. This port should be used for events that are properly tagged in the data source. You can also create custom rules to tag and/or filter the events sent to the associated port on the relay. Each rule is stored in one of these two folders:
/opt/devo/ng-relay/conf/relay/run/rules/
→ Rules managed from the Devo web application./opt/devo/ng-relay/conf/relay/run/unrules/
→ Rules not managed from the Devo web application. These rules are directly managed on the relay machine itself.
About the relay logs files
Main process logs
As the relay runs as a systemd
service, the logs can be consulted using the following command:
sudo journalctl -u devo-ng-relay [-f]
These log files are also sent to Devo by default and can be consulted in the table box.unix
.
Scoja logs
In addition, several Scoja log files are generated and stored in:
/var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/
main.log
→ main process log filesource.log
→ logs related to the sources configuredthread.log
→ threads opened for each sourcestats.log
→ statistics about received and sent eventstarget.log
→ logs related to the sending process
These logs are also sent to Devo and can be consulted in the tables syslog.scoja.*
. However, we recommend keeping them in the local machine for a few days.
The retention configuration can be set by updating the following file:
/opt/devo/ng-relay/conf/relay/maduro.properties
Troubleshooting checklist
Make sure the relay process (Java) is running
ps aux | grep java | grep -v grep
Response
devo 4001 0.3 56.8 4746500 573532 ? Ssl 05:31 0:31 java -Dfile.encoding=UTF8 -DMADURO_CONF=/opt/devo/ng-relay/conf/ @/opt/devo/ng-relay/conf/relay/devo-ng-relay.jvmoptions -classpath /opt/devo/ng-relay/lib/maduro-server-2.1.0.jar:/opt/devo/ng-relay/lib/maduro-api-client-1.4.0.jar:/opt/devo/ng-relay/lib/maduro-common-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-server-1.5.2.jar:/opt/devo/ng-relay/lib/api-clients-serrea-0.2.1.jar:/opt/devo/ng-relay/lib/jenga-echo-1.4.5.jar:/opt/devo/ng-relay/lib/jenga-monitoring-1.4.5.jar:/opt/devo/ng-relay/lib/httpclient-4.5.13.jar:/opt/devo/ng-relay/lib/crypass-1.0.2.jar:/opt/devo/ng-relay/lib/logback-syslog4j-1.0.0.jar:/opt/devo/ng-relay/lib/logback-classic-1.2.10.jar:/opt/devo/ng-relay/lib/logback-core-1.2.10.jar:/opt/devo/ng-relay/lib/janino-3.1.2.jar:/opt/devo/ng-relay/lib/commons-compress-1.21.jar:/opt/devo/ng-relay/lib/commons-configuration-1.10.jar:/opt/devo/ng-relay/lib/spring-boot-autoconfigure-2.6.3.jar:/opt/devo/ng-relay/lib/spring-boot-2.6.3.jar:/opt/devo/ng-relay/lib/spring-context-5.3.15.jar:/opt/devo/ng-relay/lib/spring-tx-5.2.19.RELEASE.jar:/opt/devo/ng-relay/lib/jinjava-2.6.0.jar:/opt/devo/ng-relay/lib/feign-jackson-10.2.3.jar:/opt/devo/ng-relay/lib/jackson-dataformat-yaml-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-databind-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-annotations-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-core-2.13.1.jar:/opt/devo/ng-relay/lib/feign-slf4j-10.2.3.jar:/opt/devo/ng-relay/lib/slf4j-api-1.7.32.jar:/opt/devo/ng-relay/lib/mysql-connector-java-8.0.28.jar:/opt/devo/ng-relay/lib/javax.annotation-api-1.3.2.jar:/opt/devo/ng-relay/lib/lutier-1.18.0.jar:/opt/devo/ng-relay/lib/scoja-beep-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-compression-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-rpc-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-client-1.5.2.jar:/opt/devo/ng-relay/lib/scoja-rpc-xc-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-cc-1.4.0.jar:/opt/devo/ng-relay/lib/jython-standalone-2.7.2b3.jar:/opt/devo/ng-relay/lib/commons-codec-1.13.jar:/opt/devo/ng-relay/lib/feign-okhttp-10.2.3.jar:/opt/devo/ng-relay/lib/feign-core-10.2.3.jar:/opt/devo/ng-relay/lib/groovy-ant-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-astbuilder-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-cli-picocli-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-groovysh-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-console-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-datetime-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-groovydoc-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-docgenerator-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-jmx-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-json-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-jsr223-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-macro-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-nio-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-servlet-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-sql-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-swing-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-templates-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-test-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-test-junit5-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-testng-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-xml-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-3.0.8.jar:/opt/devo/ng-relay/lib/guava-30.0-jre.jar:/opt/devo/ng-relay/lib/javassist-3.26.0-GA.jar:/opt/devo/ng-relay/lib/jsoup-1.14.2.jar:/opt/devo/ng-relay/lib/re2j-1.2.jar:/opt/devo/ng-relay/lib/commons-lang3-3.10.jar:/opt/devo/ng-relay/lib/commons-net-3.3.jar:/opt/devo/ng-relay/lib/java-ipv6-0.17.jar:/opt/devo/ng-relay/lib/annotations-3.0.1.jar:/opt/devo/ng-relay/lib/big-math-2.0.0.jar:/opt/devo/ng-relay/lib/spring-aop-5.3.15.jar:/opt/devo/ng-relay/lib/spring-beans-5.3.15.jar:/opt/devo/ng-relay/lib/spring-expression-5.3.15.jar:/opt/devo/ng-relay/lib/spring-core-5.3.15.jar:/opt/devo/ng-relay/lib/snakeyaml-1.30.jar:/opt/devo/ng-relay/lib/junit-4.13.2.jar:/opt/devo/ng-relay/lib/junit-platform-launcher-1.7.0.jar:/opt/devo/ng-relay/lib/junit-platform-engine-1.7.0.jar:/opt/devo/ng-relay/lib/junit-platform-commons-1.7.0.jar:/opt/devo/ng-relay/lib/junit-jupiter-engine-5.7.0.jar:/opt/devo/ng-relay/lib/junit-jupiter-api-5.7.0.jar:/opt/devo/ng-relay/lib/hamcrest-core-1.3.jar:/opt/devo/ng-relay/lib/micrometer-registry-prometheus-1.6.1.jar:/opt/devo/ng-relay/lib/commons-logging-1.2.jar:/opt/devo/ng-relay/lib/commons-lang-2.6.jar:/opt/devo/ng-relay/lib/okhttp-4.9.1.jar:/opt/devo/ng-relay/lib/protobuf-java-3.19.4.jar:/opt/devo/ng-relay/lib/jsr305-3.0.2.jar:/opt/devo/ng-relay/lib/ant-junit-1.10.9.jar:/opt/devo/ng-relay/lib/ant-1.10.9.jar:/opt/devo/ng-relay/lib/ant-launcher-1.10.9.jar:/opt/devo/ng-relay/lib/ant-antlr-1.10.9.jar:/opt/devo/ng-relay/lib/picocli-4.5.2.jar:/opt/devo/ng-relay/lib/qdox-1.12.1.jar:/opt/devo/ng-relay/lib/javaparser-core-3.18.0.jar:/opt/devo/ng-relay/lib/jline-2.14.6.jar:/opt/devo/ng-relay/lib/testng-7.4.0.jar:/opt/devo/ng-relay/lib/failureaccess-1.0.1.jar:/opt/devo/ng-relay/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/devo/ng-relay/lib/checker-qual-3.5.0.jar:/opt/devo/ng-relay/lib/error_prone_annotations-2.3.4.jar:/opt/devo/ng-relay/lib/j2objc-annotations-1.3.jar:/opt/devo/ng-relay/lib/syslog4j-0.9.30.jar:/opt/devo/ng-relay/lib/commons-compiler-3.1.2.jar:/opt/devo/ng-relay/lib/httpcore-4.4.13.jar:/opt/devo/ng-relay/lib/trove4j-3.0.3.jar:/opt/devo/ng-relay/lib/jcommander-1.78.jar:/opt/devo/ng-relay/lib/jquery-3.5.1.jar:/opt/devo/ng-relay/lib/spring-jcl-5.3.15.jar:/opt/devo/ng-relay/lib/opentest4j-1.2.0.jar:/opt/devo/ng-relay/lib/okio-jvm-2.8.0.jar:/opt/devo/ng-relay/lib/kotlin-stdlib-1.4.10.jar:/opt/devo/ng-relay/lib/micrometer-core-1.6.1.jar:/opt/devo/ng-relay/lib/simpleclient_common-0.9.0.jar:/opt/devo/ng-relay/lib/kotlin-stdlib-common-1.4.10.jar:/opt/devo/ng-relay/lib/annotations-13.0.jar:/opt/devo/ng-relay/lib/HdrHistogram-2.1.12.jar:/opt/devo/ng-relay/lib/LatencyUtils-2.0.3.jar:/opt/devo/ng-relay/lib/simpleclient-0.9.0.jar com.devo.maduro.Maduro
Check the relay process logs and check for errors
To actively follow the logs as they are being written, you can use the -f
flag. This works as you might expect if you have experience using tail -f
.
Note that you will see several exceptions (Connection refused) until Scoja starts. That’s normal behavior. In order to avoid some redundancy in the logs, you can use the following command:
Response
Confirm that the default ports are listening
Port 12996 TCP where an HTTP server is listening.
Port 12997 TCP where scoja will be listening for local syslog events.
Port 12998 TCP where the relay will be listening for commands from the Devo Relay CLI.
Port 12999 is listening on UDP and is prepared to receive external Netflow traffic (Netflow version < 9).
Ports 13000-13002 are listening on both TCP and UDP.
Port 5140 is listening on TCP and UDP to handle internal relay logs.
Response
Ensure that the client certificate is installed
The client certificate file (client.jks
) should be installed in the relay's /opt/devo/ng-relay/conf/relay/run/keys
folder.
Response
If the certificate download fails, you can force the download again:
Log in to the Devo web application.
Go to Administration → Relays and ELBs → Relays, click the ellipsis menu of the required relay, and select Edit. The Relay Input (Rules) screen appears.
Select the Force Generate New Certificate at the bottom of the screen, then click the Apply Configuration button.
Ensure that scoja has started and relay configuration is being updated correctly
A cron task runs every minute to check if any changes to the relay configuration have been made in the Devo web application. The result of this check appears in the journal of the service.
Response
The "[SUCCESS] Relay activation is still pending, waiting for user to activate the Relay from the web app" message indicates that the relay is registered but has not yet been activated in the Devo web application.
The "[SUCCESS] Downloading and deploying configuration set" message confirms that the relay has been activated and a new configuration has been downloaded.
Once activated, you will be able to define the relay rules using the web application. The relay will begin checking the configuration for updates every 1 minute.
The "[SUCCESS] Relay is active. No pending configuration changes to be applied" message occurs every minute to confirm that no changes were detected in the relay configuration.
During the first configuration, the certificate is also downloaded. If the certificate download fails, you must force the download again.
Error when not enough memory available
Error when no connection with the collector / ELB
Send test events to confirm the relay is receiving and forwarding correctly
Test through the CLI
In order to test that the relay works normally, you can use the CLI command test
that will send several events to the relay.
Response
You can also check that the events are in the table syslog.relay.monitor
.
Test with echo and netcat
Alternatively, echo and netcat commands can be used to send several events to the relay port 13000 to confirm that it is receiving and forwarding events to Devo.
Enter this command on the relay machine to send 100 test events to test.keep.free
table:
To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free
table. This table should contain 100 events.
If there are no events logged in the test.keep.free
table, check the unknown.unknown
table. If there was a problem applying the tags, events will be logged in this table.
Check if the firewall is enabled for the ports
Use the following commands to enable the required sending ports:
Copy
Check relay rules with regexes are efficient
We have updated the relay rules related to several technologies with a more efficient regex.
Paloalto rule - we need to extract the capture group 1 but we don’t need to process the whole message.
Inefficient regex: ^[^,]+,[^,]+,[^,]+,([^,]+).*$
Efficient regex: ^[^,]+,[^,]+,[^,]+,([^,]+)
Rule is misconfigured and the capture group is not pointing to the correct regex field
If you see the following message then check here for details:
Jun 26 20:32:49 relayHostnameHere devo-ng-relay[28831]: File "/opt/devo/ng-relay/conf/relay/run/rules/rule-13030.pconf", line 10, in <module>
Jun 26 20:32:49 relayHostnameHere devo-ng-relay[28831]: rule (
Jun 26 20:32:49 relayHostnameHere devo-ng-relay[28831]: File "/opt/devo/ng-relay/conf/relay/run/rule.py", line 78, in rule
Jun 26 20:32:49 relayHostnameHere devo-ng-relay[28831]: raise "A target tag with data references but no source data"
Jun 26 20:32:49 relayHostnameHere devo-ng-relay[28831]: TypeError: exceptions must be old-style classes or derived from BaseException, not str
Issues pertaining to Java memory error logs
If you query the logs under syslog.scoja.target table and uses Select raw and the following message appears in the “raw” container message (in the Devo UI, this is a column labeled “raw” within the Data Search tab after you run a query) of multiple critical, warnings, and error logs (you can hover over the “raw” column and press “p” for the ‘raw’ message):
Error while sending with [a syslog sender through [a reusing transport syslogger to SSLTransport[on: ConnectTransport[on: NBTCPTransport[on: restrictedproxy.tycoelectronics.com/135.107.16.139:80, with: TCPConf[timeout: 0, reuse address: true, send buffer size: 131072, receive buffer size: <nothing>, keep alive: true, no delay: <nothing>, traffic class: <nothing>]], with: ConnectConf[keep alive: true, user agent: <nothing>]], with: SSLConf[protocols: <nothing>, cipher suites: <nothing>, client authentication: REQUIRE, key managers: [sun.security.ssl.SunX509KeyManagerImpl@3eca82ee], trust managers: [sun.security.ssl.X509TrustManagerImpl@e80505a], secure random: <nothing>]], buffered with FilePairFall[memory max size: 671088640, work directory: /data/logt/buffer/rule, memory flush speed: 16, disk max size: 110100480000, storage size: 502824960, total events: 29477268, done events: 28992817, pending events: 484451, dropped events: 0, is first partial: false, is partial allowed: false], no pending partial, no pending terminator, 0 logs dropped, with no packet limit, with default priority to user.notice, sending timestamp, sending host, with default host usdcalog112, with default tag xcoja, with default terminator \x00]] java.lang.OutOfMemoryError: Java heap space%0A
Try increasing the memory on the Relay. In certain cases increasing it to 6GB of RAM has proven to work.
Relay runs into IndexOutOfBoundsException
When sending events through a relay, it’s possible to run into an IndexOutOfBoundsException if the regex (regular expression) in the relay rule you specified doesn’t conform to the messages that you’re sending to the relay. For example, if you have the event bsssbssssbsss and you have a regex rule that says locate the third b and return everything until the end of the message, the result would be sss. However, if you sent the event bsssbssss using the same regex rule, you would get the IndexOutOfBoundsException because you're trying to access a character that doesn't exist and goes beyond what's contained in the event.
Analyze and verify that the error message indicates an out of bounds exception related to the regex rule.
Analyze the messages that are being sent through the relay and compare them to the regex rule you have defined in the relay to see what inconsistencies there are in terms of the format.
Modify the regex rule in the relay so that it conforms to the format of the events that you’re sending through the relay.
Relay rules not applied
Relay rules don’t seem to be applied on the relay after attempting to add a rule within Devo's web GUI.
Check the table for events showing that a new configuration was downloaded and deployed.
Validate rule added to relay via CLI. Check the syslog.relay.conf table to confirm the rule has been applied to the relay.
If neither shows up then double-check the relay's connectivity.
In the event the buffer becomes full
Usually following a loss in connectivity, it can become impossible for the relay service to resume sending events. In that event, an administrator should:
Stop the relay service: systemctl stop devo-ng-relay
Manually delete the buffer directory located at var/logt/buffer/rule
Manually start the service again and verify that the syslog sender begins shipping logs to the platform: systemctl start devo-ng-relay and checking the sending part with devo-ng-relay-cli and test command
Clearing the buffer will result in events stored in the local buffer not being sent to the platform and being permanently removed.