Introduction
Devo Relay 2.X.X is based on a Java process that runs on a Linux machine. The Java process internally uses a syslog engine called Scoja.
The relay Java process runs as a service in the Linux OS and it is managed by the system manager systemd. Once running, the relay listens for TCP/UDP connections on ports and, depending on the port, modifies the syslog events and forwards them to the Devo central collector / ELB. The main role of the relay is to tag events based on a defined rule, then forward the tagged events to Devo.
The relay comes with a few ports that are preconfigured with rules for managing specific types of events. For example, port 13000 is preconfigustartured to forward the events received directly to Devo without any other processing. This port should be used for events that are properly tagged in the data source. You can also create custom rules to tag and/or filter the events sent to the associated port on the relay. Each rule is stored in one of these two folders:
/opt/devo/ng-relay/conf/relay/run/rules/→ Rules managed from the Devo web application./opt/devo/ng-relay/conf/relay/run/unrules/→ Rules not managed from the Devo web application. These rules are directly managed on the relay machine itself.
About the relay logs files
Main process logs
As the relay runs as a systemd service, the logs can be consulted using the following command:
sudo journalctl -u devo-ng-relay [-f]
These log files are also sent to Devo by default and can be consulted in the table box.unix.
Scoja logs
In addition, several Scoja log files are generated and stored in:
/var/logt/local/<year>/<month>/<day>/<relayName>/syslog/scoja/
main.log→ main process log filesource.log→ logs related to the sources configuredthread.log→ threads opened for each sourcestats.log→ statistics about received and sent eventstarget.log→ logs related to the sending process
These logs are also sent to Devo and can be consulted in the tables syslog.scoja.*. However, we recommend keeping them in the local machine for a few days.
The retention configuration can be set by updating the following file:
/opt/devo/ng-relay/conf/relay/maduro.properties
Troubleshooting checklist
Make sure the relay process (Java) is running
ps aux | grep java | grep -v grep
Response
devo 4001 0.3 56.8 4746500 573532 ? Ssl 05:31 0:31 java -Dfile.encoding=UTF8 -DMADURO_CONF=/opt/devo/ng-relay/conf/ @/opt/devo/ng-relay/conf/relay/devo-ng-relay.jvmoptions -classpath /opt/devo/ng-relay/lib/maduro-server-2.1.0.jar:/opt/devo/ng-relay/lib/maduro-api-client-1.4.0.jar:/opt/devo/ng-relay/lib/maduro-common-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-server-1.5.2.jar:/opt/devo/ng-relay/lib/api-clients-serrea-0.2.1.jar:/opt/devo/ng-relay/lib/jenga-echo-1.4.5.jar:/opt/devo/ng-relay/lib/jenga-monitoring-1.4.5.jar:/opt/devo/ng-relay/lib/httpclient-4.5.13.jar:/opt/devo/ng-relay/lib/crypass-1.0.2.jar:/opt/devo/ng-relay/lib/logback-syslog4j-1.0.0.jar:/opt/devo/ng-relay/lib/logback-classic-1.2.10.jar:/opt/devo/ng-relay/lib/logback-core-1.2.10.jar:/opt/devo/ng-relay/lib/janino-3.1.2.jar:/opt/devo/ng-relay/lib/commons-compress-1.21.jar:/opt/devo/ng-relay/lib/commons-configuration-1.10.jar:/opt/devo/ng-relay/lib/spring-boot-autoconfigure-2.6.3.jar:/opt/devo/ng-relay/lib/spring-boot-2.6.3.jar:/opt/devo/ng-relay/lib/spring-context-5.3.15.jar:/opt/devo/ng-relay/lib/spring-tx-5.2.19.RELEASE.jar:/opt/devo/ng-relay/lib/jinjava-2.6.0.jar:/opt/devo/ng-relay/lib/feign-jackson-10.2.3.jar:/opt/devo/ng-relay/lib/jackson-dataformat-yaml-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-databind-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-annotations-2.13.1.jar:/opt/devo/ng-relay/lib/jackson-core-2.13.1.jar:/opt/devo/ng-relay/lib/feign-slf4j-10.2.3.jar:/opt/devo/ng-relay/lib/slf4j-api-1.7.32.jar:/opt/devo/ng-relay/lib/mysql-connector-java-8.0.28.jar:/opt/devo/ng-relay/lib/javax.annotation-api-1.3.2.jar:/opt/devo/ng-relay/lib/lutier-1.18.0.jar:/opt/devo/ng-relay/lib/scoja-beep-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-compression-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-rpc-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-client-1.5.2.jar:/opt/devo/ng-relay/lib/scoja-rpc-xc-1.4.0.jar:/opt/devo/ng-relay/lib/scoja-cc-1.4.0.jar:/opt/devo/ng-relay/lib/jython-standalone-2.7.2b3.jar:/opt/devo/ng-relay/lib/commons-codec-1.13.jar:/opt/devo/ng-relay/lib/feign-okhttp-10.2.3.jar:/opt/devo/ng-relay/lib/feign-core-10.2.3.jar:/opt/devo/ng-relay/lib/groovy-ant-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-astbuilder-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-cli-picocli-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-groovysh-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-console-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-datetime-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-groovydoc-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-docgenerator-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-jmx-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-json-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-jsr223-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-macro-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-nio-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-servlet-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-sql-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-swing-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-templates-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-test-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-test-junit5-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-testng-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-xml-3.0.8.jar:/opt/devo/ng-relay/lib/groovy-3.0.8.jar:/opt/devo/ng-relay/lib/guava-30.0-jre.jar:/opt/devo/ng-relay/lib/javassist-3.26.0-GA.jar:/opt/devo/ng-relay/lib/jsoup-1.14.2.jar:/opt/devo/ng-relay/lib/re2j-1.2.jar:/opt/devo/ng-relay/lib/commons-lang3-3.10.jar:/opt/devo/ng-relay/lib/commons-net-3.3.jar:/opt/devo/ng-relay/lib/java-ipv6-0.17.jar:/opt/devo/ng-relay/lib/annotations-3.0.1.jar:/opt/devo/ng-relay/lib/big-math-2.0.0.jar:/opt/devo/ng-relay/lib/spring-aop-5.3.15.jar:/opt/devo/ng-relay/lib/spring-beans-5.3.15.jar:/opt/devo/ng-relay/lib/spring-expression-5.3.15.jar:/opt/devo/ng-relay/lib/spring-core-5.3.15.jar:/opt/devo/ng-relay/lib/snakeyaml-1.30.jar:/opt/devo/ng-relay/lib/junit-4.13.2.jar:/opt/devo/ng-relay/lib/junit-platform-launcher-1.7.0.jar:/opt/devo/ng-relay/lib/junit-platform-engine-1.7.0.jar:/opt/devo/ng-relay/lib/junit-platform-commons-1.7.0.jar:/opt/devo/ng-relay/lib/junit-jupiter-engine-5.7.0.jar:/opt/devo/ng-relay/lib/junit-jupiter-api-5.7.0.jar:/opt/devo/ng-relay/lib/hamcrest-core-1.3.jar:/opt/devo/ng-relay/lib/micrometer-registry-prometheus-1.6.1.jar:/opt/devo/ng-relay/lib/commons-logging-1.2.jar:/opt/devo/ng-relay/lib/commons-lang-2.6.jar:/opt/devo/ng-relay/lib/okhttp-4.9.1.jar:/opt/devo/ng-relay/lib/protobuf-java-3.19.4.jar:/opt/devo/ng-relay/lib/jsr305-3.0.2.jar:/opt/devo/ng-relay/lib/ant-junit-1.10.9.jar:/opt/devo/ng-relay/lib/ant-1.10.9.jar:/opt/devo/ng-relay/lib/ant-launcher-1.10.9.jar:/opt/devo/ng-relay/lib/ant-antlr-1.10.9.jar:/opt/devo/ng-relay/lib/picocli-4.5.2.jar:/opt/devo/ng-relay/lib/qdox-1.12.1.jar:/opt/devo/ng-relay/lib/javaparser-core-3.18.0.jar:/opt/devo/ng-relay/lib/jline-2.14.6.jar:/opt/devo/ng-relay/lib/testng-7.4.0.jar:/opt/devo/ng-relay/lib/failureaccess-1.0.1.jar:/opt/devo/ng-relay/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/devo/ng-relay/lib/checker-qual-3.5.0.jar:/opt/devo/ng-relay/lib/error_prone_annotations-2.3.4.jar:/opt/devo/ng-relay/lib/j2objc-annotations-1.3.jar:/opt/devo/ng-relay/lib/syslog4j-0.9.30.jar:/opt/devo/ng-relay/lib/commons-compiler-3.1.2.jar:/opt/devo/ng-relay/lib/httpcore-4.4.13.jar:/opt/devo/ng-relay/lib/trove4j-3.0.3.jar:/opt/devo/ng-relay/lib/jcommander-1.78.jar:/opt/devo/ng-relay/lib/jquery-3.5.1.jar:/opt/devo/ng-relay/lib/spring-jcl-5.3.15.jar:/opt/devo/ng-relay/lib/opentest4j-1.2.0.jar:/opt/devo/ng-relay/lib/okio-jvm-2.8.0.jar:/opt/devo/ng-relay/lib/kotlin-stdlib-1.4.10.jar:/opt/devo/ng-relay/lib/micrometer-core-1.6.1.jar:/opt/devo/ng-relay/lib/simpleclient_common-0.9.0.jar:/opt/devo/ng-relay/lib/kotlin-stdlib-common-1.4.10.jar:/opt/devo/ng-relay/lib/annotations-13.0.jar:/opt/devo/ng-relay/lib/HdrHistogram-2.1.12.jar:/opt/devo/ng-relay/lib/LatencyUtils-2.0.3.jar:/opt/devo/ng-relay/lib/simpleclient-0.9.0.jar com.devo.maduro.Maduro
Check the relay process logs and check for errors
To actively follow the logs as they are being written, you can use the -f flag. This works as you might expect if you have experience using tail -f.
sudo journalctl -u devo-ng-relay -f
Note that you will see several exceptions (Connection refused) until Scoja starts. That’s normal behavior. In order to avoid some redundancy in the logs, you can use the following command:
sudo journalctl -u devo-ng-relay -f | grep -v "<190>"
Response
Mar 09 15:19:41 ubuntu-bionic systemd[1]: Started Next-Gen Relay. ... Mar 09 15:19:42 ubuntu-bionic devo-ng-relay[5054]: 15:19:42.311 INFO [none] -- com.devo.maduro.Maduro : Starting Maduro v2.0.0-SNAPSHOT on ubuntu-bionic with PID 5054 (/opt/devo/devo-ng-relay/lib/maduro-server-2.0.0-SNAPSHOT.jar started by devo in /) Mar 09 15:19:42 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:19:42 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.333 INFO [none] -- o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'threadPoolConfigUpdaterScheduler' Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.792 INFO [none] -- o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'threadPoolPurgeLogsScheduler' Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.816 INFO [none] -- o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'threadPoolKeepAliveScheduler' Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.969 INFO [none] -- com.devo.maduro.Maduro : Started Maduro in 2.041 seconds (JVM running for 2.798) Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.971 INFO [none] -- com.devo.maduro.Maduro : Maduro server is running. Mar 09 15:19:43 ubuntu-bionic devo-ng-relay[5054]: 15:19:43.980 INFO [none] -- c.d.l.j.monitoring.info.ApplicationInfo : Service info file not found at the default location Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.082 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : Mapping 'com.devo.lib.jenga.echo.handlers.SystemHandler' to 'system' Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.085 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : Mapping 'com.devo.maduro.server.healthcheck.KeepAlive' to '/system/keepalive' Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.086 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : Starting server Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.089 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : Server started on port: 12996 Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.093 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : Listening at /0:0:0:0:0:0:0:0:12996 with endpoints: Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.094 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : ➤ /system: class com.devo.lib.jenga.echo.handlers.SystemHandler Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.094 INFO [none] -- com.devo.lib.jenga.echo.JengaEcho : ➤ /system/keepalive: class com.devo.maduro.server.healthcheck.KeepAlive Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.104 INFO [81469641436b] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.105 INFO [81469641436b] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay null Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] ... Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.670 INFO [81469641436b] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Relay was gracefully stopped so now is being started. Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: TCPNetSyslog: DOWN (org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)) Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: 15:19:44.763 INFO [81469641436b] -- c.d.m.server.relay.ConfigUpdaterService : Maduro context status: APPLYING_CONFIG Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:19:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.103 INFO [c41650a5b47f] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.104 INFO [c41650a5b47f] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay null Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.166 INFO [c41650a5b47f] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Downloading and deploying configuration set Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: [org.productivity.java.syslog4j.SyslogRuntimeException: java.net.ConnectException: Connection refused (Connection refused)] Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.700 INFO [c41650a5b47f] -- c.d.m.server.relay.RelayConfigFileUtil : Extracting rule rules/target.pconf... Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.701 INFO [c41650a5b47f] -- c.d.m.server.relay.RelayConfigFileUtil : Extracting rule rules/rule-13000.pconf... Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.701 INFO [c41650a5b47f] -- c.d.m.server.relay.RelayConfigFileUtil : Extracting rule rules/rule-13001.pconf... Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.702 INFO [c41650a5b47f] -- c.d.m.server.relay.RelayConfigFileUtil : Extracting rule rules/rule-13002.pconf... Mar 09 15:20:44 ubuntu-bionic devo-ng-relay[5054]: 15:20:44.702 INFO [c41650a5b47f] -- c.d.m.server.relay.RelayConfigFileUtil : Extracting rule rules/rule-12997.pconf... ... Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.source.net.udp: Starting UDP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.thread: Starting thread 1 for UDP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.thread: Starting thread 2 for UDP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.source.net.tcp: Starting Selecting TCP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.source.net.udp: Opening server socket for UDP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.thread: Starting thread 1 for Selecting TCP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.thread: Starting thread 2 for Selecting TCP source listening at 0.0.0.0:13000 Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.source.net.tcp: Opening server socket for Selecting TCP source listening at 0.0.0.0:13000 ... Mar 09 15:20:48 ubuntu-bionic devo-ng-relay[5054]: Mar 9 15:20:48 ubuntu-bionic syslog.scoja.main: Scoja started Mar 09 15:21:44 ubuntu-bionic devo-ng-relay[5054]: 15:21:44.103 INFO [67f7a9c19efa] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 15:21:44 ubuntu-bionic devo-ng-relay[5054]: 15:21:44.104 INFO [67f7a9c19efa] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay test-julio-ubuntu18 Mar 09 15:21:44 ubuntu-bionic devo-ng-relay[5054]: 15:21:44.197 INFO [67f7a9c19efa] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Relay is active. No pending configuration changes to be applied Mar 09 15:21:44 ubuntu-bionic devo-ng-relay[5054]: 15:21:44.199 INFO [67f7a9c19efa] -- c.d.m.server.relay.ConfigUpdaterService : Maduro context status: ACTIVE ...
Confirm that the default ports are listening
Port 12996 TCP where an HTTP server is listening.
Port 12997 TCP where scoja will be listening for local syslog events.
Port 12998 TCP where the relay will be listening for commands from the Devo Relay CLI.
Port 12999 is listening on UDP and is prepared to receive external Netflow traffic (Netflow version < 9).
Ports 13000-13002 are listening on both TCP and UDP.
Port 5140 is listening on TCP and UDP to handle internal relay logs.
sudo netstat -atunp | grep java | grep -v ESTABLISHED
Response
tcp6 0 0 127.0.0.1:5140 :::* LISTEN 5054/java tcp6 0 0 :::12996 :::* LISTEN 5054/java tcp6 0 0 :::12997 :::* LISTEN 5054/java tcp6 0 0 :::12998 :::* LISTEN 5054/java tcp6 0 0 :::13000 :::* LISTEN 5054/java tcp6 0 0 :::13001 :::* LISTEN 5054/java tcp6 0 0 :::13002 :::* LISTEN 5054/java udp6 0 0 127.0.0.1:5140 :::* 5054/java udp6 0 0 127.0.0.1:5141 :::* 5054/java udp6 0 0 :::12997 :::* 5054/java udp6 0 0 :::12999 :::* 5054/java udp6 0 0 :::13000 :::* 5054/java udp6 0 0 :::13001 :::* 5054/java udp6 0 0 :::13002 :::* 5054/java
Ensure that the client certificate is installed
The client certificate file (client.jks) should be installed in the relay's /opt/devo/ng-relay/conf/relay/run/keys folder.
sudo ls -al /opt/devo/ng-relay/conf/relay/run/keys
Response
total 16 drwxr-xr-x 2 devo devo 4096 Mar 9 10:42 . drwxr-xr-x 4 devo devo 4096 Mar 9 15:20 .. -rw-r--r-- 1 devo devo 7691 Mar 9 15:20 client.jks
If the certificate download fails, you can force the download again:
Log in to the Devo web application.
Go to Administration → Relays and ELBs → Relays, click the ellipsis menu of the required relay, and select Edit. The Relay Input (Rules) screen appears.
Select the Force Generate New Certificate at the bottom of the screen, then click the Apply Configuration button.
Ensure that scoja has started and relay configuration is being updated correctly
A cron task runs every minute to check if any changes to the relay configuration have been made in the Devo web application. The result of this check appears in the journal of the service.
sudo journalctl -u devo-ng-relay -f | grep -Ev "(<190>|Connection refused)"
Response
Mar 09 16:14:55 ubuntu-bionic devo-ng-relay[3988]: 16:14:55.920 INFO [1b6261efab31] -- com.devo.maduro.server.ActionVisitor : Setting Up.... Mar 09 16:15:40 ubuntu-bionic devo-ng-relay[3988]: 16:15:40.149 INFO [62d8d76a91b8] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 16:15:40 ubuntu-bionic devo-ng-relay[3988]: 16:15:40.151 INFO [62d8d76a91b8] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay test-julio-ubuntu18 Mar 09 16:15:40 ubuntu-bionic devo-ng-relay[3988]: 16:15:40.151 INFO [62d8d76a91b8] -- o.a.c.c.PropertiesConfiguration : Reloading configuration. URL is file:/opt/devo/devo-ng-relay/conf/relay/relay.properties Mar 09 16:15:40 ubuntu-bionic devo-ng-relay[3988]: 16:15:40.532 INFO [62d8d76a91b8] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay from the web app Mar 09 16:15:40 ubuntu-bionic devo-ng-relay[3988]: 16:15:40.534 INFO [62d8d76a91b8] -- c.d.m.server.relay.ConfigUpdaterService : Maduro context status: PENDING Mar 09 16:16:40 ubuntu-bionic devo-ng-relay[3988]: 16:16:40.149 INFO [c334a58327fa] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 16:16:40 ubuntu-bionic devo-ng-relay[3988]: 16:16:40.151 INFO [c334a58327fa] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay test-julio-ubuntu18 Mar 09 16:16:40 ubuntu-bionic devo-ng-relay[3988]: 16:16:40.217 INFO [c334a58327fa] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Relay activation is still pending, waiting for user to activate the Relay Mar 09 16:16:40 ubuntu-bionic devo-ng-relay[3988]: 16:16:40.218 INFO [c334a58327fa] -- c.d.m.server.relay.ConfigUpdaterService : Maduro context status: PENDING ... Mar 09 16:29:40 ubuntu-bionic devo-ng-relay[3988]: 16:29:40.149 INFO [520978c81e52] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 16:29:40 ubuntu-bionic devo-ng-relay[3988]: 16:29:40.150 INFO [520978c81e52] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay test-julio-ubuntu18 Mar 09 16:29:40 ubuntu-bionic devo-ng-relay[3988]: 16:29:40.224 INFO [520978c81e52] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Downloading and deploying configuration set ... Mar 09 18:03:08 ubuntu-bionic devo-ng-relay[4968]: 18:03:08.603 INFO [0e4a70202842] -- c.d.m.server.relay.ConfigUpdaterService : Checking relay status Mar 09 18:03:08 ubuntu-bionic devo-ng-relay[4968]: 18:03:08.604 INFO [0e4a70202842] -- c.d.m.server.relay.ConfigUpdaterService : Updater running. Checking status of the relay test-julio Mar 09 18:03:08 ubuntu-bionic devo-ng-relay[4968]: 18:03:08.673 INFO [0e4a70202842] -- c.d.m.server.relay.ConfigUpdaterService : [SUCCESS] Relay is active. No pending configuration changes to be applied Mar 09 18:03:08 ubuntu-bionic devo-ng-relay[4968]: 18:03:08.674 INFO [0e4a70202842] -- c.d.m.server.relay.ConfigUpdaterService : Maduro context status: ACTIVE
The "[SUCCESS] Relay activation is still pending, waiting for user to activate the Relay from the web app" message indicates that the relay is registered but has not yet been activated in the Devo web application.
The "[SUCCESS] Downloading and deploying configuration set" message confirms that the relay has been activated and a new configuration has been downloaded.
Once activated, you will be able to define the relay rules using the web application. The relay will begin checking the configuration for updates every 1 minute.
The "[SUCCESS] Relay is active. No pending configuration changes to be applied" message occurs every minute to confirm that no changes were detected in the relay configuration.
During the first configuration, the certificate is also downloaded. If the certificate download fails, you must force the download again.
Error when not enough memory available
Mar 09 16:29:46 ubuntu-bionic devo-ng-relay[3988]: 16:29:46.515 ERROR [520978c81e52] -- c.d.m.server.relay.CorrectiveService : The relay cannot start. Insufficient memory configured. The relay needs 128Mb for memory buffering at least
Error when no connection with the collector / ELB
Mar 09 17:05:22 ubuntu-bionic devo-ng-relay[4580]: Mar 9 17:05:22 ubuntu-bionic syslog.scoja.target.syslog: Error while sending with [a syslog sender through [a reusing transport syslogger to SSLTransport[on: NBTCPTransport[on: collector-sgt.devo.io:443, with: TCPConf[timeout: 10000, reuse address: true, send buffer size: <nothing>, receive buffer size: <nothing>, keep alive: true, no delay: <nothing>, traffic class: <nothing>]], with: SSLConf[protocols: <nothing>, cipher suites: <nothing>, client authentication: REQUIRE, key managers: [sun.security.ssl.SunX509KeyManagerImpl@2ba6a13b], trust managers: [sun.security.ssl.X509TrustManagerImpl@3a10529c], secure random: <nothing>]], buffered with FilePairFall[memory max size: 33554432, work directory: /var/logt/buffer/me, memory flush speed: 16, disk max size: 134217728, storage size: 1081344, total events: 1, done events: 0, pending events: 1, dropped events: 0, is first partial: false, is partial allowed: false], no pending partial, no pending terminator, 0 logs dropped, with no packet limit, with default priority to user.notice, sending timestamp, sending host, with default host ubuntu-bionic, with default tag xcoja, with default terminator \x00]] org.scoja.client.LoggingException: java.nio.channels.UnresolvedAddressException%0A at org.scoja.client.ReusingTransportSyslogger.ilog(ReusingTransportSyslogger.java:128)%0A at org.scoja.client.Syslogger.log(Syslogger.java:242)%0A at org.scoja.server.target.SysloggerTarget.process(SysloggerTarget.java:126)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.Link.process(Link.java:98)%0A at org.scoja.server.core.CounterLink.process(CounterLink.java:54)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.Link.process(Link.java:98)%0A at org.scoja.server.action.ActionLink.process(ActionLink.java:20)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.Link.process(Link.java:98)%0A at org.scoja.server.filter.FilteredLink.process(FilteredLink.java:24)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.Link.process(Link.java:98)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.SyncLink.propagate(SyncLink.java:59)%0A at org.scoja.server.core.Link.process(Link.java:98)%0A at org.scoja.server.core.Link.propagate(Link.java:105)%0A at org.scoja.server.core.EventContext.process(EventContext.java:64)%0A at org.scoja.server.core.EventQueue.run(EventQueue.java:125)%0A at java.lang.Thread.run(Thread.java:748)%0ACaused by: java.nio.channels.UnresolvedAddressException%0A at sun.nio.ch.Net.checkAddress(Net.java:104)%0A at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)%0A at org.scoja.trans.nbtcp.NBTCPLine$Client.connect(NBTCPLine.java:171)%0A at org.scoja.trans.lc.LCLine.connect0(LCLine.java:250)%0A at org.scoja.trans.lc.LCLine.connect(LCLine.java:106)%0A at org.scoja.trans.ssl.SSLLine.connect(SSLLine.java:144)%0A at org.scoja.trans.lc.LCLine.connect0(LCLine.java:250)%0A at org.scoja.trans.lc.LCLine.connect(LCLine.java:106)%0A at org.scoja.client.ReusingTransportSyslogger.ensureConnection(ReusingTransportSyslogger.java:140)%0A at org.scoja.client.ReusingTransportSyslogger.ilog(ReusingTransportSyslogger.java:101)%0A ... 20 more%0A Mar 09 17:05:22 ubuntu-bionic devo-ng-relay[4580]: Mar 9 17:05:22 ubuntu-bionic syslog.scoja.target.syslog: Disabling [a syslog sender through [a reusing transport syslogger to SSLTransport[on: NBTCPTransport[on: collector-sgt.devo.io:443, with: TCPConf[timeout: 10000, reuse address: true, send buffer size: <nothing>, receive buffer size: <nothing>, keep alive: true, no delay: <nothing>, traffic class: <nothing>]], with: SSLConf[protocols: <nothing>, cipher suites: <nothing>, client authentication: REQUIRE, key managers: [sun.security.ssl.SunX509KeyManagerImpl@2ba6a13b], trust managers: [sun.security.ssl.X509TrustManagerImpl@3a10529c], secure random: <nothing>]], buffered with FilePairFall[memory max size: 33554432, work directory: /var/logt/buffer/me, memory flush speed: 16, disk max size: 134217728, storage size: 1081344, total events: 1, done events: 0, pending events: 1, dropped events: 0, is first partial: false, is partial allowed: false], no pending partial, no pending terminator, 0 logs dropped, with no packet limit, with default priority to user.notice, sending timestamp, sending host, with default host ubuntu-bionic, with default tag xcoja, with default terminator \x00]] upto Tue Mar 09 17:05:24 UTC 2021
Send test events to confirm the relay is receiving and forwarding correctly
Test through the CLI
In order to test that the relay works normally, you can use the CLI command test that will send several events to the relay.
DEVO-RELAY:> test
Response
Sending test data to the relay... Retrieving the test data using the query API... | ---> SUCCESS Test successful. The relay is working correctly.
You can also check that the events are in the table syslog.relay.monitor.
Test with echo and netcat
Alternatively, echo and netcat commands can be used to send several events to the relay port 13000 to confirm that it is receiving and forwarding events to Devo.
Enter this command on the relay machine to send 100 test events to test.keep.free table:
for i in `seq 1 100`; do (echo "<14>Jan 10 00:00:00 xxx test.keep.free: test event $i"|nc -w0 localhost 13000); done
To confirm that the events were processed correctly, open Devo, go to Data Search, and locate the test.keep.free table. This table should contain 100 events.
If there are no events logged in the test.keep.free table, check the unknown.unknown table. If there was a problem applying the tags, events will be logged in this table.
Check if the firewall is enabled for the ports
Use the following commands to enable the required sending ports:
sudo firewall-cmd --zone=public --permanent --add-port=12999-13030/tcp
Copy
sudo firewall-cmd --reload
Check relay rules with regexes are efficient
We have updated the relay rules related to several technologies with a more efficient regex.
Paloalto rule - we need to extract the capture group 1 but we don’t need to process the whole message.
Inefficient regex: ^[^,]+,[^,]+,[^,]+,([^,]+).*$
Efficient regex: ^[^,]+,[^,]+,[^,]+,([^,]+)