Operational guidelines

This document is intended to help Endpoint Agent users perform the most common operations, maintenance and troubleshooting actions with the product outside of those that can be performed directly in the EA Manager UI. Check the DEAM manual for specific information for the usage of the web UI.

Solution overview

The Endpoint Agent solution is provided as a package with a set of Ansible playbooks to help with the deployment and operation of the platform.

Once the platform is deployed, and depending on the operation that an operator wants to achieve, there are typically several ways to acknowledge the desired result depending on the action that needs to be taken:

• Use the Ansible playbooks: Changes to some configurations can also be applied using the provided playbooks. The playbooks are mainly oriented to change configurations in the packs or in the EA Manager configuration files. At deployment time, a inventory is created that represents the topology of the EA Manager system. Applying changes via Ansible playbooks using the proper inventory file, makes sure that every node in the system takes the same configuration change.
  a. Operation examples: Modification in default packs, modification of endpoint configuration, EA Manager configuration change.
• Use the WEB UI: Changes to configuration applied in the WEB UI take effect immediately. The changes made via the WEB UI live in the EA Manager MySQL DB. Since the DB is shared, the change will be reflected in every EAM. Endpoints poll the configuration periodically based on the parameter config_refresh defined in the endpoint centralized configuration.
  a. Operation examples: New queries, new packs, endpoint configuration changes, server labeling
• Operations at OS level: Mainly cases related with checking logs, or related to status of the service (start/stop). It is possible to make changes in the EAM configuration files at OS level, but the use of Ansible playbooks is strongly advised to keep the integrity of your system.
  a. Operation examples: Restart EAM service, journal check, environment variable change.

The Ansible playbooks

There are several playbooks included with the EA solution. The Ansible playbooks are a set of automatized tasks that help deploying and configuring the system abstracting the operator of the way that configuration is applied. As an example, EA Manager uses a configuration file placed in /etc/devo-ea-manager/. While it is possible to change the configuration manually in the configuration file, the Ansible playbook will allow you to make the change across all the EA Managers in your topology and keep the system in sync with a single command.

The playbooks are based in YAML files, and every time a YAML file is modified, it is necessary to run a playbook to deploy the changes to the system. To run a playbook, run the command: ansible-playbook -i inventories/<inventory_name.yaml> playbooks/<playbook_name.yaml>.

devo-endpoint-agent

This playbook encloses the whole deployment of the EA solution. This playbook is used during the first deployment and subsequent upgrades. 

From an operation standpoint we should only run this playbook when there is a change in the inventory file parameters that we want to propagate to the live environment:

• Entrypoint change
• Certificate change
• FQDN change
• Password changes

deam-packs

This playbook addresses changes in:

• Centralized endpoint configuration. This configuration is located in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/options.yaml and it defines the parameters that apply to the endpoints. Every endpoint polls periodically to get the configuration from the EA Manager.
• Devo default packs and queries. EA Manager provides a number of default packages and queries that are automatically added to the system after the first deployment. This definitions are located in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/optional-devo-packs/ and can be changed in future versions of the packs along with the evolution of the product. 

Note that executing the playbook deam-packs will deploy both the configuration and the packs. In case that the operator wants to use its own version of the packs (for example, change intervals, or remove queries from the pack) it is recommended to create your own pack (easily created using the WEB UI) or modify the packs using the ansible playbook every time to avoid overriding the data. 

dea-backup

This playbook takes a backup of the relevant items composing the EA Manager solution.

Run this playbook to obtain a backup of:

• OS and System configurations
• FleetDM current configuration
• Docker volumes backup (REDIS and MySQL)

Note that in the case that the topology uses its own separate databases and not the self-generated docker volumes, those backups need to be taken manually by the operator.

Common uses guide

UA Manager operations

Start/Stop EA Manager

To stop the service:

sudo systemctl stop devo-ea-manager

To start the service:

sudo systemctl start devo-ea-manager

To check the status of the service

sudo systemctl status devo-ea-manager

Check EA Manager logs

To check the logs that the manager is writing to the disk, run:

sudo journalctl -u devo-ea-manager

To see the logs flowing in real-time:

sudo journalctl -fu devo-ea-manager

Change configurations

There are several types of configurations that can be changed in the Devo Endpoint Agent product. Each of these configurations have a different way to be applied. 

Change EA Manager options

Options related to the way that EA Manager behaves. They should not be changed unless you have the expertise and know exactly how your changes are going to affect the platform.

To see a list of what parameters can be applied to the EA manager, run the command: fleet --help. The output will show the environment variables that the EA manager accepts

To make a change follow these steps:

• Edit the file /etc/devo-ea-manager/devo-ea-manager.
• Change/add the parameter you are looking to modify.
• Restart EA Manager service: sudo systemctl restart devo-ea-manager.

Modify EA client configuration flags via ansible roles

Options related to the way EA Client behaves. The configuration is centralized in the EA Manager and then pulled from every endpoint periodically.

To make changes in the configuration file, follow the process below in Redeploy default options/packs.

Modify EA client configuration flags via EA Manager WEB UI

• Access the osquery options in https://<DEAM_IP:8080>/settings/osquery (URL for versions prior to EA 1.2.0 is https://<DEAM_IP:8080>/admin/osquery)
  
• The configuration shown in this screenshot is the same that the one included in the options.yaml file in the ansible playbooks. However, the ansible files are collapsed to ease the usage. If you want to make a change here make sure that you make changes in the proper section. Since all the configurations are expanded, each platform will have its own kind of configurations. Please make sure that changes are made in the correct sections:
  a.spec->config : Linux
  b. spec->overrides->platforms->darwin: macOS
  c. spec->overrides->platforms->windows: Windows
• In order to verify that the changes are applied to a specific endpoint once the config_refresh time has passed, the following query can be run: SELECT * FROM osquery_flags.

Change shell only flags at packaging time

Options related to the way the EA Client behaves are not centralized. These flags are used by the EA client at boot time and need to live in a file in the endpoint. These flags are stored in the file osquery.flags in the EA Client installation folder.

The flags can be modified manually client by client for testing purposes. To make configurations in the flags that will be available for download in the manager, follow the process below in Repackage EA Client.

Redeploy default options / packs 

• To make changes in the configurations, update the file $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/options.yaml.
• To make changes in the default packs, so the changes persist in successive deployments, modify the corresponding file inside $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/.
• Once modifications are done, re-deploy the deam-packs role playbook by running:

ansible-playbook -i inventories/<your_inventory>.yaml playbooks/deam-packs.yaml

Repackage EA Client

• Make your changes in the osquery.flags file in $HOME/devo-ea-deployer/playbooks/roles/dea-agent-packager/templates/<platformOS>.
• Re-package the clients using: ansible-playbook -i inventories/<your_inventory>.yaml playbooks/dea-agent-packager.yaml.
• Access https://<DEAM_IP>:8081/ The new agents should be available from download.

Add new log files to be parsed by the EA Client (Files Fetcher)

The Endpoint Agent includes the ability to parse flat log files and uploading them to Devo. Only a handful of files are configured by default, so to add new logs to the processing list, have a look at the configuration options of File Fetcher. Once your changes to the configuration file are ready, deploy them by following the redeploy procedure.

Add new Windows Event Channels to be available to the EA Client

Windows event log capturing is supported in Devo Endpoint Agent. However, due to the huge amount of channels in where Windows OS stores the event logs, only some of them are configured by default as a comma-separated list under the tag all->overrides->patform->windows->options in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/options.yaml.

In order to add a new channel, add the full name of the event channel that you want to add in windows_event_channels.

To know what is the Full Name of an event log channel:

• Open the Event Viewer.
• Navigate to the channel that you want to capture.
• Right-click on it and copy the Full Name field.

Once the channel is added, redeploy the deam-packs playbook (keep in mind that default packs will be overwritten when doing so).

If you are experiencing issues while adding new event channels, perform an EA backing store refresh to ensure the new configurations take effect.

EA Client operations

Start/Stop EA Client

Windows

Stop the service:

Stop-Service -Force -Name "osqueryd"

Start the service:

Start-Service -Force -Name "osqueryd"

Linux

Stop the service:

sudo osqueryctl stop

Start the service:

sudo osqueryctl start

macOS

Stop the service:

sudo osqueryctl stop

Start the service:

sudo osqueryctl start

EA backing store refresh

The Endpoint Agent uses RocksDB to ensure resiliency and buffer data. Data in the backstore is persistent in memory and in the disk. In case you need to flush the data in the RocksDB (so as to re-enable denylisted queries), use the following procedure:

Windows

• Windows menu → Run → services.msc
• Stop service osqueryd
• Delete the DB folder: C:\Program Files\osquery\osquery.db
• Start service osqueryd

Linux / macOS

• Stop osqueryd service: sudo osqueryctl stop
• Delete the DB folder: sudo rm /var/osquery/osquery.db -r
• Start the service: sudo osqueryctl start

Enable native support for syslog linux events

There are two ways to send linux events to Devo using the Endpoint Agent:

1. Configure the specific files to be monitored in the File Fetcher. 

2. Enable osquery native syslog capabilities (requires rsyslog or equivalent installed):
   a.Add pipe configuration in rsyslog:
   i. Add a new file (i.e 60-osquery.conf) to /etc/rsyslog.d:

   $template OsqueryCsvFormat, "%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
   *.* |/var/osquery/syslog_pipe;OsqueryCsvFormat

   ii. Restart rsyslogd to take the new configuration: sudo systemctl restart rsyslogd.
b. Restart osqueryd: sudo osqueryctl restart.

   c. Ensure that the query SELECT *, "events_linux" as __devoSubTag FROM syslog_events; exists in one of your packs (by default, it is included in the DevoEventsPack).
   d. Events will show automatically in box.devo_ea.events_linux

Uninstall EA

Windows

• Open the Windows menu and go to “Add or Remove programs”.
• Select “osquery” and click in “Uninstall”.
• Remove the installation folder (C:\Program Files\osquery).

Linux

• Stop osquery service: sudo osqueryctl stop.
• Uninstall the package using the repo manager. I.e, for ubuntu: sudo apt remove osquery.
• Remove the installation folder (/var/osquery).

macOS

• Unload and remove com.facebook.osquery.plist launchdaemon:

sudo launchctl unload /Library/LaunchDaemons/com.facebook.osqueryd.plist
sudo rm /Library/LaunchDaemons/com.facebook.osqueryd.plist

• Remove files/directories created by osquery installer pkg:

sudo rm -rf /private/var/log/osquery
sudo rm -rf /private/var/osquery
sudo rm /usr/local/bin/osquery*

• Execute the following command: sudo pkgutil --forget com.facebook.osquery