Operational guidelines

This document is intended to help Endpoint Agent users to perform the most common operations and maintenance actions with the product outside of those that can be performed directly in the EA Manager UI. Check the DEAM manual for specific information for the usage of the web UI.

Solution overview

The Endpoint Agent solution is provided as a package with a set of Ansible playbooks to help with the deployment and operation of the platform.

Once the platform is deployed, and depending on the operation that an operator wants to achieve, there are typically several ways to acknowledge the desired result depending on the action that needs to be taken:

• Use the Ansible playbooks: Changes to some configurations can also be applied using the provided playbooks. The playbooks are mainly oriented to change configurations in the packs or in the EA Manager configuration files. At deployment time, an inventory is created that represents the topology of the EA Manager system. Applying changes via Ansible playbooks using the proper inventory file makes sure that every node in the system takes the same configuration change.

  • Operation examples: Modification in default packs, modification of endpoint configuration, EA Manager configuration change.

• Use the WEB UI: Changes to configuration applied in the WEB UI take effect immediately. The changes made via the WEB UI live in the EA Manager MySQL DB. Since the DB is shared, the change will be reflected in every EAM. Endpoints poll the configuration periodically based on the parameter config_refresh defined in the endpoint centralized configuration.

  • Operation examples: New queries, new packs, endpoint configuration changes, server labeling

• Operations at OS level: Mainly cases related to checking logs, or related to the status of the service (start/stop). It is possible to make changes in the EAM configuration files at OS level, but the use of Ansible playbooks is strongly advised to ensure the integrity of your system.

  • Operation examples: Restart EAM service, journal check, environment variable change…

The Ansible playbooks

There are several playbooks included with the EA solution. The Ansible playbooks are a set of automatized tasks that help deploy and configure the system abstracting the operator of the way that configuration is applied. As an example, EA Manager uses a configuration file placed in /etc/devo-ea-manager/. While it is possible to change the configuration manually in the configuration file, the Ansible playbook will allow you to make the change across all the EA Managers in your topology and keep the system in sync with a single command.

The playbooks are based on YAML files, and every time a YAML file is modified, it is necessary to run a playbook to deploy the changes to the system. To run a playbook, run the command: ansible-playbook -i inventories/<inventory_name.yaml> playbooks/<playbook_name.yaml>

devo-endpoint-agent

This playbook encloses the whole deployment of the EA solution. This playbook is used during the first deployment and subsequent upgrades.

From an operation standpoint we should only run this playbook when there is a change in the inventory file parameters that we want to propagate to the live environment, for example:

• Entrypoint change

• Certificate change

• FQDN change

• Password changes

deam-packs

This playbook addresses changes in:

• Centralized endpoint configuration. This configuration is located in the inventory file we used for the deployment (in $HOME/devo-ea-deployer/inventories) and it defines the parameters that apply to the endpoints. Every endpoint polls periodically to get the configuration from the EA Manager.

• Devo default packs and queries. EA Manager provides a number of default packages and queries that are automatically added to the system after the first deployment. This definitions are located in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/optional-devo-packs/ and can be changed in future versions of the packs along with the evolution of the product.

Note that executing the playbook deam-packs will deploy both the configuration and the packs. In case that the operator wants to use its own version of the packs (for example, change intervals, or remove queries from the pack) it is recommended to create your own pack (easily created using the WEB UI) or modify the packs using the ansible playbook every time to avoid overriding the data.

dea-backup

This playbook takes a backup of the relevant items composing the EA Manager solution.

Run this playbook to obtain a backup of:

• OS and System configurations

• FleetDM current configuration

• Docker volumes backup (REDIS and MySQL)

Note that in the event of the topology using its own separate databases and not the self-generated docker volumes, those backups need to be taken manually by the operator.

Common uses guide

EA Manager operations

Start/Stop EA Manager

To stop the service:

sudo systemctl stop devo-ea-manager

To start the service:

sudo systemctl start devo-ea-manager

To check status of the service:

sudo systemctl status devo-ea-manager

Check EA Manager logs

To check the logs that the manager is writing to the disk, run:

sudo journalctl -u devo-ea-manager

To see the logs flowing in real time:

sudo journalctl -fu devo-ea-manager

Change configurations

There are several type of options that can be changed in Devo Endpoint Agent product. Each of these configurations have a different way to be applied.

Change EA Manager options

Options related to the way that EA Manager behaves. They should not be changed unless you have the expertise and know exactly how your changes are going to affect the platform.

To see a list of what parameters can be applied to the EA manager, run the command: fleet --help. The output will show the environment variables that the EA manager accepts.

To modify/add any parameter:

• Edit the file /etc/devo-ea-manager/devo-ea-manager

• Change/add the parameter you are looking to modify

• Restart EA Manager service: sudo systemctl restart devo-ea-manager

Modify EA client configuration flags via ansible roles

Configuration related to the way EA Client behaves. The configuration is centralized in the EA Manager, and then pulled from every endpoint periodically.

To make changes in the configuration file, follow the process in Redeploy default options / packs .

Modify EA client configuration flags via EA Manager WEB UI

• Access the osquery configuration in https://<DEAM_IP:8080>/ -> settings -> Global agent options

  Open

  
  Open

  

• The configuration shown in this screenshot is the same that the default one included in the deam-packs/defaults/main.yml file in the ansible playbooks. However, the ansible files are collapsed to ease the usage. If you want to make a change here make sure that you make changes in the proper section. Since all the configurations are expanded, each platform will have its own kind of configurations. Please make sure that changes are made in the correct sections:

  • config : Linux

  • overrides->platforms->darwin: macOS

  • overrides->platforms->windows: Windows

• In order to verify that the changes are applied to a specific endpoint once the config_refresh time has passed, the following query can be run: SELECT * FROM osquery_flags

Change shell only flags at packaging time

Options related to the way the EA Client behaves that are not centralized. These flags are used by the EA client at boot time and need to live in a file in the endpoint. These flags are stored in the file osquery.flags in the EA Client installation folder.

The flags can be modified manually client by client for testing purposes. To make configurations in the flags that will be available for download in the manager, follow the process in Repackage EA Client.

Redeploy default options / packs via ansible

• To make changes in the configuration options, update the inventory file used in the deployment $HOME/devo-ea-deployer/inventories under the vars level.

• To make changes in the default packs, so the changes persist in successive deployments, modify the corresponding file inside $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/

• Once modifications are done, enable your virtual environment and re-deploy the deam-packs role playbook by running:

  source "/opt/ansible-2.9/venv/bin/activate" ansible-playbook -i inventories/<your_inventory>.yaml playbooks/deam-packs.yaml

Repackage EA Client

• Make your changes in the osquery.flags file in $HOME/devo-ea-deployer/playbooks/roles/dea-agent-packager/templates/<platformOS>

• Enable your viertual envrionment using: source "/opt/ansible-2.9/venv/bin/activate"

• Re-package the clients using: ansible-playbook -i inventories/<your_inventory>.yaml playbooks/dea-agent-packager.yaml

• Access https://<DEAM_IP>:8081/ , the new agents should be available from download.

Add new log files to be parsed by the EA Client (Files Fetcher)

The Endpoint Agent includes the ability to parse flat log files and upload them to Devo. Only a handful of files are configured by default, so to add new logs to the processing list, have a look at the configuration options of Files fetcher. Once your changes to the configuration file are ready, deploy them by following the redeploy procedure.

Add new Windows Event Channels to be available to the EA Client

Windows event log capturing is supported in Devo Endpoint Agent. However, due to the huge amount of channels in where Windows OS stores the event logs, only some of them are configured by default as a comma-separated list, you can modify them under the tag deam_fleet_config_agent_opts_win_windows_event_channels in the inventory we used for deployment.

In order to add a new channel, add the full name of the event channel that you want to add in windows_event_channels.

To know what is the Full Name of a event log channel:

• Open the Event Viewer

• Navigate to the channel that you want to capture.

• Right click on it and copy the Full Name field.

Once the channel is added, redeploy the deam-packs playbook (keep in mind that default packs will be overwritten when doing so).

Change password of root user in MySQL service (self creating dockerized MySQL service by deployment)

We strongly recommend that you change the default root password of MySQL service for security reasons.

The default value for the password of the root user configured in MySQL service when it is deployed as docker container is very insecure.

If this value is not changed during deployment configuration then replace it now with a more robust one.

The next steps are designed to do it without redeploying the EA manager and related components.

1. Check if the MySQL service has configured the default password. Run the command in the server where MySQL container was deployed.

   sudo /usr/local/bin/docker-compose -f /srv/deam-internal-services/docker-compose.yaml exec mysql mysql -u root --password=toor

   If the login action ended without an error with a similar output such as:

   mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 10 Server version: 5.7.37 MySQL Community Server (GPL) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>

   Then the default password (toor) was configured for the root user of MySQL. Continue with next steps to change it.

2. Change the password in docker-compose file. Run the commands in the server where the MySQL container was deployed.
   Edit /srv/deam-internal-services/docker-compose.yaml file

   Find the configuration property with the MySQL root password:

   Replace the value of the password with a stronger value (<<NEW PASSWORD HERE>> in the code snipped bellow). Be carefully to maintain line indentation.

3. Change the password in running MySQL service. Run the commands in the server where MySQL container was deployed.
   Log in to the MySQL service as the root user using the default password:

   Change the root password using the same value selected in the previous step (replacing <<NEW PASSWORD HERE>>)

   Press Ctrl + D to exit from MySQL-client session

4. Additionally you should edit your inventory file and add deam_mysql_root_password var with the value of the password previously chosen, to prevent changes to the default password during future aplatform upgrade or similar procedure.

EA Client operations

Start/Stop EA Client

Windows

Stop the service:

Stop-Service -Force -Name "osqueryd"

Start the service:

Start-Service -Force -Name "osqueryd"

Linux/macOS

Stop the service:

sudo osqueryctl stop

Start the service:

sudo osqueryctl start

EA backing store refresh

The Endpoint Agent uses RocksDB to ensure resiliency and buffer data. Data in the backstore is persistent in memory and in the disk. In case you need to flush the data in the RocksDB (i.e: to re-enable denylisted queries), use the following procedure:

Windows

• Windows menu → Run → services.msc

• Stop service osqueryd

• Delete the DB folder: C:\Program Files\osquery\osquery.db

• Start service osqueryd

Linux / macOS

• Stop osqueryd service: sudo osqueryctl stop

• Delete the DB folder: sudo rm /var/osquery/osquery.db -r

• Start the service: sudo osqueryctl start

Enable native support for syslog linux events

There are two ways to send linux events to Devo using the Endpoint Agent:

1. Configure the specific files to be monitored in the File Fetcher.

2. Enable osquery native syslog capabilities (requires rsyslog or equivalent installed):

   1. Add pipe configuration in rsyslog:

      1. Add a new file (i.e 60-osquery.conf) to /etc/rsyslog.d:

      2. Restart rsyslogd to take the new configuration: sudo systemctl restart rsyslogd

   2. Restart osqueryd: sudo osqueryctl restart

   3. Ensure that the query SELECT *, "events_linux" as __devoSubTag FROM syslog_events; exists in one of your packs (by default, it is included in the DevoEventsPack)

   4. Events will show automatically in box.devo_ea.events_linux

Modify EA client configuration flags

OSQuery has also some options decentralized to make easier to adapt EA client to more demanding hosts.

To make changes in the configuration file, we need to move to the following path:

Linux:

Windows:

Darwin:

and modify the corresponding flags in osquery.flags file.

Once modifications are done, we need to stop and start the service following in Start/Stop EA Client section

Uninstall EA

Windows

• Open the Windows menu and go to “Add or Remove programs”

• Select “osquery” and click in “Uninstall”

• Remove the installation folder (C:\Program Files\osquery)

Linux

• Stop osquery service: sudo osqueryctl stop

• Uninstall the package using the repo manager. For ubuntu, for example: sudo apt remove osquery

• Remove the installation and configuration folders:

  • /var/osquery

  • /

  • opt/osquery

  • /etc/osquery
    
    

macOS

• Unload and remove io.osquery.agent.plist launchdaemon:

• Remove files/directories created by osquery installer pkg:

• Execute the following command sudo pkgutil --forget io.osquery.agent.plist