Document toolboxDocument toolbox

Universal Agent 1.1.0 upgrade procedure

Owned by Juan Tomás Alonso Nieto (Deactivated)
May 23, 2022
5 min read

The Universal Agent solution v1.1.0 includes the most recent (4.8.0) version of osquery. While the new version of the Universal Agent Manager ensures compatibility with older agents, it is mandatory to upgrade those to utilize the latest functionality added in this new release (for example, native Devo tagging support for the Files Fetcher component).

UAM 1.1.0 certificates

It has been detected that RSA-4096 certificates cause the manager to spend a significant amount of resources (CPU) in the TLS decryption process. This can cause overload in the UA Manager when in scenarios with a significant amount of managed endpoints and/or connections (such as frequent queries).

Because of this, UA Manager will use now ECDSA certificates by default, which provide an equivalent level of security to RSA keys while using smaller ones. When using your own certificates, we recommend that you use this algorithm to avoid overloads in the UA Manager.

Changing the certificate in the UA Manager will cause all connected agents to be unable to reach the UA Manager, so it is required to change certificates in the agents as well when doing this task. To minimize service downtime when planning for the upgrade, please refer to the section below on how to complete the upgrade procedure with certificates change.

If the UA Manager is already working and there are no plans to grow in terms of endpoints or connections towards the UA Manager, then it is not necessary to replace the existing certificates.

UA Manager V1.0 → V1.1.0 upgrade procedure

The upgrade process will overwrite some existing folders in the UA Manager filesystem. Make sure you back up the folders indicated in the instructions below. Please read all of the instructions before starting the upgrade process.
  1. Back up your existing devo-ua-deployer folder. If you used one of the existing example inventory files (for example, py3-1host-example.yaml) make sure to have a copy at hand so you can use it during the upgrade process.
  2. Back up your existing /etc/devo-ua-manager folder in all DUAM instances.
  3. Download the new package and copy in the same directory level where the existing devo_ua_deployer folder is listed.
  4. Untar the downloaded package so that it overrides the contents of the existing devo_ua_deployer folder.

  5. Make sure that your inventory file still exists in $HOME/devo-ua-deployer/inventories and that it still applies to your environment.
    a. If your inventory file is not there or it has been overwritten, copy the existing inventory file from the backup folder created in step 1.
    b. If necessary, consider adding the optional pack features to your inventory file, otherwise all packs remain enabled.
    c. Check that the Devo domain certificates are still in place in $HOME/devo-ua-deployer/domain-certs.
    d. If you want to purge older versions of the agents (previous agents won't be available for download anymore), add the following variable to the inventory file:

    all:
  vars:
    ...
    dua_ap_builder_purge_old_packages: true


    e. Important: If you are using self-signed RSA certificates (or your own certificates) and do not want to upgrade to the new certificates, add the following variable to the inventory file, otherwise the new ECDSA certificates will be deployed replacing the old ones (agents will lose connectivity until the new agent version is deployed):

    self_signed_certificates_private_key_algorithm: RSA
  6. Verify any configuration changes applied to the baseline configuration (such as enabled/disabled packs) and check whether those need to continue to be applied after the upgrade. Both packs and configurations will be updated as part of the platform upgrade process, so they might be lost after the installation is completed. If there are specific changes in the centralized configuration (options.yaml), those changes need to be merged before running the playbook. If you want to keep those changes applied, please contact Devo to get specific support for that migration.
  7. Run universal-agent playbook using the following command (replace your_inventory.yaml for the actual name of your inventory file): ansible-playbook -i inventories/your_inventory.yaml playbooks/devo-universal-agent.yaml.

Endpoints upgrade

  1. The new UA package is available in https://DUAM_IP:8081.
  2. Extract the files in the .zip / .tgz file.
  3. Run the installation script:   
     a. Windows: Run install.ps1
     b. Linux/Mac: Run install.sh as root user, for example run with sudo.

Post-installation steps

  1. Check that the UA manager interface is accessible.
  2. Check that all the packs and queries are available:
    a. The default packs are installed in ENABLED state. If you want to disable any of them, simply deactivate them via the administration web UI.
    b. Any other additional packs should continue being available.
  3. Check that the data collected from the endpoints flows into Devo after the installation.

UA Manager V1.0.X → V1.1.0 upgrade procedure with certificates change and reduced service downtime

The upgrade process will overwrite some existing folders in the UA Manager filesystem. Make sure you back up the folders indicated in the instructions below. Please read all of the instructions before starting the upgrade process.
  1. Back up your existing devo-ua-deployer folder. If you used one of the existing example inventory files (for example, py3-1host-example.yaml) make sure to have a copy at hand so you can use it during the upgrade process.
  2. Back up your existing /etc/devo-ua-manager folder in all DUAM instances.
  3. Download the new package and copy it in the same directory level where the existing devo_ua_deployer folder is listed.
  4. Untar the downloaded package so that it overrides the contents of the existing devo_ua_deployer folder.
  5. Make sure that your inventory file still exists in $HOME/devo-ua-deployer/inventories, and that it still applies to your environment.
    a. If your inventory file is not there or it has been overwritten, copy the existing inventory file from the backup folder created in step 1.
    b. If required, consider adding the optional pack features to your inventory file as in Initial pack configuration, otherwise all packs remain enabled.
    c. Check that the Devo domain certificates are still in place in $HOME/devo-ua-deployer/domain-certs.
    d. If you want to purge older version of the agents (previous agents will not be available for download anymore) add the following variable to the inventory file: 
all:
  vars:
    ...
    dua_ap_builder_purge_old_packages: true

6. Generate new ECDSA certificates by running: ansible-playbook -i inventories/<your_inventory_name>.yaml playbooks/duam-new-selfsignedcerts.yaml

7. The previous step generated the new ECDSA certificates. Back up $HOME/devo-ua-deployer/new-duam-certs in a safe path.

8. Generate new agents with ECDSA + RSA certs: ansible-playbook -i inventories/<your_inventory_name>.yaml playbooks/dua-agent-packager.yaml

9. Update the agents to UA1.1.0. You can follow the Endpoint upgrade procedure. After this point, you will be running UA Manager 1.0.1 with osquery 4.8.0.

10. If your architecture contains a Load Balancer, add the new ECDSA certificate to the trusted certificates.

11. To avoid certificates merge to happen again, delete the following folder: sudo rm /etc/devo-ua-manager/new-certs/ -r in all DUAM instances.

12. Copy ECDSA certificate to provided-duam-certs folder in the deployment folder: sudo cp new-duam-certs/* provided-duam-certs/ in the devo-ua-deployer extracted path.

13. Replace in your inventory file selfsigenedcertificates: with providedcertificates:

14. Run the devo-universal-agent playbook using the following command (replace your_inventory.yaml for the actual name of your inventory file): ansible-playbook -i inventories/your_inventory.yaml playbooks/devo-universal-agent.yaml

Endpoints upgrade

  1. The new UA package is available in https://DUAM_IP:8081.
  2. Extract the files in the .zip / .tgz file.
  3. Run the installation script:   
     a. Windows: Run install.ps1
     b. Linux/Mac: Run install.sh

Post-installation steps

  1. Check that the UA manager interface is accessible.
  2. Check that all the packs and queries are available:
    a. The default packs are installed in ENABLED state. If you want to disable any of them, simply deactivate them via the administration web UI.
    b. Any other additional packs should continue being available.
  3. Check that the data collected from the endpoints flows into Devo after the installation.

UA Manager rollback procedure

In the unlikely event of the UA Manager node ending up in a faulty status after an upgrade process, apply the following rollback procedure:

1. Restore your original devo-ua-deployer folder from the backup location.

2. Connect to all manager hosts and restore /etc/devo-ua-manager folder from the backup location.

3. Connect to all manager hosts and ensure duam-fleet is purged (choose the right one for your linux distro):

sudo systemctl apt remove -f duam-fleet

or

sudo yum remove -f duam-fleet

4. Ensure packs are removed. Step is required only on first duam-manager host.

sudo rm -f /var/devo-ua-manager/packs

5. Change current directory to $HOME/devo-ua-deployer and run the following command (replace your_inventory.yaml for the actual name of your inventory file):  ansible-playbook -i inventories/your_inventory.yaml playbooks/devo-universal-agent.yaml