Endpoint Agent 1.2 upgrade procedure

[ 1 Processes query ] [ 2 UA Manager V1.1.0 → V1.2.0 upgrade procedure ] [ 3 Files Fetcher configuration change ] [ 4 Endpoints upgrade ] [ 5 Post-installation steps ] [ 6 UA Manager rollback procedure ]

The Endpoint Agent solution v1.2.0 includes the following:

Name change: This product, formerly known as Universal Agent, is now known as Endpoint Agent.
Default data structures changed in Devo platform.
osquery version upgrade to 4.9.0
FleetDM version upgrade to 3.13.0

The Universal Agent solution v1.2.0 has significant changes including the change of the data structures from box.devo_ua to box.devo_ea and from devo.ua to devo.ea. Existing data structures will remain usable but every customer is recommended to upgrade as ua structures will be deprecated in the future.

Processes query

Upgrade of osquery to version 4.9.0 includes a name change in the field elevated_token in the processes table (used in the running processes queries in the status pack).

Query for osquery 4.9.0:

SELECT pid, name, path, cmdline, state, cwd, root, uid, gid, euid,
      egid, suid, sgid, on_disk, start_time, parent, pgroup, nice, elevated_token,
      upid, uppid, cpu_type, cpu_subtype, "status.processes" as __devoSubTag
    FROM processes;

Query for osquery 4.8.0:

 SELECT pid, name, path, cmdline, state, cwd, root, uid, gid, euid,
      egid, suid, sgid, on_disk, start_time, parent, pgroup, nice, is_elevated_token,
      upid, uppid, cpu_type, cpu_subtype, "status.processes" as __devoSubTag
    FROM processes;

During the upgrade process, if you reuse the same certificates in the EAM, there will be a situation where the EAM is already upgraded but the agents will still be in the previous version (based in osquery 4.8.0). This situation will cause errors to appear in the EA logs, so the queries using that table have been marked automatically to be executed only in agents running osquery 4.9.0. Data generated by this query (box.devo_ea.status.processes) will stop flowing until the endpoints are upgraded.

UA Manager V1.1.0 → V1.2.0 upgrade procedure

The upgrade process will overwrite some existing folders in the UA Manager filesystem. Make sure you backup the folders indicated in the instructions below.

Please read entirely the following instructions before starting the upgrade procedure.

Back up your existing devo-ua-deployer folder. If you used one of the existing example inventory files (py3-1host-example.yaml), make sure to have a copy at hand.
Back up your existing /etc/devo-ua-manager folder in all DUAM instances.
Make sure that your options.yaml file is consistent with your actual configuration file. If you have made changes to the osquery configuration using the EAM Web UI instead of the options.yaml there could be discrepancies. To download your current configurations for comparison you can run the following command:
sudo env ROOT_CA_CERTIFICATE=/etc/devo-ua-manager/certs/devo-ua-manager.crt /usr/local/bin/duam_fleetctl.sh https://devo-ua-manager:8080 <your_user> '<your_password>' get options
Default packs will be changed during the deployment process so if you have made any changes to those, make sure to translate them to the YAML files in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/optional-devo-packs/.
If you have provided your MySQL instance (required on HA environments) you should make a backup of it and save in a secure place in order to allow fallback procedure. If you are using the docker version of MySQL automatically generated by the previous version of EA deployer, you can skip this step as Ansible will take care of backing up your database.
Making a backup from MySQL data depends on your environment, but a good starting point is use mysqldump command. For example:
Move to the $HOME folder: cd $HOME.
Download the new package into the current folder. Former and actual deployment folders should co-exist during the migration process as in the following image:
Untar the downloaded package: tar -xzvf devo-ea-deployer-1.2.0.1.tgz.
Move to the new deployer folder: cd devo-ea-deployer.
Create a inventory compliant with EA 1.2.0 using the miginvt tool. This step, depends on the deployment model that you have followed when the previous version was deployed. When the default fqdn is used with self-generated certificates, fqdn is named as devo-ua-manager and the certificate is generated accordingly. The miginvt will replace the FQDN with devo-ea-manager if requested, by adding a specific argument to the command line. It is important to note that if the FQDN and certificates are changed, all the agents need to be reinstalled to use the new certificate. Choose one of the following options according to your deployment scenario (use python or python3 according to your deployment platform):
a. If you are using your own certificates: python3 tools/miginvt.py -i ../devo-ua-deployer/inventories/<your_inventory>.yaml -o inventories/<your_new_inventory>.yaml
b. If you are using certificates previously generated by UA Deployer:
i. If you want to re-use the existing certificates (FQDN does not change therefore we can reuse the certificate): python3 tools/miginvt.py -i ../devo-ua-deployer/inventories/<your_inventory>.yaml -o inventories/<your_new_inventory>.yaml
ii. If you want to adapt FQDN to the new naming (FQDN changes, therefore we need to create new certificates): python3 tools/miginvt.py -i ../devo-ua-deployer/inventories/<your_inventory>.yaml -o inventories/<your_new_inventory>.yaml --new-certs-fqdn-devo-ua-manager
If you are using your own certificates instead of letting the the deployer generate self-signed certificates, copy the certificates into $HOME/devo-ea-deployer/provided-deam-certs. If you are using self-signed certificates, you can ignore this step.
Make sure that the new inventory is created in the inventories folder. Below is an example of a migrated inventory

13. Run the dea-migrations playbook: ansible-playbook -i inventories/<your_new_inventory>.yaml playbooks/dea-migrations.yaml. NOTE: If the deployment uses self-generated dockers, they will be regenerated in this step so the service will be down for some time (less than a minute).
14. The dea-migrations playbook generates a number of backups. It is recommended to download them and keep them safe as they will be required during the rollback procedure. In order to list all the backups generated execute: ansible all -m command -a "find /var/devo-ea-deployer/migrations" -e "ansible_become=true" -i inventories/<your_new_inventory>.yaml. The result should look like the example below:

15. Copy the Devo domain certificates to the new deployment folder: sudo cp $HOME/devo-ua-deployer/domain-certs/ $HOME/devo-ea-deployer/ -r.

16. Ensure that all you configs in $HOME/devo-ua-deployer/playbooks/roles/duam-packs/files/devo-packs/options.yaml are included in $HOME/devo-ea-deployer/playbooks/roles/duam-packs/files/devo-packs/options.yaml. In the following example, we have some File Fetcher configurations applied as opposed to the default:

To check the differences between your options.yaml and the default one you can run the following command:

17. Merge your extra configurations to $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/options.yaml. NOTE: Do not override the whole file, as there could be a tag conflict when doing so. The tag box.devo_ua should be kept as changed to box.devo_ea.

18. Run the upgrade playbook to EA 1.2.0: ansible-playbook -i inventories/<your_new_inventory>.yaml playbooks/devo-endpoint-agent.yaml.

Files Fetcher configuration change

As part of the upgrade procedure, it is required to change the destination structures for File Fetcher manually if there is any pattern sending data to box.devo_ua.files. If you are not using Files Fetcher or you are not sending any file data to box.devo_ua.files, you can ignore this section.

Due to security mechanisms implemented in the File Fetcher, the configuration cannot be changed directly. Files Fetcher stores the configured patterns, and does not allow a tag update on-the-fly. In order to change the configuration to the new Devo tables, follow the below procedure:

Edit $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/devo-packs/options.yaml.
Remove all the pattern lines under the tag: box.devo_ea.files. If all your pattern tags are pointing to box.devo_ea.files instead of commenting out all the patterns, let at least one pattern tag pointing to a fake route (i.e: “c:\tmp\fake.txt").
Redeploy the options.yaml using: ansible-playbook -i inventories/newInventory.yaml playbooks/deam-packs.yaml.
Wait until the configuration is updated in every host. Configuration refresh cycle is defined in the parameter config_refresh in the options.yaml file (by default , 900 seconds).
Restore again the initial pattern lines in the config file.
Redeploy the options.yaml using: ansible-playbook -i inventories/newInventory.yaml playbooks/deam-packs.yaml.
Wait until the configuration is propagated.
Data should appear in Devo in the new tables. NOTE: Existing files will be ingested again in the new structure from the beginning.

Endpoints upgrade

The new UA package is available in https://DUAM_IP:8081.
Extract the files in the .zip / .tgz file.
Run the installation script:
a. Windows: Run install.ps1
b. Linux/Mac: Run install.sh as root user, for example run with sudo.

Post-installation steps

Check that the UA Manager interface is accessible.
Check that all the packs and queries are available:
a. The default packs are installed in ENABLED state. If you want to disable any of them, simply deactivate them via the administration web UI.
b. Any other additional packs should continue being available.
Check that the data collected from the endpoints flows into Devo after the installation.
(Optional) Delete $HOME/devo-ua-deployer as it is not required anymore. It is recommended to let the platform soak for some time before deleting this folder.

UA Manager rollback procedure

Due to the nature of this upgrade, depending on the point the procedure failed, the commands to be executed can differ. If the dea-migrations playbook has been already executed and service names have been changed, execute the commands with the references to ea, otherwise execute the commands that make reference to ua.

In the unlikely event of the UA Manager node ending up in a faulty status after an upgrade process, apply the following rollback procedure:

Ensure that you have all the backup files created by hand and created by dea-migrations Ansible playbook.
Stop all manager and agent repository services:

3. Stop internal services (mysql and redis) deployed in no-HA mode (dockerized mode)

4. Remove all configuration from managers:

5. Restore database backup only if you have provide your MySQL instance.

If you have provided your MySQL instance (required on HA environments) you should restore the backup previously created.

Restoring the backup from MySQL data depends on your environment, but a good starting point is use mysql command. For example:

Remember that you should ignore this step if MySQL database was deployed by devo-ua-deployer in dockerized moded.

6. Restore manager and agent-repository backup files.

Set appropriate value for date pattern (2021-08-12 in the example):

7. Restore MySQL docker volume only if your environment is no-HA mode (dockerized mode).

Set appropriate value for date pattern (2021-08-12 in the example):

8. Start internal services only if your environment is no-HA (dockerized mode):

9. Purge fleet packages:

Debian distro based only:

RedHat distro based only:

10. Install OLD duam-fleet directly from remote repository:

Debian distro based only:

RedHat distro based only:

11. Restart all manager and agent repository services:

12. Check and clean up /etc/hosts and hostname.

13. Check that service is working and agents are properly enrolled.

14. Uninstall OSQuery in each agent target host and reinstall agents if you upgraded agents to new version too.