Endpoint Agent 1.3 upgrade procedure

Login with email instead of username

Login to the Web UI of EA Manager has changed due to the upgrade of the underlying FleetDM version to 4.10.0. From now on, the login will be done using the admin email instead of the admin username.

After the upgrade, the default admin email will be in use, so if you want to change the email used for login, create a new admin user in the Web UI and delete the previous one after the platform is upgraded (you can view how to do here). After the new user is created, replace it in your ansible inventory file so the next time a playbook is run the new user will be used.

Options.yaml is replaced by config.yaml

Due to the FleetDM upgrade to 4.10.0, the options.yaml format has changed slightly. Make sure that changes performed in options.yaml are merged properly to the new config.yaml file when doing the upgrade of the system.

To do this merge smoothly, a script is provided and it is referenced during the upgrade procedure. This script will override the configurations with the new standard configurations to be applied from EA 1.3. The existing Files Fetcher configuration and Windows events channels will be maintained after the merge, but other values will be overwritten to apply newer configuration standards. If you have any custom configuration that you want to keep over the upgrade make sure to add it manually after the merge is done.

Extended pre-check validation

In order to improve the pre-requisites checks for validation, a new optional method of connectivity validation towards Devo is included. When enabled, the Devo EA Deployer will try to send data to Devo and will show an error if:

• The connectivity with Devo does not work.

• EA deployer cannot send data to Devo.

• Devo has not received all the events that EA deployer has sent.

Since in an upgrade scenario we assume that the platform is already working, there is no specific requirement to enable this validation.

Ansible operations moved to Virtual Environment

To avoid issues with different versions of Python and Ansible, from EA 1.3 we recommend using Virtual Environments when running the deployment procedure and other operations over the platform. The upgrade procedure will prepare the Virtual Environment in your existing EA Manager.

EA Manager V1.2.X → V1.3.0 upgrade procedure

1. Back up your existing devo-ea-deployer folder. If you used one of the existing example inventory files (e.g., py3-1host-example.yaml) make sure to have a copy at hand.

2. Back up your existing /etc/devo-ea-manager folder in all DEAM instances.

3. Make sure that your options.yaml file is consistent with your actual configuration file. If you have made changes to the osquery configuration using the EAM Web UI instead of the options.yaml there could be discrepances. To download your current configurations for comparison you can run the following command:

   sudo env ROOT_CA_CERTIFICATE=/etc/devo-ea-manager/certs/devo-ea-manager.crt /usr/local/bin/deam_fleetctl.sh https://devo-ea-manager:8080 <your_user> '<your_password>' get options

4. Save your current deployment folder as a backup:

   sudo mv $HOME/devo-ea-deployer $HOME/devo-ea-deployer-backup

5. If you have provided your MySQL instance (required on HA environments) you should make a backup of it and save in a secure place in order to allow fallback procedure. If you are using the docker version of MySQL automatically generated by the previous version of EA deployer, you can skip this step as Ansible will take care of backing up your database.

   Making a backup from MySQL data depends on your environment, but a good starting point is use mysqldump command. For example:

   mysqldump -u root -p kolide --result-file=migrations-backup.sql

6. Move to the $HOME folder:

7. Download the new package into the current folder.

8. Untar the downloaded package:

9. Copy the Devo domain certs to the new deployment folder:

10. Move to the new deployer folder:

11. Run the preparation script to create the new virtual environment:

12. Activate the virtual environment with the command displayed at the end of the execution of the previous command. Typically it will be something like:

13. Run migration inventory tool. After the migration script is executed, File Fetcher configurations and Windows Event Channel configurations should be kept, however other configuration flags will be overwritten with the new configuration baseline for EA 1.3. If there is any custom configuration that you want to keep specifically for your environment, add it manually after the script is run.

14. Run migration playbook:

15. The dea-migrations playbook generates a number of backups. It is recommended to download them and keep them safe as they will be required during the rollback procedure. In order to list all the backups generated execute: ansible all -m command -a "find /var/devo-ea-deployer/migrations" -e "ansible_become=true" -i inventories/deployment_inventory.yaml . Result should look like below:

16. Run the upgrade playbook to EA 1.3:

Endpoints upgrade

1. The new EA package is available in https://DEAM_IP:8081.

2. Extract the files in the .zip / .tgz file.

3. Run the installation script:

   1. Windows: Run install.ps1 in a Powershell session with elevated privileges.

   2. Linux/Mac: Run install.sh as root user, for example run with sudo.

Post-installation steps

1. Check that the EA Manager interface is accessible.

2. Access Settings and add your organization name. Save the changes clicking on Update Settings at the bottom of the screen.
   
   

   Open

   

3. OPTIONAL: After deployment, and since older versions of EA deployer used the default email, it’s possible thar the login will be used with the default email. If you want to change that, create a new admin user and update the deployment inventory to run with the new administrator user.

   1. Create a user.

   2. Modify inventory file to add the admin information (add the parameter lines that do not exist in the current deployment inventory), so future runs of the playbooks work correctly:

4. Check that all the packs and queries are available:

   1. The default packs are installed in ENABLED state. If you want to disable any of them, simply deactivate them via the administration web UI.

   2. Any other additional packs should continue being available.

5. Check that the data collected from the endpoints flows into Devo after the installation.

EA Manager rollback procedure

In the unlikely event of the EA Manager node ending up in a faulty status after an upgrade process, apply the following rollback procedure:

1. Ensure that you have all the backup files created by hand and created by dea-migrations ansible playbook.

2. Move to deployment folder:

3. Stop all manager and agent repository services:

    

4. Stop internal services (mysql and redis) deployed in no-HA mode (dockerized mode):

5. Remove all configuration from managers:

    

6. Restore database backup only if you have provided your MySQL instance. Ignore this step if MySQL database was deployed by devo-ea-deployer in dockerized mode.

   If you have provided your MySQL instance (required on HA environments) you should restore the backup previously created.

   Restoring the backup from MySQL data depends on your environment, but a good starting point is use mysql command. For example:

7. Restore manager and agent-repository backup files.

   Set appropriate value for date pattern 2021-08-12 in the example:

    

8. Restore MySQL docker volume only if your environment is no-HA mode (dockerized mode).

   Set appropriate value for date pattern 2021-08-12 in the example (check the date in the step 13 of the deployment phase):

    

9. Start internal services only if your environment is no-HA (dockerized mode):

10. Purge fleet packages:

    • Debian distro based only:

    • RedHat distro based only:

11. Install OLD deam-fleet directly from remote repository:

    • Debian distro based only:

       

    • RedHat distro based only:

       

12. Restart all manager and agent repository services:

     

13. Check that service is working and agents are properly enrolled.

14. Uninstall OSQuery in each agent target host and reinstall agents if you upgraded agents to the new version too.