Endpoint Agent 1.5.0 upgrade procedure

Changes in EA 1.5.0

There is not any significant change that affects the upgrade procedure in EA 1.5.0. Check out our release notes for the list of new features, changes and bug fixes.

EA Manager upgrade from versions lower than 1.4.0

The upgrade from versions older than V1.4.0 can not performed in one single step; you must upgrade first to V1.4.0 in order to upgrade to V1.5.0.

Follow the instructions here upgrade to V1.4.0 and then continue with the next chapter.

EA Manager V1.4.0 → V1.5.0 upgrade procedure

The upgrade process will overwrite some existing folders in the EA Manager filesystem. Make sure you backup the folders indicated in the instructions below.

Read the entirety of the following instructions before starting the upgrade procedure.

Back up your existing devo-ea-deployer folder. If you used one of the existing example inventory files (inventories/deployment_inventory.yaml) make sure to have a copy at hand.
Back up your existing /etc/devo-ea-manager and /etc/devo-ea folders in all DEAM instances.
Make sure that agent configuration properties in your inventory file are consistent with your actual configuration file. If you have made changes to the osquery configuration using the EAM Web UI instead of the inventory file there could be discrepancies. To download your current configurations for comparison you can run the following command :
```
sudo env ROOT_CA_CERTIFICATE=/etc/devo-ea-manager/certs/devo-ea-manager.crt /usr/local/bin/deam_fleetctl.sh https://<DEA FQDN>:8080 <admin_email> '<your_password>' get config
```
The agent configuration properties are saved in spec -> agent_options -> config section of the output from previous command.

Save your current deployment folder as a backup:

sudo mv $HOME/devo-ea-deployer $HOME/devo-ea-deployer-backup

If you have provided your MySQL instance (required on HA environments) you should make a backup of it and save in a secure place in order to allow fallback procedure. If you are using the docker version of MySQL automatically generated by the previous version of EA deployer, you can skip this step as Ansible will take care of backing up your database.
Making a backup from MySQL data depends on your environment, but a good starting point is to use mysqldump command. For example:
```
mysqldump -u root -p devoea --result-file=migrations-backup.sql
```
Move to the $HOME folder :
```
cd $HOME
```
Download the new package in the current folder.
Untar the downloaded package :
```
tar -xzvf devo-ea-deployer-1.5.0.tgz
```

Copy the Devo domain certs to the new deployment folder:

cp devo-ea-deployer-backup/domain-certs/* devo-ea-deployer/domain-certs/

Move to the new deployer folder:
```
cd devo-ea-deployer
```
Activate, if you did not activate before, the ansbile-2.9 virtual environment.
```
source "/opt/ansible-2.9/venv/bin/activate"
```
Run migration inventory tool. After the migration script is executed, File Fetcher configurations and Windows Event Channel configurations should be kept. However, other configuration flags will be overwritten with the new configuration baseline for EA 1.3. If there is any custom configuration that you want to keep specifically for your environment, add it manually after the script is run.
```
python tools/miginvt.py -i ../devo-ea-deployer-backup/inventories/<< EA_140_inventory name >>.yaml -o inventories/deployment_inventory.yaml
```

Run migration playbook:

ansible-playbook -i inventories/deployment_inventory.yaml playbooks/dea-migrations.yaml

The dea-migrations playbook generates a number of backups. It is recommended to download them and keep them safe as they will be required during the rollback procedure. In order to list all the backups generated execute: ansible all -m command -a "find /var/devo-ea-deployer/migrations" -e "ansible_become=true" -i inventories/deployment_inventory.yaml . Result should look like below:
```
devo-ea-manager | CHANGED | rc=0 >>
/var/devo-ea-deployer/migrations
/var/devo-ea-deployer/migrations/migrations-backup.2023-02-02.tgz
/var/devo-ea-deployer/migrations/deam-internal-services_mysqldata.2023-02-02.tgz
/var/devo-ea-deployer/migrations/migrations-backup_fleet-items.2023-02-02.tgz
```

Run the upgrade playbook to EA 1.5.0:

ansible-playbook -i inventories/deployment_inventory.yaml playbooks/devo-endpoint-agent.yaml

Endpoints upgrade

The new EA package is available in https://DEAM_IP:8081.
Extract the files in the .zip / .tgz file.
Run the installation script:
1. Windows: Run install.ps1 in a Powershell session with elevated privileges.
2. Linux/Mac: Run install.sh as root user, for example run with sudo.

Post-installation steps

Check that the EA Manager interface is accessible.
Check that all the packs and queries are available:
1. The default packs are installed in ENABLED state. If you want to disable any of them, simply deactivate them via the administration web UI.
2. Any other additional packs should continue being available.
Check that the data collected from the endpoints flows into Devo after the installation.

EA Manager rollback procedure (V1.5.0 → V1.4.0)

In the unlikely event of the EA Manager node ending up in a faulty status after an upgrade process, apply the following rollback procedure:

Ensure that you have all the backup files created by hand and created by dea-migrations ansible playbook.
Move to deployment folder:
```
cd $HOME/devo-ea-deployer
```
Activate, if you did not activate before, the ansbile-2.9 virtual environment:
```
source "/opt/ansible-2.9/venv/bin/activate"
```

Stop all manager and agent repository services:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a "systemctl stop devo-ea-manager"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone -m command -a "systemctl stop dea-invent"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deaagentpackager -m command -a "systemctl stop nginx"

Stop internal services (mysql and redis) deployed in no-HA mode (dockerized mode):

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deamintsrvs -m command -a \
    "/usr/local/bin/docker-compose -f /srv/deam-internal-services/docker-compose.yaml down -v --rmi all"

Remove all configuration from managers:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    "rm -fr /etc/devo-ea-manager/ /etc/devo-ea /var/devo-ea-manager/ /var/log/devo-ea-manager/ /srv/deam-internal-services"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deamintsrvs -m command -a \
    "rm -fr /srv/deam-internal-services"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deaagentpackager -m command -a \
    "rm -fr /etc/nginx"

Restore database backup only if you have provided your MySQL instance. Ignore this step if MySQL database was deployed by devo-ea-deployer in dockerized mode.
If you have provided your MySQL instance (required on HA environments) you should restore the backup previously created.
Restoring the backup from MySQL data depends on your environment, but a good starting point is use mysql command. For example:
```
mysql -u root -p insecure < migrations-backup.sql
```

Restore manager and agent-repository backup files.

Set appropriate value for date pattern 2023-02-02 in the example:

backupDate="2023-02-02"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas:deaagentpackager:deamintsrvs -m command -a \
    "tar -zxvf /var/devo-ea-deployer/migrations/migrations-backup.$backupDate.tgz -C /"

Restore MySQL docker volume only if your environment is no-HA mode (dockerized mode).

Set appropriate value for date pattern 2023-02-02 in the example (check the date in the step 14 of the deployment phase):

backupDate="2023-02-02"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deamintsrvs -m command -a \
    "docker run --rm -it \
         -v /var/devo-ea-deployer/migrations/deam-internal-services_mysqldata.$backupDate.tgz:/input_backup.tgz \
         -v deam-internal-services_mysqldata:/dest \
         alpine \
         /bin/sh -c 'cd /dest && tar zx -f /input_backup.tgz'"

Start internal services only if your environment is no-HA (dockerized mode):

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deamintsrvs -m command -a \
    "/usr/local/bin/docker-compose -f /srv/deam-internal-services/docker-compose.yaml up -d"

Purge fleet packages.

Debian distro based only:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    "apt purge -y deam-fleet"

RedHat distro based only:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    "yum remove -y deam-fleet"

Install OLD deam-fleet directly from remote repository.

Debian distro based only:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    'curl -u "dea-repo:Th3R3p0!" \
        "https://d2ur64jmn3k7yt.cloudfront.net/mgm/deam-fleet_1.3.0_amd64.deb" \
        -o /tmp/deam-fleet_1.3.0_amd64.deb'
ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    'apt install -y /tmp/deam-fleet_1.3.0_amd64.deb'

RedHat distro based only:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    'curl -u "dea-repo:Th3R3p0!" \
        "https://d2ur64jmn3k7yt.cloudfront.net/mgm/deam-fleet-1.3.0-1.x86_64.rpm" \
        -o /tmp/deam-fleet-1.3.0-1.x86_64.rpm'
ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    'yum install -y /tmp/deam-fleet-1.3.0-1.x86_64.rpm'

Restart all manager and agent repository services:

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone:devoeamanagerreplicas -m command -a \
    "systemctl restart devo-ea-manager"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" devoeamanagerserverone -m command -a \
    "systemctl restart dea-invent"

ansible -i ./inventories/deployment_inventory.yaml -e "ansible_become=true" deaagentpackager -m command -a \
    "systemctl restart nginx"

Check that service is working and agents are properly enrolled.
Uninstall OSQuery in each agent target host and reinstall agents if you upgraded agents to the new version too.