Document toolboxDocument toolbox

Generic deployment guidelines

Owned by Juan Tomás Alonso Nieto (Deactivated)
Jul 06, 2022
25 min read
[ 1.1 The inventory ] [ 1.2 Standalone deployment template ] [ 1.3 Distributed deployment template ] [ 2 Deployment credentials ] [ 2.1 Using SSH user and password ] [ 2.2 Using SSH private key ] [ 2.3 Initial packs configuration (new from EA v1.1.0) ] [ 2.4 Python Interpreter ] [ 2.5 Firewalld and ufw management ] [ 2.6 Endpoint Agent <> EA Manager communication ] [ 2.7 EA Manager <> Devo communication ] [ 2.8 Other special deployment cases ] [ 2.9 High availability configuration ] [ 2.10 Databases installation ] [ 2.11 Devo EA Manager users and passwords ] [ 2.12 Certificates ] [ 2.13 Appendix A - Example inventory files ] [ 2.14 Appendix B - Security Recommendations ]

Depending on the desired deployment scenarios, there are a handful of different alternatives that can be approached for a particular customer installation. The Devo Endpoint Agent installation process is flexible so as to allow a number of deployment models by using a specific inventory file. In this document, the most common scenarios are depicted, as well as the list of additions to the inventory that are required to fulfill those. Thus, each subsection can be considered as an installation step that offers different alternatives for completion, and not a step-by-step installation procedure.

If there is any special need not included in this guideline, please contact Devo for more detailed information.

The inventory

The deployment of the Endpoint Agent is done via Ansible playbooks. The playbooks instrument all needed steps to deploy the platform according to your needs, defining what and where you want to deploy every component. To do so, the Ansible playbooks will use a YAML inventory file to understand what tasks are needed to accomplish this. The YAML inventory file is the base of the deployment and it is very important to have clear what type of deployment you want to achieve and to create a valid inventory file before starting the deployment process.

When preparing a new deployment of the Endpoint Agent, it is important to have in mind some questions:

  • If you are looking for PoC or non-productive environment deployment, the standalone All-in-One deployment is provided, where the ansible playbooks will create two dockers with the ancillary services for EA Manager to run properly.

  • If you are looking for a small production environment and you do not need HA, you can use the standalone deployment, but use your own ancillary services (MySQL and REDIS) to give the system more robustness and security.

  • If you are looking for a bigger production deployment enabling HA, you should consider a distributed deployment using several managers and your own DB clusters to ensure robustness, security and availability. Agents should communicate with the EA Manager via a Load Balancer to enable the use of more than one EA Manager.

  • The endpoints connect to the manager via /etc/hosts change

  • The endpoints connect to the manager with a known FQDN

  • The endpoints connect to the manager using a public IP but no FQDN

  • What data do i want to collect? Take a look at the out-of-the-box packs available here.

Endpoint Agent provides several YAML templates to base your inventory file in, depending on the case that you are trying to deploy. An Ansible YAML inventory file is composed of three main sections:

  • all → vars : This section defines all the variables that are going to be used in the tasks during the playbook run.

  • all → hosts: This section defines the information related to the hosts where the solution is going to be deployed. It can include one or more hosts, and it is composed of blocks named after the hostname of the destination host.

  • all → children: This section defines the topology that is going to be deployed. You can define hosts groups with a specific name and include there the hosts defined in the all->hosts section . The playbook will then deploy specific roles to those groups when its run. The pre-defined hosts groups are the following:

    • (Mandatory) devoeamanagerserverone: First EA Manager server

    • (Optional) devoeamanagerreplicas: Secondary (or more) manager servers. This role needs to include a variable deam_am_i_replica: true in this section.

    • (Optional) deamintsrvs: EA Manager required ancillary services that should be enabled only in one host. If this role is included, creates a docker for MySQL and Redis . If you will be using your own databases, this host group should not be included in the inventory.

    • (Mandatory) deaagentpackager: EA repository that can be enabled in one or more hosts. In each host included in this group, playbook packages osquery installers with certificates and secrets and generates the auto-installation scripts for the agents that will be deployed in the endpoints. It also installs and configures a NGINX service to easily download said packages.

    • (Mandatory) selfsignedcertificates / providedcertificates: Only one of these hosts groups can be enabled at the same time and are only required in devoeamanagerserverone:

      • selfsignedcertificates: Instructs the EA deployer to generate self-signed certificates that will be used to secure the communication between the endpoints and the manager. By default, the self-signed certificates created by the EA deployer will only trust the FQDN defined in deam_fqdn. In the case you need your certificate to trust different names, you can use the variable SAN_extensions and define a list of Subject Alternative Names (IPs and DNSs) to be included in the certificate.

      • providedcertificates: Instruct the EA deployer to use existing certificates instead of creating new ones. You should use this group if you want to use your own certificates for EA Manager.

Standalone deployment template

The below snippet is a valid inventory file for a standalone deployment.

all: vars: deam_fqdnname: << FQDN >> dea_ap_builder_update_etc_host: << yes/no >> dea_ap_overwrite_deam_fqdnname: << PUBLIC_HOST_IP >> set_deam_fqdnname_as_hostname: false deam_admin_email: << ADMIN_EMAIL >> deam_admin_username: << ADMIN_USER >> deam_admin_passwd: << ADMIN_PASS >> deam_client_org_name: << ORGANIZATION_NAME >> dea_ap_repo_user: << REPO_USER >> dea_ap_repo_passwd: << REPO_PASS >> deam_redis_address: << REDIS_ADDRESS:PORT >> deam_redis_password: << REDIS_PASSWORD >> deam_mysql_address: << MYSQL_ADDRESS:PORT >> deam_mysql_database: << MYSQL_DATABASE_NAME >> deam_mysql_username: << MYSQL_USER >> deam_mysql_password: << MYSQL_PASSWORD >> deam_relay_entrypoint: << ENTRYPOINT_URL >> pre_checks_tls_certs_query_bearer: << API v2 Bearer Token >> pre_checks_tls_certs_query_api_url: << Query API entrypoint>> deam_packs_enabled: - configuration.yaml - events.yaml - fetchfiles.yaml - performance.yaml - status.yaml hosts: << HOST_NAME >>: ansible_host: << HOST_IP >> ansible_user: << HOST_USER >> ansible_ssh_pass: << HOST_PASS >> # Only required if you connect with password ansible_ssh_common_args: '-o StrictHostKeyChecking=no' ansible_python_interpreter: << PYTHON_INTERPRETER >> children: devoeamanagerserverone: hosts: << HOST_NAME >>: deamintsrvs: hosts: << HOST_NAME >>: deaagentpackager: hosts: << HOST_NAME >>: selfsigenedcertificates: # Alternative providedcertificates hosts: << HOST_NAME >>: vars: #If no Subject Alternate Names are required, completely remove the SAN_extensions block. SAN_extensions: - IP: << HOST_PUBLIC_IP >> - DNS: << HOST_FQDN >> - DNS: << HOST_FQDN_2 >>

deam_fqdnname: FQDN of the EA Manager. Agents will use this FQDN to communicate to the EA Manager.

dea_ap_builder_update_etc_host: Indicates if the installation package of the endpoints will update /etc/hosts to communicate with the manager.

  • If the FQDN set in deam_fqdnname is public and the agents can reach it, set it as “no”.

  • If the FQDN set in deam_fqdnname is not reachable by the agents and the communication with the EA Manager is via internet, set it as “no”. Configure the parameter dea_ap_overwrite_deam_fqdnname with the public IP that the agents will use to communicate.

  • If the FQDN set in deam_fqdnname is not reachable by the agents and the communication with the EA Manager is via private network, set it as “yes”. When an agent is deployed, the /etc/hosts will be modified to assign the IP configured in ansible_host to the FQDN value.

dea_ap_overwrite_deam_fqdnname: If the FQDN is not reachable by the agent, but the manager needs to be accessed via the internet, put here the public IP that the agent will use to connect. When using public FQDNs or local connection using /etc/hosts this parameter is not necessary and can be removed completely from the inventory.

set_deam_fqdnname_as_hostname: This parameter can be kept always as false. It is a legacy parameter that will be deprecated in the future.

deam_admin_email: E-mail of the admin user. The email is used to log in to the UI interface of the EA Manager.

deam_admin_username: Username for the administrator user.

deam_admin_passwd: Password used to log in to the UI interface of the EA Manager.

deam_client_org_name: Organization Name. It is displayed in the UI.

dea_ap_repo_user: User for the agent download repository.

dea_ap_repo_passwd: Password for the agent download repository.

deam_redis_address: REDIS address and port:

  • If the deployment is self creating dockers for REDIS and MySQL, the value should be configured as localhost:6379.

  • If the deployment is using your own DB services, the value should be configured with the address of the REDIS database and the port of access.

deam_redis_password: REDIS password (when in use).

deam_mysql_address: MySQL address and port:

  • If the deployment is self creating dockers for REDIS and MySQL, the value should be configured as localhost:3306.

  • If the deployment is using your own DB services, the value should be configured with the address of the MySQL database and the port of access.

deam_mysql_database: Name of the database to be used by EA Manager.

deam_mysql_username: Username to access the database.

deam_mysql_password: Password to access the database.

deam_relay_entrypoint: Includes the entry point value of the Devo collector for your deployment:

  • If your domain is deployed in the US Cloud (us.devo.com): tcp://us.elb.relay.logtrust.net:443.

  • If your domain is deployed in the EU Cloud (eu.devo.com): tcp://eu.elb.relay.logtrust.net:443.

  • If your domain is deployed in a different cloud, contact your Devo representative to know your entry point.

pre_checks_tls_certs_query_bearer: (Optional) Obtain a valid Query API v2 token that grant access to siem.logtrust.collector.counter to enable EA deployer to be able to validate data sent during deployment. How to obtain an API Token.

pre_checks_tls_certs_query_api_url: (Optional) Configure the query API endpoint corresponding to your domain.

deam_packs_enabled: List of YAML files including the definition of the packs that will be deployed into the EA Manager.

hosts: The hosts section of the inventory is composed by a group of hosts named with the hostname of the host. In the case of standalone deployment it will include only one. Replace << HOST_NAME >> with the hostname of the server where the installation is happening. Ansible will connect via SSH to every host where it needs to take action using the data configured in this section.

ansible_host: IP of the EA Manager server that ansible is going to use to connect to the host.

ansible_user: SSH User to connect to this host.

ansible_ssh_pass: Password for the SSH user. It is not mandatory to use a password when using the playbooks. If you want to use SSH keys, just comment out this parameter and add your ssh key at Linux session level or include it in this section using the parameter ansible_ssh_private_key_file .

ansible_python_interpreter: Specify what python interpreter will be used depending on your Linux distribution and version:

  • Python 2: /usr/bin/python

  • Python 3: /usr/bin/python3

Distributed deployment template

The following template is a deployment of Endpoint Agent with two EA Managers using existing databases. This deployment also requires a Load Balancer to be placed between the endpoints and the two EA Managers.

all: vars: deam_fqdnname: << FQDN >> dea_ap_builder_update_etc_host: << yes/no >> set_deam_fqdnname_as_hostname: false deam_admin_email: << ADMIN_EMAIL >> deam_admin_username: << ADMIN_USER >> deam_admin_passwd: << ADMIN_PASS >> deam_client_org_name: << ORGANIZATION_NAME >> dea_ap_repo_user: << REPO_USER >> dea_ap_repo_passwd: << REPO_PASS >> deam_redis_address: << REDIS_ADDRESS:PORT >> deam_redis_password: << REDIS_PASSWORD >> deam_mysql_address: << MYSQL_ADDRESS:PORT >> deam_mysql_database: << MYSQL_DATABASE_NAME >> deam_mysql_username: << MYSQL_USER >> deam_mysql_password: << MYSQL_PASSWORD >> deam_relay_entrypoint: << ENTRYPOINT_URL >> pre_checks_tls_certs_query_bearer: << API v2 Bearer Token >> pre_checks_tls_certs_query_api_url: << Query API entrypoint>> deam_packs_enabled: - configuration.yaml - events.yaml - fetchfiles.yaml - performance.yaml - status.yaml hosts: << HOST_NAME_SERVER_1 >>: ansible_host: << HOST_IP >> ansible_user: << HOST_USER >> ansible_ssh_pass: << HOST_PASS >> # Only required if you connect with password ansible_ssh_common_args: '-o StrictHostKeyChecking=no' ansible_python_interpreter: << PYTHON_INTERPRETER >> << HOST_NAME_SERVER_2 >>: # Set ansible_host your public ip used to conncet from ansbile and devo-ea agents ansible_host: << HOST_IP >> ansible_user: << HOST_USER >> ansible_ssh_pass: << HOST_PASS >> # Only required if you connect with password ansible_ssh_common_args: '-o StrictHostKeyChecking=no' # python3 required for Ubuntu 18 ansible-playbooks ansible_python_interpreter: << PYTHON_INTERPRETER >> children: devoeamanagerserverone: hosts: << HOST_NAME_SERVER_1 >>: devoeamanagerreplicas: hosts: << HOST_NAME_SERVER_2 >>: vars: deam_am_i_replica: true deamintsrvs: hosts: << HOST_NAME_SERVER_1 >>: deaagentpackager: hosts: << HOST_NAME_SERVER_1 >>: << HOST_NAME_SERVER_2 >>: selfsigenedcertificates: # Alternative providedcertificates hosts: << HOST_NAME_SERVER_1 >>:

Deployment credentials

The Devo EA Deployer uses SSH to connect to the different servers where it needs to install different components. When configuring the hosts in the inventory file, it is required to configure the SSH user and authentication method that the deployer needs to use in each task.

Using SSH user and password

For a user and password-based authentication method, configuration for the host should look like the one in the following snippet:

all: ... hosts:     devo-ea-manager:       ansible_host: 192.168.57.5       ansible_user: ssh_user       ansible_ssh_pass: ssh_pass       ansible_ssh_common_args: '-o StrictHostKeyChecking=no' ...

Using SSH private key

Alternatively, in order to use a private key, use the following code:

Adding the path to the private key to the inventory file is not mandatory. It can be added to the environment using ssh-add 'key file' instead of adding it to the inventory file. Note that you should make sure that the ssh-agent is running to be able to add your identity. You can start the agent with eval $(ssh-agent).

Initial packs configuration (new from EA v1.1.0)

From v1.1.0 onwards, the EA deployer enables customers to decide which query packs will be deployed with the solution, as opposed to all of them being added automatically during the installation process. As a result, it will be up to the installer responsible to decide which packs will be added to the baseline configuration to fulfill the customer’s particular needs and use cases. The complete list of available packs, along with a description of their functionality and set of supported use cases, can be found here in the pre-integrated query packs section.

The process to enable packs is handled with two parameters in the inventory file:

  • deam_packs_enabled: Sets what packs will be deployed when running both devo-endpoint-agent and deam-packs playbooks.

In the following example, only the configuration (represented by configuration.yaml) and events (events.yaml) packs will be deployed:

The file names included in this list come from the files placed in $HOME/devo-ea-deployer/playbooks/roles/deam-packs/files/optional-devo-packs.

(Only From 1.2.1) The packs (and queries included in the pack) not included in the deam_packs_enabled will be automatically removed from the actual EA Manager if they were previously enabled.

  • (Deprecated in V1.2.1) deam_packs_purge_cached_files: Use this option when you do not want to deploy a pack anymore. Once the packs are deployed for the first time, they are cached under (/var/devo-ea-manager/packs/devo-packs). If you want to avoid some of the packages being redeployed, make sure to include them in the purge list before triggering the deploy operation. Otherwise, the deployer will use the cached version and deploy again the packs even if they are not included in deam_packs_enabled.

In the following example, the installation user is deploying only the configuration.yaml pack and is purging the events.yaml pack that was previously deployed from the temporary folder:

Python Interpreter

When deploying in CentOS 7 or Red Hat 7 OS, make sure that the host is configured to use python 2 instead of python 3 in the inventory file.

Hosts in Centos7 or RedHat 7:

Other OS:

Firewalld and ufw management

(Only from 1.2.1)

By default, the EA deployer will take care of the configuration of ufw and firewalld to create the required exceptions in the default Linux firewalls if firewall services are up and running at deployment time.

When using firewalld (in Red Hat based linux) the firewall zones that EA Deployer will use are configurable in the inventory using the following parameters:

deam_fleet_firewall_zone. The zone name where the Devo EA manager listening port (by default, port 8080) will be attached, in the event that firewalld service is up and running. No network interface is added to this zone. Default value public.

deam_intsrv_firewall_zone. The zone name where the new docker network interface (For MySQL and REDIS) will be attached, in the event that firewalld service is up and running. The network interface that docker creates is added to this zone. Default value trusted.

dea_ap_firewall_zone. The zone name where the Devo EA packager repository listening port (by default, port 8081) will be attached, in the event that firewalld service is up and running. No network interface is added to this zone. Default value public.

If you do not want the EA deployer to make any configuration in the firewall, you can add the following variable in the inventory file:

When the variable firewall_configuration_disabled is true, no action is taken in the firewall configuration. You need to do firewall configuration manually to allow the service to operate correctly.

Endpoint Agent <> EA Manager communication

Using FQDN

In case the system administrator does not want to rely on explicitly setting the IP address of the EAM servers in the configuration, hence easing the introduction of changes in the future, or if the EA Manager is going to be available on an open Internet environment, it is recommended to set the deployer to use an FQDN (Fully Qualified Domain Name) for endpoints <> manager connection. To do so, add a code section such as in the following example to the inventory file -only add the fields that are not already included:

  • deam_fqdnname: FQDN of the Devo EA Manager or the Load Balancer FQDN in the case of multiple EA Managers. Endpoints will use this FQDN to connect to the Devo EA Manager.

  • set_deam_fqdnname_as_hostname: Set to false to avoid changing the EA Manager hostname to the value set in the above field.

  • dea_ap_builder_update_etc_host: Set to "no" to instruct the endpoint installers to not modify the endpoint hosts file.

Using hosts File

A simpler approach to enable the communication between agents and the manager is through the modification of the agents' /etc/hosts file.

The deployment script is pre-defined to enable this mechanism by default, so there is no specific configuration change needed to use it.

EA Manager <> Devo communication

Direct connectivity

This is the default scenario, where the EA Manager has direct connectivity to Devo. In this case, there is no need for any specific configuration as the installer should work as is.

Using a Devo Relay

In some cases, it is necessary to centralize all ingested traffic for a particular deployment, such as to enforce an extra level of security or isolation of the EA Manager.

Assuming that the in-house Relay’s IP is 192.168.43.147, we just need to configure deam_relay_entrypoint: tcp://192.168.43.147:13000. We chose port number 13000 port because the Relay service is configured to listen to that port by default.
Example snippet of inventory file based on that configuration:

Other special deployment cases

High availability configuration

Some deployment scenarios will require the deployment of more than one EA Manager to handle all traffic generated from the endpoints and to ensure high availability of the data processing and forwarding is observed. In this case, a specific configuration during the installation process is required to instruct the Devo EA Deployer to instantiate those roles.

*NOTE: Devo EA Deployer will take care of deploying several EA Managers, but in order to provide a single entry point to the endpoints, it is required that you set up a Load Balancer separately.

Databases installation

The EAM relies on MySQL and Redis for data persistence. There are several scenarios to consider depending on the type of installation required for a particular customer.

The next section summarizes the most common alternatives for the deployment of the databases, as well as the procedure to implement them:

Devo EA Manager users and passwords

After a successful deployment of Devo EA Manager, it will create two services that require authentication. User and password can be configured to include your own passwords. If these parameters are not included in the inventory file, default values are used.

From EA 1.3 onwards, login the platform is done using deam_admin_email instead of deam_admin_username

  • Devo EA Manager Web Interface (port 8080): To control the user and password for this service, the following parameters can be used in the inventory file:

  • Devo Endpoint Agent download (port 8081): To control the user and password for this service, the following parameters can be used in the inventory file:

Certificates

The Devo EA Deployer is configured to generate self-signed certificates and configure them by default but you can configure EA Deployer to use your own certificates.

It is recommended to use ECDSA certificates instead of RSA as TLS decryption in the EA Manager side is smoother and increases the performance significantly. The EA Deployer will generate ECDSA certificates by default when using self-generated certificates.

Appendix A - Example inventory files

Appendix B - Security Recommendations

EA Manager deployment will not modify any configuration present on the system as it is the responsibility of a system administrator to have the system hardened following their company’s policies.

However, since the EA Manager will be an element present there are several recommended good practices that can be followed at your discretion:

  • Ensure following directives are included in sshd_config:

  • Restrict access to connections coming from certain ips using certain users, even including password authentication, but only from trusted origins, for example:

  • Create unique users associated to a certain person and avoid usage of generic users with shared keys.