{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"maxLevel":{"value":"2"},"_parentId":{"value":"120918224"},"type":{"value":"flat"}},"macroMetadata":{"macroId":{"value":"66c6891d-354d-46c9-af67-46cf29784578"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"6ff782dc-b50e-46b6-9202-4212f0a4c4af"}},{"type":"paragraph","content":[{"text":"This document is geared toward people in charge of Devo Endpoint Agent deployment or administration and includes ways of troubleshooting Devo EA, as well as information about common errors cases that have been identified in the field.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Deployment","type":"text"}]},{"type":"paragraph","content":[{"text":"This section describes typical trouble scenarios and troubleshooting guidelines for deployment scenarios.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Controlled error messages during deployment","type":"text"}]},{"type":"paragraph","content":[{"text":"There are some tasks that execute commands or operations that can end with some kind of error. These tasks are designed to check some services or configurations and apply customized changes in some cases or ignore those configurations in others. ","type":"text"},{"text":"…ignoring","type":"text","marks":[{"type":"code"}]},{"text":" text will be displayed just after displaying error message and can help us to identify these kinds of controlled error tasks. ","type":"text"}]},{"type":"paragraph","content":[{"text":"For example: Task that is checking if the ","type":"text"},{"text":"firewalld","type":"text","marks":[{"type":"code"}]},{"text":" service is running:","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"inline-media-image","parameters":{"width":"600","id":"87146398-0d40-408d-803a-20541412413a","collection":"contentId-120918224","height":""}}},{"type":"hardBreak"},{"text":"This error will pop up when the ","type":"text"},{"text":"firewalld","type":"text","marks":[{"type":"code"}]},{"text":" service is not present or it is stopped. The error can be safely ignored and does not affect the deployment. As a rule, any message that is tagged as ","type":"text"},{"text":"...ignoring","type":"text","marks":[{"type":"code"}]},{"text":" can be safely ignored and has no effect in the deployment sequence.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Timeout when waiting for 127.0.0.1:8080","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":45.0},"content":[{"type":"media","attrs":{"__fileSize":61847,"__fileMimeType":"image/png","width":1240,"id":"b973f1dc-5cdc-4564-99a3-653d3490bd3b","collection":"contentId-120918224","type":"file","__fileName":"trouble_20.png","height":84}}]},{"type":"paragraph","content":[{"text":"The deployment process cannot connect with the interface that Fleet starts in the port 8080 or the connection took longer than 60 seconds. Typically, there has been a problem in the deployment sequence and the fleet instance could not boot up. Check the Endpoint Agent Manager section and see if the logs give any hint as to what they problem may be.","type":"text"}]},{"type":"paragraph","content":[{"text":"The most common root cause for this error message is that the certificates used to send data from the EA Manager to Devo have not been properly configured, make sure that all the steps in the guide have been followed properly. In a default installation process, the ","type":"text"},{"text":"domain-certs","type":"text","marks":[{"type":"code"}]},{"text":" folder should look like the following screenshot:","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"inline-media-image","parameters":{"width":"600","id":"0b95edbd-02dd-4c80-87f5-af99d2c7ccfb","collection":"contentId-120918224","height":""}}},{"type":"hardBreak"},{"text":"Make sure that the name of the certificates are using the same names as in the screenshot above.","type":"text"}]},{"type":"paragraph","content":[{"text":"Also, make sure that the collector configured in the entrypoint is correct as explained in the deployment guide.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Proxy is needed to access Internet","type":"text"}]},{"type":"paragraph","content":[{"text":"Set up the following settings:","type":"text"}]},{"type":"paragraph","content":[{"text":"Enable and set up correctly ","type":"text"},{"text":"http_proxy","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"https_proxy","type":"text","marks":[{"type":"code"}]},{"text":" environment variables.","type":"text"}]},{"type":"paragraph","content":[{"text":"Enable proxy for docker environment as described ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.docker.com/network/proxy/"}}]},{"text":", by editing file ","type":"text"},{"text":"~/.docker/config.json.","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Shared connection closed","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see this error during the deployment process (it is likely to happen at the beginning of the deployment process and do not let the process continue if so):","type":"text"}]},{"type":"codeBlock","content":[{"text":"TASK [duam-internal-services : Set hostname with name in inventory] ************\nfatal: [devo-ua-manager]: FAILED! => {\"changed\": false, \"module_stderr\": \"Shared connection to 10.239.74.38 closed.\\r\\n\", \"module_stdout\": \"\", \"msg\": \"MODULE FAILURE\\nSee stdout/stderr for the exact error\", \"rc\": 1}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"The reason for this issue is that Ansible is not able to make use of SSH to properly perform the deployment. Ansible will connect via SSH to every server included in the inventory to perform the installation. The solution is to fix the environment so Ansible can make proper use of SSH.","type":"text"}]},{"type":"paragraph","content":[{"text":"If the deployment is being done in a single server the following workaround has been proved to work:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit your inventory file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add ","type":"text"},{"text":"ansible_connection: local","type":"text","marks":[{"type":"code"}]},{"text":" in the all->vars section.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Save your inventory file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run the playbook again.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Be aware that this solution will tell Ansible to not use SSH so this is only valid for local deployments.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Docker-compose: error while loading shared libraries: libz.so.1: failed to map segment from shared object","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see this error during the deployment process (only in the case that you have configured ","type":"text"},{"text":"dockerized","type":"text","marks":[{"type":"em"}]},{"text":" versions of MySQL and Redis in your inventory file):","type":"text"}]},{"type":"codeBlock","content":[{"text":"TASK [duam-internal-services : Deploy and configure int-srv for DEA manager (RedHat / CentOS)] ************\nfatal: [devo-ua-manager]: FAILED! => {\"changed\": true, \"cmd\": \"cd /srv/duam-internal-services\\n/usr/local/bin/docker-compose up -d\\n\", \"delta\": \"0:00:00.141889\", \"end\": \"2021-07-13 16:11:48.867585\", \"msg\": \"non-zero return code\", \"rc\": 127, \"start\": \"2021-07-13 16:11:48.725696\", \"stderr\": \"/usr/local/bin/docker-compose: error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted\", \"stderr_lines\": [\"/usr/local/bin/docker-compose: error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted\"], \"stdout\": \"\", \"stdout_lines\": []}\n","type":"text"}]},{"type":"paragraph","content":[{"text":"The reason for this issue usually is a problem with ","type":"text"},{"text":"/tmp","type":"text","marks":[{"type":"code"}]},{"text":" permissions. Permissions are managed directly by file-system or by mount system. If ","type":"text"},{"text":"/tmp","type":"text","marks":[{"type":"code"}]},{"text":" is a mounted path with ","type":"text"},{"text":"noexec","type":"text","marks":[{"type":"code"}]},{"text":" option set, then we will see this error even though filesystem permissions are properly granted (read, write and exec: ","type":"text"},{"text":"777","type":"text","marks":[{"type":"code"}]},{"text":").","type":"text"}]},{"type":"paragraph","content":[{"text":"Next command helps to check ","type":"text"},{"text":"noexec","type":"text","marks":[{"type":"code"}]},{"text":" mount option:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ mount | grep \"/tmp\" | fgrep noexec","type":"text"}]},{"type":"paragraph","content":[{"text":"Empty output is expected if ","type":"text"},{"text":"noexec","type":"text","marks":[{"type":"code"}]},{"text":" option is not set or ","type":"text"},{"text":"/tmp","type":"text","marks":[{"type":"code"}]},{"text":" is not a mounted path. In another case, we will see output data similar to:","type":"text"}]},{"type":"codeBlock","content":[{"text":"tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec, ...","type":"text"}]},{"type":"paragraph","content":[{"text":"To temporally fix the ","type":"text"},{"text":"noexec","type":"text","marks":[{"type":"code"}]},{"text":" permission issue, remount ","type":"text"},{"text":"/tmp","type":"text","marks":[{"type":"code"}]},{"text":" with ","type":"text"},{"text":"exec","type":"text","marks":[{"type":"code"}]},{"text":" option:","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo mount /tmp -o remount,exec","type":"text"}]},{"type":"paragraph","content":[{"text":"Then run ","type":"text"},{"text":"ansible-playbook ...","type":"text","marks":[{"type":"code"}]},{"text":" command again","type":"text"}]},{"type":"paragraph","content":[{"text":"Be aware that this solution is temporal in the sense that exec permission will maintain until VM will be restarted, you should persist this change following appropriate procedure (editing ","type":"text"},{"text":"/etc/fstab","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"tmp.mount","type":"text","marks":[{"type":"code"}]},{"text":" service, ...) for your Linux distribution.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Unable to find any of pip3/pip to use. pip needs to be installed (can appear with both pip or pip3)","type":"text"}]},{"type":"paragraph","content":[{"text":"If once requirements are installed executing:","type":"text"}]},{"type":"paragraph","content":[{"text":"ansible-galaxy install -r requirements.txt","type":"text","marks":[{"type":"code"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"From now on it is assumed that the issue is related with pip3, but the procedure works in the same way with pip.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"... you still get the following error launching devo-endpoint-agent playbook...","type":"text"}]},{"type":"codeBlock","content":[{"text":"ASK [geerlingguy.pip : Ensure pip_install_packages are installed.] ***************************************************************************\nfailed: [devo-ea-manager] (item={'name': 'pip', 'state': 'latest'}) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": {\"name\": \"pip\", \"state\": \"latest\"}, \"msg\": \"Unable to find any of pip3 to use. pip needs to be installed.\"}\nfailed: [devo-ea-manager] (item={'name': 'docker'}) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": {\"name\": \"docker\"}, \"msg\": \"Unable to find any of pip3 to use. pip needs to be installed.\"}\nfailed: [devo-ea-manager] (item={'name': 'docker-compose'}) => {\"ansible_loop_var\": \"item\", \"changed\": false, \"item\": {\"name\": \"docker-compose\"}, \"msg\": \"Unable to find any of pip3 to use. pip needs to be installed.\"}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Since Ansible will use ","type":"text"},{"text":"sudo","type":"text","marks":[{"type":"code"}]},{"text":" to execute the tasks, you should check if pip3 is in root user ","type":"text"},{"text":"$PATH","type":"text","marks":[{"type":"code"}]},{"text":" using which:","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This is an example, paths may differ depending on the host.","type":"text"},{"type":"hardBreak"}]}]},{"type":"codeBlock","content":[{"text":"user@server:~/ sudo -s\nroot@server:/# which pip3\n/usr/bin/pip3\nroot@server:/# echo $PATH\n/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/bin","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Since ","type":"text"},{"text":"/usr/bin","type":"text","marks":[{"type":"code"}]},{"text":" is not included in ","type":"text"},{"text":"$PATH","type":"text","marks":[{"type":"code"}]},{"text":" it must be added:","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@server:/# export PATH=$PATH:$(which pip3 | sed 's/\\/pip3//')\nroot@server:/# echo $PATH\n/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/bin:/usr/bin","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"And then launch the playbook again:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ansible-galaxy -i ... devo-endpoint-agent.yaml","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If error persists, it may be due to sudoers secure_path configuration, which limits value of root user environment variable ","type":"text"},{"text":"$PATH","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"To check it use the following:","type":"text"}]},{"type":"codeBlock","content":[{"text":"user@server:~/ sudo cat /etc/sudoers | grep secure_path | grep $(which pip3 | sed 's/\\/pip3//')","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If nothing is returned it must be added to the sudoers file.","type":"text"}]},{"type":"paragraph","content":[{"text":"First of all make a copy of the sudoers file as backup:","type":"text"}]},{"type":"codeBlock","content":[{"text":"user@server:~/ sudo cp /etc/sudoers /etc/sudoers_bck","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Modify the original and check if it is ok:","type":"text"}]},{"type":"codeBlock","content":[{"text":"user@server:~/ sudo which pip3\n/usr/bin/pip3\nuser@server:~/ sudo sed -i '/.*secure_path.*/ s/\"$/:\\/usr\\/bin\"/' /etc/sudoers\nuser@server:~/ sudo cat /etc/sudoers | grep secure_path | grep $(which pip3 | sed 's/\\/pip3//')\nDefaults secure_path=\"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"And then launch the playbook again:","type":"text"}]},{"type":"codeBlock","content":[{"text":"ansible-galaxy -i ... devo-endpoint-agent.yaml","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Unsupported nginxinc dependency roles","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see one or both errors similar to following ones:","type":"text"}]},{"type":"codeBlock","content":[{"text":"TASK [nginxinc.nginx : (Amazon Linux/CentOS/RHEL) Configure NGINX repository] *************************************************************************************************************************************\nfatal: [devo-ea-manager]: FAILED! => {\"changed\": false, \"msg\": \"Unsupported parameters for (yum_repository) module: module_hotfixes Supported parameters include: async, attributes, backup, bandwidth, baseurl, content, cost, delimiter, deltarpm_metadata_percentage, deltarpm_percentage, description, directory_mode, enabled, enablegroups, exclude, failovermethod, file, follow, force, gpgcakey, gpgcheck, gpgkey, group, http_caching, include, includepkgs, ip_resolve, keepalive, keepcache, metadata_expire, metadata_expire_filter, metalink, mirrorlist, mirrorlist_expire, mode, name, owner, params, password, priority, protect, proxy, proxy_password, proxy_username, regexp, remote_src, repo_gpgcheck, reposdir, retries, s3_enabled, selevel, serole, setype, seuser, skip_if_unavailable, src, ssl_check_cert_permissions, sslcacert, sslclientcert, sslclientkey, sslverify, state, throttle, timeout, ui_repoid_vars, unsafe_writes, username\"}","type":"text"}]},{"type":"paragraph","content":[{"text":"or","type":"text"}]},{"type":"codeBlock","content":[{"text":"TASK [nginxinc.nginx_config : Ensure NGINX HTTP directory exists] *************************************************************************************************************************************************\nfatal: [devo-ea-manager]: FAILED! => {\"msg\": \"Invalid data passed to 'loop', it requires a list, got this instead: {'agents': {'template_file': 'http/default.conf.j2', 'conf_file_name': 'agents.conf', 'conf_file_location': '/etc/nginx/conf.d/', 'servers': {'agents': {'listen': {'listen_localhost': {'ip': '0.0.0.0', 'port': 8081, 'ssl': True}}, 'server_name': 'devo-ea-manager', 'http_error_pages': {}, 'error_page': '/var/devo-ea-manager/agents/www', 'access_log': [{'name': 'main', 'location': '/var/log/nginx/agents-access.log'}], 'error_log': {'location': '/var/log/nginx/agents-error.log', 'level': 'warn'}, 'root': '/var/devo-ea-manager/agents/www', 'autoindex': True, 'auth_basic': 'Devo Endpoint agent repo', 'auth_basic_user_file': '/etc/nginx/agents_password', 'client_max_body_size': '100m', 'ssl': {'cert': '/etc/nginx/certs/dea-agents-repo.crt', 'key': '/etc/nginx/certs/dea-agents-repo.key'}}}}}. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.\"}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Usually these errors are related to the version of ","type":"text"},{"text":"nginxinc","type":"text","marks":[{"type":"code"}]},{"text":" dependency roles. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Implement the following steps to fix it:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit ","type":"text"},{"text":"playbooks/roles/requirements.txt","type":"text","marks":[{"type":"code"}]},{"text":" file (with vi for example), and set fixed version for ","type":"text"},{"text":"nginxinc.nginx","type":"text","marks":[{"type":"code"}]},{"text":" and/or ","type":"text"},{"text":"nginxinc.nginx_config","type":"text","marks":[{"type":"code"}]},{"text":" roles. ","type":"text"},{"text":"nginxinc.nginx","type":"text","marks":[{"type":"code"}]},{"text":" version must be 0.21.0 and ","type":"text"},{"text":"nginxinc.nginx_config","type":"text","marks":[{"type":"code"}]},{"text":" version must be 0.3.3:","type":"text"},{"type":"hardBreak"}]}]}]},{"type":"codeBlock","content":[{"text":"- src: nginxinc.nginx\n version: 0.21.0\n- src: nginxinc.nginx_config\n version: 0.3.3","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Uninstall current versions of these roles using ","type":"text"},{"text":"ansible-galaxy","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"ansible-galaxy remove nginxinc.nginx\nansible-galaxy remove nginxinc.nginx_config","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Reinstall roles dependencies using ","type":"text"},{"text":"ansible-galaxy","type":"text","marks":[{"type":"code"}]},{"text":" with ","type":"text"},{"text":"-f","type":"text","marks":[{"type":"code"}]},{"text":" option:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"ansible-galaxy install -r playbooks/roles/requirements.yaml -f","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Continue from step that calls ","type":"text"},{"text":"ansible-playbook","type":"text","marks":[{"type":"code"}]},{"text":" command with the deployment/configuration process.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Cannot start NGINX (address already in use)","type":"text"}]},{"type":"paragraph","content":[{"text":"If the ansible playbook fails because NGINX cannot start, and with the following errors:","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":" ","type":"text"},{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"inline-media-image","parameters":{"width":"600","id":"ad7d611e-4019-491c-8ff3-e05532e205a6","collection":"contentId-120918224","height":""}}}]},{"type":"paragraph","content":[{"text":"These errors are caused because there is another service running in the port 80. The EA Manager is intended to run by itself and to not share resources with other products. While NGINX will only listen in the port 8081 (by default), during the installation process it will use the port 80 until the port 8081 is configured.","type":"text"}]},{"type":"paragraph","content":[{"text":"To solve the issue, stop the service from listening in the port 80 and run the playbook again.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"NGINX deployment management","type":"text"}]},{"type":"paragraph","content":[{"text":"If you are experiencing issues when deploying NGINX using Ansible, EA Deployer gives you the option to ignore the installation and do it manually. NGINX works decoupled from the EA Manager's normal service, and it is only used to provide a repository where you will be able to download the generated agents. This is only available from version 1.2.1 on.","type":"text"}]},{"type":"paragraph","content":[{"text":"Include one of the following variables in your inventory to disable NGINX deployment. Using this variable requires the user to deploy manually the NGINX server:","type":"text"}]},{"type":"codeBlock","content":[{"text":"all:\n vars:\n dea_ap_deploy_nginx_software_base: false","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"dea_ap_deploy_nginx_software_base","type":"text","marks":[{"type":"code"}]},{"text":": completely disables the NGINX software deployment. However, NGINX ","type":"text"},{"text":"http","type":"text","marks":[{"type":"code"}]},{"text":" server configuration (ansible role: ","type":"text"},{"text":"nginxinc.nginx_config","type":"text","marks":[{"type":"code"}]},{"text":") will still run and configure your service appropriately.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Deployment pre-check of https://pkg.osquery.io/ URL fails","type":"text"}]},{"type":"paragraph","content":[{"text":"(Only in 1.2.1)","type":"text","marks":[{"type":"strong"}]},{"text":" If you see the following error during execution of ","type":"text"},{"text":"pre-checks: Checking url access with curl ","type":"text","marks":[{"type":"code"}]},{"text":"ansible task when ","type":"text"},{"text":"devo-ea-deployer.yaml","type":"text","marks":[{"type":"code"}]},{"text":" playbook is executing:","type":"text"},{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"TASK [pre-checks : Checking url access with curl] *****************************************************************************************************************************************************************\nfailed: [devo-ea-manager] (item={'url': 'https://pkg.osquery.io/', 'options': '-I', 'expected_pattern': 'HTTP/.*200'}) => {\"ansible_loop_var\": \"item\", \"changed\": true, \"cmd\": \"echo -n 'https://pkg.osquery.io/ ..\n.' >> /tmp/dead/pre-checks/checks.log\\ncurl -sS -o /tmp/dead/pre-checks/out.std --max-time 15 -I 'https://pkg.osquery.io/'\\nrc=$?\\nif [ \\\"${rc}\\\" != 0 ]\\nthen\\n echo \\\"curl ERROR\\\" >> /tmp/dead/pre-checks/check\ns.log\\n exit $rc\\nfi\\nif [ \\\"HTTP/.*200\\\" != \\\"\\\" ]\\nthen\\n if grep -E 'HTTP/.*200' \\\"/tmp/dead/pre-checks/out.std\\\" > /dev/null\\n then\\n echo \\\"OK\\\"\\n echo \\\"OK\\\" >> /tmp/dead/pre-checks/checks.log\\n\nelse\\n echo 'Pattern HTTP/.*200 not found when load https://pkg.osquery.io/'\\n echo 'Pattern HTTP/.*200 not found' >> /tmp/dead/pre-checks/checks.log\\n exit 1\\n fi\\nfi\\n\", \"delta\": \"0:00:00.293876\", \"e\nnd\": \"2022-02-21 14:17:52.835791\", \"item\": {\"expected_pattern\": \"HTTP/.*200\", \"options\": \"-I\", \"url\": \"https://pkg.osquery.io/\"}, \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2022-02-21 14:17:52.541915\", \"s\ntderr\": \"\", \"stderr_lines\": [], \"stdout\": \"Pattern HTTP/.*200 not found when load https://pkg.osquery.io/\", \"stdout_lines\": [\"Pattern HTTP/.*200 not found when load https://pkg.osquery.io/\"]}\nchanged: [devo-ea-manager] => (item={'url': 'https://nginx.org/', 'options': '-I', 'expected_pattern': 'HTTP/.*200'})\n\nPLAY RECAP ********************************************************************************************************************************************************************************************************\ndevo-ea-manager : ok=32 changed=11 unreachable=0 failed=1 skipped=11 rescued=0 ignored=0","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"And which failing URL checked is ","type":"text"},{"text":"'url': 'https://pkg.osquery.io/'","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]},{"type":"paragraph","content":[{"text":"The reason is that pkg.osquery.io service was migrated and it now does a redirection (HTTP 302) instead of replying with a HTTP 200 as usual. New configuration allows redirections when performing the pre-check.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can update the check parameters with the following command (assuming that the working path is the path where devo-ea-deployer-1.2.1.tgz was extracted):","type":"text"}]},{"type":"codeBlock","content":[{"text":"[ \"$(md5sum playbooks/roles/pre-checks/defaults/main.yml | cut -f 1 -d' ')\" == \"3da20fb33ea5a9258b82488edf8aec36\" ] \\\n && sed -i '71s/$/L/' playbooks/roles/pre-checks/defaults/main.yml \\\n || echo \"ERROR md5 code returned is not matching with the expected value, or error opening file\"","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If no error is returned after running these commands then you will be able to run ","type":"text"},{"text":"devo-ea-manager.yaml","type":"text","marks":[{"type":"code"}]},{"text":" ansible-playbook again.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Deployment pre-check of Devo certificates","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see an error during ","type":"text"},{"text":"pre-checks: Checking Devo certificates with tls connection","type":"text","marks":[{"type":"code"}]},{"text":" ansible step like the following one:","type":"text"}]},{"type":"codeBlock","content":[{"text":"TASK [pre-checks : Checking Devo certificates with tls connection] *************\nfatal: [devo-ea-manager]: FAILED! => {\"changed\": true, \"cmd\": \"echo -n 'us.elb.relay.logtrust.net:443 ...' >> /tmp/dead/pre-checks/tls_certs_checks.log\\n\\ntryNum=1\\nwhile [ \\\"$tryNum\\\" -le \\\"20\\\" ]\\ndo\\n echo -n \\\" $tryNum ...\\\" >> /tmp/dead/pre-checks/tls_certs_checks.log\\n echo \\\"<14>$( date --iso=s ) devo-ea-manager test.drop.devoea: Devo Endpoint Agent:pre-check test\\\" | openssl s_client -connect \\\"us.elb.relay.logtrust.net:443\\\" -key \\\"/etc/devo-ea-manager/devo-certs/domain.key\\\" -cert \\\"/etc/devo-ea-manager/devo-certs/domain.crt\\\" -CAfile \\\"/etc/devo-ea-manager/devo-certs/chain.crt\\\"\\n\\n rc=$?\\n if [ \\\"${rc}\\\" != 0 ]\\n then\\n echo \\\"ERROR\\\" >> /tmp/dead/pre-checks/tls_certs_checks.log\\n exit $rc\\n fi\\n\\n sleep \\\"0.2\\\"\\n tryNum=$(( tryNum + 1 ))\\ndone\\necho \\\"OK\\\" >> /tmp/dead/pre-checks/tls_certs_checks.log\\n\", \"delta\": \"0:00:00.299806\", \"end\": \"2022-04-05 13:30:48.616521\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2022-04-05 13:30:48.316715\", \"stderr\": \"depth=2 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\\nverify return:1\\ndepth=1 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\\nverify return:1\\ndepth=0 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\\nverify return:1\\n140238574314816:error:1413C18E:SSL routines:ssl_add_cert_chain:ca md too weak:../ssl/statem/statem_lib.c:972:\", \"stderr_lines\": [\"depth=2 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\", \"verify return:1\", \"depth=1 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\", \"verify return:1\", \"depth=0 C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\", \"verify return:1\", \"140238574314816:error:1413C18E:SSL routines:ssl_add_cert_chain:ca md too weak:../ssl/statem/statem_lib.c:972:\"], \"stdout\": \"CONNECTED(00000003)\\n---\\nCertificate chain\\n 0 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\\n i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\\n 1 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\\n i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\\n 2 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\\n i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\\n---\\nServer certificate\\n-----BEGIN CERTIFICATE-----\\nMIIEaTCCAlECCQDEnVpaeewKTDANBgkqhkiG9w0BAQsFADBoMQswCQYDVQQGEwJF\\nUzEPMA0GA1UECAwGTWFkcmlkMQ8wDQYDVQQHDAZNYWRyaWQxETAPBgNVBAoMCExv\\nZ1RydXN0MRIwEAYDVQQLDAlMb2cgcmVsYXkxEDAOBgNVBAMMB3JlbGF5Q0EwHhcN\\nMTkwODMwMDg0NzQ2WhcNMjkwODI3MDg0NzQ2WjCBhDELMAkGA1UEBhMCRVMxDzAN\\nBgNVBAgMBk1hZHJpZDEPMA0GA1UEBwwGTWFkcmlkMREwDwYDVQQKDAhMb2dUcnVz\\ndDEcMBoGA1UECwwTTG9nIHJlbGF5IGF0IEFtYXpvbjEiMCAGA1UEAwwZdXMuZWxi\\nLnJlbGF5LmxvZ3RydXN0Lm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\\nggEBAOFckGfv5RvMB7/YIDT9XaALJ7mBmzP4vHY29SaQyVw2eDEULO60GzP2RYB1\\n4zmwYb/YVFlpXCwoiHJgFG7wTutZ6Vpg+qhwvIlO5woiUvXXImK6oiRTgaY4UAF1\\nK/m0gXqtyL21LobLAX8YLaQV4wK2SOBVykF4EEIVJG0nh+Xn1U0thGj9P22jcUsD\\ndQRkMFP+XpdIGOS8SLPXLKy93IcETBc5LTUDGjEoYcUnAScHyY5ZgS39FVYcrDea\\nYzcxVEIbUmL9UnovXV2UhI+i2yyNDdKFtUfVNyVXEVAfoack+hUu2uackCMeNtO7\\n+rkxNb6lo5fk4p7dFkSCbzyZyNECAwEAATANBgkqhkiG9w0BAQsFAAOCAgEAJX3J\\ngmxwASZcJ0UbSGc/T4mu7J11Epe9cO+YRa2kj/Ai2U03zH/PLc98NAjuLCUFDaKv\\n9+gsViXwOVur5f/Bpzem43Wpgs2Fq/dyTk34lWulJkJ4xRQ9gpgZXrt9GdiP645j\\nts6xRe7S9apNmDhzdmzNY3wSUtSA4DxhQ6Q0TAr+gV8T1sBThIWlkvh8IBuURkW8\\n42FZvA+5n/EW/1nUGDSO4D1fidux2AmoEpoBbX/YG+e6LMeW/csnHZi5/rcT8HSm\\ne9qCNlmXF8tRHMpxyKcmO/FJdiBwXfwWF9A1zeDyKof3u0xw5KAAeeq5hHYe5uZv\\nRRhWKh6IQ3L8+uVUCQ1ESAihDYhq7u1tznsPtkw+NMhBbBtcTLccyUnhMg7C1Ug/\\nXgjtQk6L7CO+0NQCVmLyeEGNKQVb529ZJlEWuDZbR+f9l3ggcGw+KC1M38dmovHY\\nocaaU/5zfUI+oiu/KxOI3W5M6g7G1oTgzyvDg/qsp2J5en31oLjyl2r5t6unawf6\\nyJWdcsgECPCK6miX1QN6fN7Yca22R37iKeuXRCKEU8mmivDu9s0AjAxTeWGA/2DX\\nAQAUwv1EAMlptZ78LZ7l9IQLPNQXALrCPtRz4ZkOst0W//eGKnOAEo2d/WGVBksd\\nWJqAGSoftrz44qvjTQH5RWWLCw5pWlwWFUred54=\\n-----END CERTIFICATE-----\\nsubject=C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\\n\\nissuer=C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\\n\\n---\\nAcceptable client certificate CA names\\nC = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\\nC = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Devo AWS US Clients, CN = userAWSUSCA\\nC = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust AWS USA Clients, CN = userAWSUSACA\\nRequested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1\\nShared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512\\nPeer signing digest: SHA256\\nPeer signature type: RSA-PSS\\nServer Temp Key: X25519, 253 bits\\n---\\nSSL handshake has read 5291 bytes and written 330 bytes\\nVerification: OK\\n---\\nNew, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384\\nServer public key is 2048 bit\\nSecure Renegotiation IS NOT supported\\nCompression: NONE\\nExpansion: NONE\\nNo ALPN negotiated\\nEarly data was not sent\\nVerify return code: 0 (ok)\\n---\", \"stdout_lines\": [\"CONNECTED(00000003)\", \"---\", \"Certificate chain\", \" 0 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\", \" i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\", \" 1 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\", \" i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\", \" 2 s:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\", \" i:C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\", \"---\", \"Server certificate\", \"-----BEGIN CERTIFICATE-----\", \"MIIEaTCCAlECCQDEnVpaeewKTDANBgkqhkiG9w0BAQsFADBoMQswCQYDVQQGEwJF\", \"UzEPMA0GA1UECAwGTWFkcmlkMQ8wDQYDVQQHDAZNYWRyaWQxETAPBgNVBAoMCExv\", \"Z1RydXN0MRIwEAYDVQQLDAlMb2cgcmVsYXkxEDAOBgNVBAMMB3JlbGF5Q0EwHhcN\", \"MTkwODMwMDg0NzQ2WhcNMjkwODI3MDg0NzQ2WjCBhDELMAkGA1UEBhMCRVMxDzAN\", \"BgNVBAgMBk1hZHJpZDEPMA0GA1UEBwwGTWFkcmlkMREwDwYDVQQKDAhMb2dUcnVz\", \"dDEcMBoGA1UECwwTTG9nIHJlbGF5IGF0IEFtYXpvbjEiMCAGA1UEAwwZdXMuZWxi\", \"LnJlbGF5LmxvZ3RydXN0Lm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\", \"ggEBAOFckGfv5RvMB7/YIDT9XaALJ7mBmzP4vHY29SaQyVw2eDEULO60GzP2RYB1\", \"4zmwYb/YVFlpXCwoiHJgFG7wTutZ6Vpg+qhwvIlO5woiUvXXImK6oiRTgaY4UAF1\", \"K/m0gXqtyL21LobLAX8YLaQV4wK2SOBVykF4EEIVJG0nh+Xn1U0thGj9P22jcUsD\", \"dQRkMFP+XpdIGOS8SLPXLKy93IcETBc5LTUDGjEoYcUnAScHyY5ZgS39FVYcrDea\", \"YzcxVEIbUmL9UnovXV2UhI+i2yyNDdKFtUfVNyVXEVAfoack+hUu2uackCMeNtO7\", \"+rkxNb6lo5fk4p7dFkSCbzyZyNECAwEAATANBgkqhkiG9w0BAQsFAAOCAgEAJX3J\", \"gmxwASZcJ0UbSGc/T4mu7J11Epe9cO+YRa2kj/Ai2U03zH/PLc98NAjuLCUFDaKv\", \"9+gsViXwOVur5f/Bpzem43Wpgs2Fq/dyTk34lWulJkJ4xRQ9gpgZXrt9GdiP645j\", \"ts6xRe7S9apNmDhzdmzNY3wSUtSA4DxhQ6Q0TAr+gV8T1sBThIWlkvh8IBuURkW8\", \"42FZvA+5n/EW/1nUGDSO4D1fidux2AmoEpoBbX/YG+e6LMeW/csnHZi5/rcT8HSm\", \"e9qCNlmXF8tRHMpxyKcmO/FJdiBwXfwWF9A1zeDyKof3u0xw5KAAeeq5hHYe5uZv\", \"RRhWKh6IQ3L8+uVUCQ1ESAihDYhq7u1tznsPtkw+NMhBbBtcTLccyUnhMg7C1Ug/\", \"XgjtQk6L7CO+0NQCVmLyeEGNKQVb529ZJlEWuDZbR+f9l3ggcGw+KC1M38dmovHY\", \"ocaaU/5zfUI+oiu/KxOI3W5M6g7G1oTgzyvDg/qsp2J5en31oLjyl2r5t6unawf6\", \"yJWdcsgECPCK6miX1QN6fN7Yca22R37iKeuXRCKEU8mmivDu9s0AjAxTeWGA/2DX\", \"AQAUwv1EAMlptZ78LZ7l9IQLPNQXALrCPtRz4ZkOst0W//eGKnOAEo2d/WGVBksd\", \"WJqAGSoftrz44qvjTQH5RWWLCw5pWlwWFUred54=\", \"-----END CERTIFICATE-----\", \"subject=C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay at Amazon, CN = us.elb.relay.logtrust.net\", \"\", \"issuer=C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Log relay, CN = relayCA\", \"\", \"---\", \"Acceptable client certificate CA names\", \"C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust Certification Authority, CN = logtrustCA\", \"C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = Devo AWS US Clients, CN = userAWSUSCA\", \"C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust AWS USA Clients, CN = userAWSUSACA\", \"Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1\", \"Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512\", \"Peer signing digest: SHA256\", \"Peer signature type: RSA-PSS\", \"Server Temp Key: X25519, 253 bits\", \"---\", \"SSL handshake has read 5291 bytes and written 330 bytes\", \"Verification: OK\", \"---\", \"New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384\", \"Server public key is 2048 bit\", \"Secure Renegotiation IS NOT supported\", \"Compression: NONE\", \"Expansion: NONE\", \"No ALPN negotiated\", \"Early data was not sent\", \"Verify return code: 0 (ok)\", \"---\"]}\n\nPLAY RECAP *********************************************************************\ndevo-ea-manager : ok=26 changed=5 unreachable=0 failed=1 skipped=13 rescued=0 ignored=0\n","type":"text"}]},{"type":"paragraph","content":[{"text":"It is probably caused by deprecated Devo certificates (","type":"text"},{"text":"chain","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":".crt","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":".key","type":"text","marks":[{"type":"code"}]},{"text":"), due to security reasons, the platform now offers certificates with stronger encryption and is deprecating the previous ones. To solve it, you can create and download new ones at Devo.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Certifi dependency library incompatible with python 2 up to devo-ea-deployer 1.2.x","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see the following error, or similar, during ","type":"text"},{"text":"ansible-playbook","type":"text","marks":[{"type":"code"}]},{"text":" command runs when using python 2 to run Ansible commands in target hosts:","type":"text"}]},{"type":"codeBlock","content":[{"text":"An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SyntaxError: invalid syntax\nfatal: [devo-ea-manager]: FAILED! => {\"changed\": false, \"module_stderr\": \"Shared connection to \n...\n File \"/usr/lib/python2.7/site-packages/requests/__init__.py\", line 133, in \n from . import utils\n File \"/usr/lib/python2.7/site-packages/requests/utils.py\", line 27, in \n from . import certs\n File \"/usr/lib/python2.7/site-packages/requests/certs.py\", line 15, in \n from certifi import where\n File \"/usr/lib/python2.7/site-packages/certifi/__init__.py\", line 1, in \n from .core import contents, where\n File \"/usr/lib/python2.7/site-packages/certifi/core.py\", line 17\n def where() -> str:\n ^\nSyntaxError: invalid syntax\n\", \"msg\": \"MODULE FAILURE\nSee stdout/stderr for the exact error\", \"rc\": 1}","type":"text"}]},{"type":"paragraph","content":[{"text":"It is probably caused by a non-supported version of ","type":"text"},{"text":"certifi","type":"text","marks":[{"type":"code"}]},{"text":" python library installed in the target hosts during the deployment process.","type":"text"}]},{"type":"paragraph","content":[{"text":"This issue will be fixed in future ","type":"text"},{"text":"devo-ea-deployer","type":"text","marks":[{"type":"code"}]},{"text":" versions but affects versions prior to 1.2.X. Run next commands in the target host (or target hosts) as a workaround to solve the problem (replacing 2.7 for the python 2.X version in your host).","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"sudo su -\nmv /usr/lib/python2.7/site-packages/certifi /usr/lib/python2.7/site-packages/certifi.back\npip uninstall -y docker certifi\npip install docker==4.4.4 certifi==2021.10.8\nrm -fr /usr/lib/python2.7/site-packages/certifi.back\nexit","type":"text"}]},{"type":"paragraph","content":[{"text":"Then run the ","type":"text"},{"text":"ansible-playbook","type":"text","marks":[{"type":"code"}]},{"text":" command again to continue with the deployment process.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Endpoint Agent client","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"This section describes typical trouble scenarios and troubleshooting guidelines for the client (OSQuery + extensions).","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Effective configuration applied in a running and connected agent.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can get the effective configuration options set value in each agent with next SQL query:","type":"text"}]},{"type":"codeBlock","content":[{"text":"SELECT * FROM osquery_flags","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"One perfect way to run this query is using ","type":"text"},{"text":"devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":" web UI because we are in the scenario that agent is connected to DEAM.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Packages loaded in a running and connected agent.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can get the packs loaded for each agent with next SQL query:","type":"text"}]},{"type":"codeBlock","content":[{"text":"SELECT * FROM osquery_packs","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"One perfect way to run this query is using ","type":"text"},{"text":"devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":" web UI because we are on the scenario that agent is connected to DEAM.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Scheduled queries in a running and connected agent.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can get the current scheduled queries loaded for each agent with next SQL query:","type":"text"}]},{"type":"codeBlock","content":[{"text":"SELECT * FROM osquery_schedule","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"One perfect way to run this query is using ","type":"text"},{"text":"devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":" web UI because we are in the scenario that agent is connected to DEAM.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Temporally set log-level to debug in agent.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can temporally set agent level logger to debug and displaying messages to stdout following the next steps.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This configuration must be enabled only during a short period of time and use as last resource to identify issues, because this mode can severely affect to agent performance and significantly increases the amount of data ingested into Devo.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Windows platform","type":"text"}]},{"type":"paragraph","content":[{"text":"All commands detailed below must be run as admin in a Power-shell console","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ensure ","type":"text"},{"text":"osqueryd","type":"text","marks":[{"type":"code"}]},{"text":" service is stopped:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"Stop-Service -Force -Name \"osqueryd\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run osqueryd with Debug log-level (assuming ","type":"text"},{"text":"devo-ea","type":"text","marks":[{"type":"code"}]},{"text":" agent was installed following default installation):","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"cd \"c:\\Program Files\\osquery\"\n.\\osqueryd\\osqueryd.exe --flagfile \"osquery.flags\" --verbose","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Alternatively, you can save ","type":"text"},{"text":"stdout","type":"text","marks":[{"type":"code"}]},{"text":" to a file running next commands, instead of running previous ones:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"cd \"c:\\Program Files\\osquery\"\n.\\osqueryd\\osqueryd.exe --flagfile \"osquery.flags\" --verbose 2>&1 | Tee \"$Env:HOMEPATH\\Desktop\\devo-ea-agent-verbose.out\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Stop typing ","type":"text"},{"text":"Ctrl + C","type":"text","marks":[{"type":"code"}]},{"text":" in PowerShell console to stop current debug process when you will finish your tests or probes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start service:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"Start-Service -Force -Name \"osqueryd\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"type":"hardBreak"},{"text":"Linux platform","type":"text"}]},{"type":"paragraph","content":[{"text":"Next commands are based on ","type":"text"},{"text":"systemd","type":"text","marks":[{"type":"code"}]},{"text":" init system. Adapt start/stop services for other init systems.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ensure ","type":"text"},{"text":"osqueryd","type":"text","marks":[{"type":"code"}]},{"text":" service is stopped:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo systemctl stop osqueryd","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"osqueryd","type":"text","marks":[{"type":"code"}]},{"text":" with Debug log-level (assuming ","type":"text"},{"text":"devo-ea","type":"text","marks":[{"type":"code"}]},{"text":" agent was installed following default installation):","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryd --flagfile \"/etc/osquery/osquery.flags\" --verbose","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Alternatively, you can save ","type":"text"},{"text":"stdout","type":"text","marks":[{"type":"code"}]},{"text":" to a file running next commands, instead of running previous ones:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryd --flagfile \"/etc/osquery/osquery.flags\" --verbose 2>&1 | tee \"$HOME/devo-ea-agent-verbose.out\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Stop typing ","type":"text"},{"text":"Ctrl + C","type":"text","marks":[{"type":"code"}]},{"text":" in previous console to stop current debug process when you will finish your tests or probes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Set right owner if you saved output to a file:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo chown $(id -u):$(id -g) \"$HOME/devo-ea-agent-verbose.out\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start service:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo systemctl start osqueryd","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"type":"hardBreak"},{"text":"macOS platform","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ensure ","type":"text"},{"text":"osqueryd","type":"text","marks":[{"type":"code"}]},{"text":" service is stopped:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryctl stop","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"osqueryd","type":"text","marks":[{"type":"code"}]},{"text":" with Debug log-level (assuming ","type":"text"},{"text":"devo-ea","type":"text","marks":[{"type":"code"}]},{"text":" agent was installed following default installation):","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryd /private/var/osquery/osquery.flags --verbose","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Alternatively, you can save ","type":"text"},{"text":"stdout","type":"text","marks":[{"type":"code"}]},{"text":" to a file running next commands, instead of run previous ones:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryd /private/var/osquery/osquery.flags --verbose 2>&1 | tee \"$HOME/devo-ea-agent-verbose.out\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Stop typing ","type":"text"},{"text":"Ctrl + C ","type":"text","marks":[{"type":"code"}]},{"text":"in previous console to stop current debug process when you will finish your tests or probes.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Set right owner if you saved output to a file:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo chown $(id -u):$(id -g) \"$HOME/devo-ea-agent-verbose.out\"","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Start service:","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"sudo osqueryctl start","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"My agent host is not showing up in the Manager Interface.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Make sure you can reach the DEAM manager from your client:","type":"text"},{"type":"hardBreak"},{"text":"a. ","type":"text"},{"text":"Linux","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"sudo telnet devo-ea-manager:8080 ","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"b. ","type":"text"},{"text":"Windows","type":"text","marks":[{"type":"strong"}]},{"text":": Open a PS shell in admin mode: ","type":"text"},{"text":"Test-NetConnection -ComputerName devo-ea-manager -InformationLevel Detailed -Port 8080","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Make sure there is no firewall or antivirus interfering with the connection. ","type":"text"},{"type":"hardBreak"},{"text":"a. ","type":"text"},{"text":"Windows","type":"text","marks":[{"type":"strong"}]},{"text":": We have seen previous issues with some installed antivirus: It could be necessary to create an Outbound rule both in antivirus and windows firewall to enable communication with the manager. After rule is created, test again with the step in the previous step. In order to check FW status: ","type":"text"},{"text":"netsh firewall show state","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Will not autoload extension with unsafe directory permissions","type":"text"}]},{"type":"paragraph","content":[{"text":"Osquery will refuse to load an extension from the filesystem if the file’s permissions allow it to be written or modified accounts that lack required privileges. The installation script of the EA should take care of it, but in case of error, make sure that the extensions files are owned by the root account.","type":"text"}]},{"type":"paragraph","content":[{"text":"On Windows, because of permission inheritance, just changing the owner of a file is not sufficient. You must also change the owner of the parent directory, remove all inherited DACLs, and disable inheritance. EA installation script should take care of the permissions, but in case of issues, the following commands will set permissions that satisfy osquery:","type":"text"}]},{"type":"codeBlock","content":[{"text":"icacls \"C:\\Program Files\\osquery\\osqueryd\" /setowner Administrators /t\nicacls \"C:\\Program Files\\osquery\\osqueryd\" /grant Administrators:f /t\nicacls \"C:\\Program Files\\osquery\\osqueryd\" /inheritance:r /t\nicacls \"C:\\Program Files\\osquery\\osqueryd\" /inheritance:d /t","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Make sure that the group “Administrators” exists. In localized Windows servers, the name of the group will be in the localized language. Replace the “Administrators” group with the localized name. In order to check the local groups in the Windows server, use the cmdlet ","type":"text"},{"text":"Get-LocalGroup","type":"text","marks":[{"type":"code"}]},{"text":" in a Powershell window.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Make sure that your organization is not including extra permissions globally to the files in ","type":"text"},{"text":"\"C:\\Program Files\\osquery\\osqueryd\"","type":"text","marks":[{"type":"code"}]},{"text":". Write/modify permissions should only be given to privileged accounts like “Administrators” or “System”. ","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":11.0},"content":[{"type":"media","attrs":{"__fileSize":30951,"__fileMimeType":"image/png","width":361,"id":"5cd1bb24-c9f8-4aca-8d18-add3fa57f0af","collection":"contentId-120918224","type":"file","__fileName":"trouble_60.png","height":480}}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":11.0},"content":[{"type":"media","attrs":{"__fileSize":29455,"__fileMimeType":"image/png","width":362,"id":"5ffecc5c-efcf-40a2-83e7-b37eebe77b88","collection":"contentId-120918224","type":"file","__fileName":"trouble_70.png","height":478}}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":11.0},"content":[{"type":"media","attrs":{"__fileSize":24919,"__fileMimeType":"image/png","width":363,"id":"aa35e6a3-b238-49e7-86a2-afe3ac620631","collection":"contentId-120918224","type":"file","__fileName":"trouble_80.png","height":480}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"macOS: extension cannot be opened because the developer cannot be verified","type":"text"}]},{"type":"paragraph","content":[{"text":"To run software in MacOS, it needs to be signed and notarised, otherwise the Gatekeeper will block it as you might have noticed.","type":"text"}]},{"type":"paragraph","content":[{"text":"Osquery itself is signed by the osquery devs, and we are working to be able to sign our extensions (that run on top of osquery).","type":"text"}]},{"type":"paragraph","content":[{"text":"Until Devo is able to sign their own software for macOS, the only thing that can be done is to manually allow the extensions to run as user even though they are not signed. Follow the instructions in ","type":"text"},{"text":"this link","type":"text","marks":[{"type":"link","attrs":{"href":"https://support.apple.com/en-us/HT202491"}}]},{"text":" to do so.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Endpoint Agent Manager","type":"text"}]},{"type":"paragraph","content":[{"text":"This section describes typical trouble scenarios and troubleshooting guidelines for the manager (Fleet).","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"How to check DEAM logs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"systemctl status devo-ea-manager ","type":"text","marks":[{"type":"code"}]},{"text":"to check status of the process.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"journalctl -u devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":" to check manager logs.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"journalctl -fu devo-ea-manager ","type":"text","marks":[{"type":"code"}]},{"text":"to check real time logs.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"DEAM certificates were not properly generated or uploaded","type":"text"}]},{"type":"paragraph","content":[{"text":"If you see error messages similar to next in DEAM logs (j","type":"text"},{"text":"ournalctl -u devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":"):","type":"text"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 05:52:40 devo-ea-manager fleet[1517]: {\"terminated\":\"open /etc/devo-ea-manager/certs/devo-ea-manager.key: no such file or directory\",\"ts\":\"2020-12-02T10:52:40.484570752Z\"}\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 05:53:52 devo-ea-manager fleet[1538]: {\"terminated\":\"open /etc/devo-ea-manager/certs/devo-ea-manager.crt: no such file or directory\",\"ts\":\"2020-12-02T10:53:52.453080543Z\"}\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If you provided your custom certificates ensure that they are placed in the ","type":"text"},{"text":"provided-deam-certs","type":"text","marks":[{"type":"code"}]},{"text":" folder under devo-ea-deployer path.","type":"text"}]},{"type":"paragraph","content":[{"text":"Then run the ","type":"text"},{"text":"ansible","type":"text","marks":[{"type":"code"}]},{"text":" command again (follow steps described in the deployment guide).","type":"text"}]},{"type":"paragraph","content":[{"text":"If you are delegating creation of self-signed certificates to ","type":"text"},{"text":"devo-ea-deployer","type":"text","marks":[{"type":"code"}]},{"text":", run ","type":"text"},{"text":"ansible","type":"text","marks":[{"type":"code"}]},{"text":" command again and pay attention to messages marked with the ","type":"text"},{"text":"service-certificates","type":"text","marks":[{"type":"code"}]},{"text":" tag. This can help you to identify the root cause.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Domain certificates were not properly configured","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If you see error messages similar to next in DEAM logs (j","type":"text"},{"text":"ournalctl -u devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":"):","type":"text"}]},{"type":"codeBlock","content":[{"text":"..\nDec 02 06:01:54 devo-ea-manager fleet[1660]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:01:54.919850542Z\"}\nDec 02 06:01:54 devo-ea-manager fleet[1660]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not load keypair /etc/devo-ea-manager/devo-certs/domain.crt:/etc/devo-ea-manager/devo-certs/domain.key: open /etc/devo-ea-manager/devo-certs/domain.key: no such file or directory\nDec 02 06:01:54 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 06:04:25 devo-ea-manager fleet[1749]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:04:25.923778139Z\"}\nDec 02 06:04:25 devo-ea-manager fleet[1749]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not load keypair /etc/devo-ea-manager/devo-certs/domain.crt:/etc/devo-ea-manager/devo-certs/domain.key: open /etc/devo-ea-manager/devo-certs/domain.crt: no such file or directory\nDec 02 06:04:25 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 06:05:21 devo-ea-manager fleet[1847]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:05:21.667553321Z\"}\nDec 02 06:05:21 devo-ea-manager fleet[1847]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not read certificate \"/etc/devo-ea-manager/devo-certs/chain.crt\": open /etc/devo-ea-manager/devo-certs/chain.crt: no such file or directory\nDec 02 06:05:21 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 06:07:32 devo-ea-manager fleet[2005]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:07:32.912871438Z\"}\nDec 02 06:07:32 devo-ea-manager fleet[2005]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not load keypair /etc/devo-ea-manager/devo-certs/domain.crt:/etc/devo-ea-manager/devo-certs/domain.key: tls: failed to parse private key\nDec 02 06:07:32 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 06:09:01 devo-ea-manager fleet[2083]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:09:01.417074837Z\"}\nDec 02 06:09:01 devo-ea-manager fleet[2083]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not load keypair /etc/devo-ea-manager/devo-certs/domain.crt:/etc/devo-ea-manager/devo-certs/domain.key: asn1: structure error: tags don't match (16 vs {class:2 tag:24 length:1122 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue: tag: stringType:0 timeType:0 set:false omitEmpty:false} certificate @4\nDec 02 06:09:01 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 02 06:09:59 devo-ea-manager fleet[2131]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-02T11:09:59.176293468Z\"}\nDec 02 06:09:59 devo-ea-manager fleet[2131]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): x509: certificate signed by unknown authority\nDec 02 06:09:59 devo-ea-manager systemd[1]: devo-ea-manager.service: Main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"...\nDec 09 16:14:15 devo-ea-manager fleet[16683]: {\"certPath\":\"/etc/devo-ea-manager/devo-certs/domain.crt\",\"chainPath\":\"/etc/devo-ea-manager/devo-certs/chain.crt\",\"component\":\"Devo-logging\",\"keyPath\":\"/etc/devo-ea-manager/devo-certs/domain.key\",\"level\":\"info\",\"msg\":\"Setup Devo connection using TLS\",\"ts\":\"2020-12-09T15:14:15.95249458Z\"}\nDec 09 16:14:15 devo-ea-manager fleet[16683]: Error initializing service: initializing osquery logging: create devo status logger: create Devo client (TLS): could not load keypair /etc/devo-ea-manager/devo-certs/domain.crt:/etc/devo-ea-manager/devo-certs/domain.key: tls: found a certificate rather than a key in the PEM for the private key\nDec 09 16:14:15 devo-ea-manager systemd[1]: devo-ea-manager.service: main process exited, code=exited, status=1/FAILURE\n...","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"The most likely cause is that the domain certificates were not properly configured. Ensure that you deploy them following instructions described in the deployment guide.","type":"text"}]},{"type":"paragraph","content":[{"text":"One point to check is that the certificates are owned by the ansible use configured in inventories, and not by root.","type":"text"}]},{"type":"paragraph","content":[{"text":"Then run the ","type":"text"},{"text":"ansible","type":"text","marks":[{"type":"code"}]},{"text":" command again (follow steps described in the deployment guide).","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"MySQL issues","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Error 500 after login / \"mysql\":\"could not connect to db: dial tcp [::1]:3306: connect: connection refused","type":"text","marks":[{"type":"em"}]}]},{"type":"paragraph","content":[{"text":"If you get a 500 error right after logging in, and a screen similar to the following, you are likely running into MySQL issues. Verify that you have connectivity with MySQL.","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"inline-media-image","parameters":{"width":"400","id":"79095ece-76a2-4980-b2dc-df058cf30fbe","collection":"contentId-120918224","height":""}}},{"type":"hardBreak"},{"text":"The EA Manager may also show some of the following errors:","type":"text"}]},{"type":"codeBlock","content":[{"text":"Jun 28 15:37:14 devo-ea-manager fleet[710]: {\"component\":\"service\",\"err\":\"find user: dial tcp [::1]:3306: connect: connection refused\",\"level\":\"info\",\"method\":\"Login\",\"took\":\"787.149µs\",\"ts\":\"2021-06-28T15:37:14.012392917Z\",\"user\":\"admin\"}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"Jun 28 15:35:44 devo-ea-manager fleet[710]: {\"component\":\"service\",\"err\":\"SSOSettings getting app config: selecting app config: dial tcp [::1]:3306: connect: connection refused\",\"level\":\"info\",\"method\":\"SSOSettings\",\"took\":\"729.832µs\",\"ts\":\"2021-06-28T15:35:44.770049722Z\"}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"Jun 28 15:35:33 devo-ea-manager fleet[710]: {\"component\":\"http\",\"err\":\"authentication error: finding host\",\"ts\":\"2021-06-28T15:35:33.429262653Z\"}","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"If using dockerized version of the internal services (MySQL and Redis) check that the dockers are up & running in the EA Manager server with: ","type":"text"},{"text":"sudo docker ps -a.","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Execute the following to restart the internal services:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd /srv/deam-internal-services\n/usr/local/bin/docker-compose down\n/usr/local/bin/docker-compose up -d mysql redis\nsystemctl restart devo-ea-manager","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"type":"hardBreak"},{"text":"REDIS issues","type":"text"}]},{"type":"paragraph","content":[{"text":"REDIS is not in the critical path unless the labeling feature in the EA Manager is in use. REDIS issues will surface with the following symptoms:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Cannot run Live Queries.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Labeling does not work.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Errors in EA Manager logs.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"If you see a similar error when accessing the ","type":"text"},{"text":"Queries","type":"text","marks":[{"type":"strong"}]},{"text":" menu in the Web UI, it's likely your REDIS instance is not available:","type":"text"}]},{"type":"paragraph","content":[{"text":" ","type":"text"},{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.migration","extensionKey":"inline-media-image","parameters":{"width":"600","id":"f43ebe3b-50f9-4ef9-9a5f-9c3ecc7fe484","collection":"contentId-120918224","height":""}}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Send events through a Relay","type":"text"}]},{"type":"paragraph","content":[{"text":"Assuming that the relay in-house IP is 192.168.43.147, you should configure ","type":"text"},{"text":"deam_relay_entrypoint: tcp://192.168.43.147:13000","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"},{"type":"hardBreak"},{"text":"Example snipped of inventory file based on that configuration:","type":"text"}]},{"type":"codeBlock","content":[{"text":"all:\n vars:\n ...\n deam_relay_entrypoint: tcp://192.168.43.147:13000\n deam_devo_key: \"\"\n deam_devo_cert: \"\"\n deam_devo_chain: \"\"\n ...","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"type":"hardBreak"},{"text":"Modify listen port of Devo EA package repository","type":"text"}]},{"type":"paragraph","content":[{"text":"Ensure that the 8081 port is available and isn’t busy by another service in the client infrastructure. To overwrite the port you must add in the inventory file based on that configuration the parameter:","type":"text"}]},{"type":"codeBlock","content":[{"text":"all:\n vars:\n ...\n dea_ap_repo_port: \n ...","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"type":"hardBreak"},{"text":"Events are not ingested in right Devo domain","type":"text"}]},{"type":"paragraph","content":[{"text":"Ensure that right ","type":"text"},{"text":"devo-domain","type":"text","marks":[{"type":"code"}]},{"text":" certificates were configured. You can inspect ","type":"text"},{"text":"domain.crt","type":"text","marks":[{"type":"code"}]},{"text":" certificate with next command:","type":"text"}]},{"type":"codeBlock","content":[{"text":"openssl x509 -in /etc/devo-ea-manager/devo-certs/domain.crt -text -noout","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Then look for a line similar to:","type":"text"}]},{"type":"codeBlock","content":[{"text":"Subject: C = SP, ST = Madrid, L = Madrid, O = LogTrust, CN = XXXX ","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"CN","type":"text","marks":[{"type":"code"}]},{"text":" value should be the domain name.","type":"text"}]},{"type":"paragraph","content":[{"text":"In the same output, you can look for a line similar to:","type":"text"}]},{"type":"codeBlock","content":[{"text":"Issuer: C = ES, ST = Madrid, L = Madrid, O = LogTrust, OU = LogTrust AWS USA Clients, CN = userAWSUSACA","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"CN = userAWSxxxxx","type":"text","marks":[{"type":"code"}]},{"text":" indicates to us which is the site that creates certificates. ","type":"text"},{"text":"US","type":"text","marks":[{"type":"code"}]},{"text":" in the example, that matches with ","type":"text"},{"text":"https://us.devo.com","type":"text","marks":[{"type":"link","attrs":{"href":"https://us.devo.com"}}]},{"text":". ","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Missing --devo_relay/KOLIDE_DEVO_RELAY setting in deam-fleet configuration ","type":"text"}]},{"type":"paragraph","content":[{"text":"You will see similar traces in DEAM logs and process does not start when Devo relay property was not set. ","type":"text"}]},{"type":"codeBlock","content":[{"text":"panic: runtime error: index out of range [1] with length 1\n\ngoroutine 1 [running]:\ngithub.com/fleetdm/fleet/server/logging.devoConnection(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1267020, 0xc000282b10, ...)\n /usr/src/app/server/logging/devo.go:316 +0x7a4\ngithub.com/fleetdm/fleet/server/logging.NewDevoLogWriter(0x0, 0x0, 0xff4e06, 0xb, 0x10002d2, 0x15, 0x0, 0x0, 0x0, 0x0, ...)\n /usr/src/app/server/logging/devo.go:73 +0xb8\ngithub.com/fleetdm/fleet/server/logging.New(0xfed8ff, 0x3, 0xff7ec3, 0xe, 0xfefd9d, 0x6, 0xfefd9d, 0x6, 0xfefd9d, 0x6, ...)\n /usr/src/app/server/logging/logging.go:152 +0x3db\ngithub.com/fleetdm/fleet/server/service.NewService(0x129cfa0, 0xc0004aa000, 0x12795a0, 0xc00020a060, 0x1267020, 0xc000282b10, 0xfed8ff, 0x3, 0xff7ec3, 0xe, ...)\n /usr/src/app/server/service/service.go:27 +0x83\nmain.createServeCmd.func1(0xc00049ef00, 0xc000205b00, 0x0, 0x4)\n /usr/src/app/cmd/fleet/serve.go:184 +0x836\ngithub.com/spf13/cobra.(*Command).execute(0xc00049ef00, 0xc000205a40, 0x4, 0x4, 0xc00049ef00, 0xc000205a40)\n /go/pkg/mod/github.com/spf13/cobra@v0.0.2/command.go:760 +0x29d\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xc00049e780, 0xc0005dff58, 0x1, 0x1)\n /go/pkg/mod/github.com/spf13/cobra@v0.0.2/command.go:846 +0x2ea\ngithub.com/spf13/cobra.(*Command).Execute(...)\n /go/pkg/mod/github.com/spf13/cobra@v0.0.2/command.go:794\nmain.main()\n /usr/src/app/cmd/fleet/main.go:29 +0x1cd","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"devo-ea-deployer ","type":"text","marks":[{"type":"code"}]},{"text":"installation procedure fills this value from ","type":"text"},{"text":"deam_relay_entrypoint","type":"text","marks":[{"type":"code"}]},{"text":" variable (set to ","type":"text"},{"text":"tcp://us.elb.relay.logtrust.net:443","type":"text","marks":[{"type":"code"}]},{"text":" value by default) the value is assigned to ","type":"text"},{"text":"KOLIDE_DEVO_RELAY","type":"text","marks":[{"type":"code"}]},{"text":" environment variable configured in ","type":"text"},{"text":"/etc/devo-ea-manager/devo-ea-manager","type":"text","marks":[{"type":"code"}]},{"text":" file by default.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Sending Windows Events for testing, using the command line","type":"text"}]},{"type":"paragraph","content":[{"text":"The Windows Events are generated by the system according to some internal conditions, not controlled directly by the system administrators. Sometimes is useful for testing purposes to be able to send certain events by demand., for instance to check that we are receiving events of some type. This is possible using the windows utility ","type":"text"},{"text":"eventcreate","type":"text","marks":[{"type":"em"}]},{"text":" from the command line. For example: ","type":"text"}]},{"type":"paragraph","content":[{"text":"eventcreate /t ERROR /id 100 /l application /d \"test event\"","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"A complete description of the tool and more examples can be found ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/eventcreate"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"cookiecutinv.py does not work in old distributions.","type":"text"}]},{"type":"paragraph","content":[{"text":"This issue happens in old Linux distributions, we identified it in RedHat 7 but it can happen in other old Linux versions too. ","type":"text"},{"text":"cookiecutinv.py","type":"text","marks":[{"type":"code"}]},{"text":" displays next error message just when it starts to create inventory file:","type":"text"}]},{"type":"codeBlock","content":[{"text":"Traceback (most recent call last):\n File \"tools/cookiecutinvt.py\", line 668, in \n main()\n File \"tools/cookiecutinvt.py\", line 47, in main\n make_inventory(cfg, data)\n File \"tools/cookiecutinvt.py\", line 171, in make_inventory\n j2_tmpl = Template(tpl_data)\n File \"/usr/lib/python2.7/site-packages/jinja2/environment.py\", line 906, in __new__\n return env.from_string(source, template_class=cls)\n File \"/usr/lib/python2.7/site-packages/jinja2/environment.py\", line 841, in from_string\n return cls.from_code(self, self.compile(source), globals, None)\n File \"/usr/lib/python2.7/site-packages/jinja2/environment.py\", line 554, in compile\n self.handle_exception(exc_info, source_hint=source)\n File \"/usr/lib/python2.7/site-packages/jinja2/environment.py\", line 742, in handle_exception\n reraise(exc_type, exc_value, tb)\n File \"\", line 5, in template\njinja2.exceptions.TemplateAssertionError: no filter named 'tojson'\n","type":"text"}]},{"type":"paragraph","content":[{"text":"The main cause is very old version of libraries required by ","type":"text"},{"text":"cookiecutinv.py","type":"text","marks":[{"type":"code"}]},{"text":". Remember that all dependency libraries of tools are just the same as ","type":"text"},{"text":"Ansible","type":"text","marks":[{"type":"em"}]},{"text":", but in this case, ","type":"text"},{"text":"a","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"b","type":"text","marks":[{"type":"code"}]},{"text":" versions installed in those old distributions are very old.","type":"text"}]},{"type":"paragraph","content":[{"text":"To fix the problem you can install more recent ","type":"text"},{"text":"Ansible","type":"text","marks":[{"type":"em"}]},{"text":" version, with mentioned dependencies updated, using ","type":"text"},{"text":"getansible.sh","type":"text","marks":[{"type":"code"}]},{"text":" script. This script will install ","type":"text"},{"text":"Ansible","type":"text","marks":[{"type":"em"}]},{"text":" 2.9 ","type":"text"},{"text":"only for the user","type":"text","marks":[{"type":"strong"}]},{"text":" that is executing the script.","type":"text"}]},{"type":"paragraph","content":[{"text":"Follow the next steps to complete the installation:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Remove ","type":"text"},{"text":"Ansible","type":"text","marks":[{"type":"em"}]},{"text":" tool installed using package manager. Remember that ","type":"text"},{"text":"ansible","type":"text","marks":[{"type":"code"}]},{"text":" command will not be available to other host users after running this step (yum is the package manager tool used in this example and maybe you should use the appropriate tool for your case):","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo yum remove -y ansible","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"getansible.sh","type":"text","marks":[{"type":"code"}]},{"text":" downloading directly from public ","type":"text"},{"text":"Devo agent repository. ","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"curl -sSu '<>:<>' https://d2ur64jmn3k7yt.cloudfront.net/gtls/getansible.sh | /bin/bash","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"NOTE:","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"sudo","type":"text","marks":[{"type":"code"}]},{"text":" access is required in order to install package dependencies.","type":"text"},{"type":"hardBreak"},{"text":"Where:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"<>","type":"text","marks":[{"type":"code"}]},{"text":" is the username required to authenticate in ","type":"text"},{"text":"Devo agent repository","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"<>","type":"text","marks":[{"type":"code"}]},{"text":" is the password required to authenticate in ","type":"text"},{"text":"Devo agent repository","type":"text","marks":[{"type":"em"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Be aware that might you need to close and open a new session in order to get ","type":"text"},{"text":"ansible","type":"text","marks":[{"type":"code"}]},{"text":" command available for use that ran the script.","type":"text"}]},{"type":"codeBlock","content":[{"text":"ansible --version","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"cookiecutinv.py","type":"text","marks":[{"type":"code"}]},{"text":" to create inventory and continue with the installation procedure.","type":"text"}]}]}]}],"version":1}

Browser not supported