is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io\n2023-01-25T12:54:13.498 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The

is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io\n2023-01-25T12:54:13.502 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The

is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io\n2023-01-25T12:54:13.503 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,relay_0) -> Starting thread (every 300 seconds)\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,relay_0) -> Starting thread\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Events delivery and Devo ingestion","type":"text"}]},{"type":"paragraph","content":[{"text":"The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.","type":"text"}]},{"type":"paragraph","content":[{"text":"A successful run has the following output messages for the initializer module:","type":"text"}]},{"type":"codeBlock","content":[{"text":"2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added \"/devo-collector\" directory to the Python path\n2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added \"/devo-collector/config_internal\" directory to the Python path\n2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added \"/devo-collector/schemas\" directory to the Python path\n2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Production mode: True, execute only setup and exit: False, Python version: \"3.9.10 (main, Jan 24 2023, 13:13:38) [GCC 9.4.0]\", current dir: \"/devo-collector\", exists \"config\" dir: True, exists \"config_internal\" dir: True, exists \"certs\" dir: True, exists \"schemas\" dir: True, exists \"credentials\" dir: True\n2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Loading configuration using the following files: {\"full_config\": \"config.yaml\", \"job_config_loc\": null, \"collector_config_loc\": null}\n2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> List of files in directory \"/devo-collector/config\": []\n2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for \"job_config_loc\" file: \"/etc/devo/job/job_config.json\"\n2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for \"collector_config_loc\" file: \"/etc/devo/collector/collector_config.json\"\n2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Results of validation of config files parameters: {\"config\": \"/devo-collector/config/config.yaml\", \"config_validated\": False, \"job_config_loc\": \"/etc/devo/job/job_config.json\", \"job_config_loc_default\": True, \"job_config_loc_validated\": True, \"collector_config_loc\": \"/etc/devo/collector/collector_config.json\", \"collector_config_loc_default\": True, \"collector_config_loc_validated\": True}\n2023-01-25T12:54:13.436 INFO MainProcess::MainThread -> Build time: \"2023-01-24T17:01:02.855958522+0000\", OS: \"Linux-5.4.149-73.259.amzn2.x86_64-x86_64-with-glibc2.31\", collector(name:version): \"SOCRadar_collector:1.0.0\", owner: \"integrations_factory@devo.com\", started at: \"2023-01-25T12:54:13.433352Z\"\n2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> Initialized all object from \"MainProcess\" process\n2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=120s)\n2023-01-25T12:54:13.443 INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=120s)\n2023-01-25T12:54:13.444 INFO OutputProcess::MainThread -> Process started\n2023-01-25T12:54:13.448 INFO MainProcess::MainThread -> Started all object from \"MainProcess\" process\n2023-01-25T12:54:13.448 INFO InputProcess::MainThread -> Process Started\n2023-01-25T12:54:13.449 INFO MainProcess::CollectorThread -> global_status: {\"main_process\": {\"process_id\": 7, \"process_status\": \"sleeping\", \"thread_counter\": 5, \"thread_names\": [\"MainThread\", \"QueueFeederThread\", \"OutputControllerThread\", \"InputControllerThread\", \"CollectorThread\"], \"memory_info\": {\"rss\": \"51.41MiB\", \"vms\": \"360.62MiB\", \"shared\": \"14.38MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"76.79MiB\", \"dirty\": \"0.00B\"}, \"CollectorThread\": {\"running_flag\": true, \"status\": \"RUNNING(10)\", \"shutdown_timestamp\": \"None\", \"message_queues\": {\"standard\": {\"name\": \"standard_queue_multiprocessing\", \"max_size_in_messages\": 10000, \"max_size_in_mb\": 1024, \"max_wrap_size_in_items\": 100, \"current_size\": 0, \"put_lock\": \"\", \"input_lock\": \"\"}, \"lookup\": {\"name\": \"lookup_queue_multiprocessing\", \"max_size_in_messages\": 10000, \"max_size_in_mb\": 1024, \"max_wrap_size_in_items\": 100, \"current_size\": 0, \"put_lock\": \"\", \"input_lock\": \"\"}, \"internal\": {\"name\": \"internal_queue_multiprocessing\", \"max_size_in_messages\": 10000, \"max_size_in_mb\": 1024, \"max_wrap_size_in_items\": 100, \"current_size\": 2, \"put_lock\": \"\", \"input_lock\": \"\"}}, \"controllers\": {\"InputControllerThread\": {\"running_flag\": true, \"status\": \"RUNNING(10)\", \"underlying_process\": {\"process_id\": 12, \"process_status\": \"running\", \"num_threads\": 2, \"memory_info\": {\"rss\": \"41.45MiB\", \"vms\": \"212.62MiB\", \"shared\": \"4.45MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"56.54MiB\", \"dirty\": \"0.00B\"}, \"exit_code\": null}}, \"OutputControllerThread\": {\"running_flag\": true, \"status\": \"RUNNING(10)\", \"underlying_process\": {\"process_id\": 10, \"process_status\": \"running\", \"num_threads\": 1, \"memory_info\": {\"rss\": \"40.77MiB\", \"vms\": \"138.61MiB\", \"shared\": \"3.79MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"46.41MiB\", \"dirty\": \"0.00B\"}, \"exit_code\": null}}}}}}\n2023-01-25T12:54:13.451 INFO MainProcess::CollectorThread -> environment_status: {\"cpu_count\": 8, \"cpu_usage\": \"1:3.09, 5:3.26, 15:3.67, \", \"virtual_memory\": {\"total\": \"62.03GiB\", \"available\": \"56.24GiB\", \"percent\": 9.3, \"used\": \"5.50GiB\", \"free\": \"43.61GiB\", \"active\": \"4.72GiB\", \"inactive\": \"11.08GiB\", \"buffers\": \"2.64MiB\", \"cached\": \"12.92GiB\", \"shared\": \"9.35MiB\", \"slab\": \"2.19GiB\"}, \"disk_usage\": {\"/dev/termination-log\": 5.3, \"/etc/hosts\": 5.3, \"/etc/resolv.conf\": 5.3, \"/etc/hostname\": 5.3, \"/etc/devo/collector\": 5.3}, \"processes_info\": [{\"pid\": 1, \"ppid\": 0, \"process_name\": \"sh\", \"process_cmd\": \"/bin/sh -c devo-collector --config ${CONFIG_FILE:-config.yaml} --prod-mode\", \"memory_usage\": {\"rss\": \"596.00KiB\", \"vms\": \"2.55MiB\", \"shared\": \"528.00KiB\", \"text\": \"96.00KiB\", \"lib\": \"0.00B\", \"data\": \"316.00KiB\", \"dirty\": \"0.00B\"}}, {\"pid\": 7, \"ppid\": 1, \"process_name\": \"devo-collector\", \"process_cmd\": \"/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode\", \"memory_usage\": {\"rss\": \"51.57MiB\", \"vms\": \"360.62MiB\", \"shared\": \"14.49MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"76.79MiB\", \"dirty\": \"0.00B\"}}, {\"pid\": 10, \"ppid\": 7, \"process_name\": \"devo-collector\", \"process_cmd\": \"/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode\", \"memory_usage\": {\"rss\": \"40.77MiB\", \"vms\": \"138.61MiB\", \"shared\": \"3.79MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"46.41MiB\", \"dirty\": \"0.00B\"}}, {\"pid\": 12, \"ppid\": 7, \"process_name\": \"devo-collector\", \"process_cmd\": \"/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode\", \"memory_usage\": {\"rss\": \"41.46MiB\", \"vms\": \"212.62MiB\", \"shared\": \"4.45MiB\", \"text\": \"8.00KiB\", \"lib\": \"0.00B\", \"data\": \"56.54MiB\", \"dirty\": \"0.00B\"}}]}\n2023-01-25T12:54:13.493 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The

is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io\n2023-01-25T12:54:13.503 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,relay_0) -> Starting thread (every 300 seconds)\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,relay_0) -> Starting thread\n2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread\n2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,relay_0) -> Starting thread (every 300 seconds)\n2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,relay_0) -> Starting thread\n2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread\n2023-01-25T12:54:13.506 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,relay_0) -> Starting thread (every 300 seconds)\n2023-01-25T12:54:13.506 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,relay_0) -> Starting thread\n2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> InputThread(socradar,17812) - Starting thread (execution_period=60s)\n2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> ServiceThread(socradar,17812,incidents,predefined) - Starting thread (execution_period=60s)\n2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> SOCRadarPullerSetup(SOCRadar_collector,socradar#17812,incidents#predefined) -> Starting thread\n2023-01-25T12:54:13.527 INFO InputProcess::MainThread -> SOCRadarPuller(socradar,17812,incidents,predefined) - Starting thread","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"By default, these information traces will be displayed every 10 minutes.","type":"text"}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Sender services","type":"text"}]},{"type":"paragraph","content":[{"text":"The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (","type":"text"},{"text":"internal","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"standard","type":"text","marks":[{"type":"code"}]},{"text":", and ","type":"text"},{"text":"lookup","type":"text","marks":[{"type":"code"}]},{"text":"). This collector uses the following Sender Services:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"6bd704ef-0e1e-4407-992c-002264d15132"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-neutral, #F4F5F7)","rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Sender services","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-neutral, #F4F5F7)","rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"internal_senders","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"In charge of delivering internal metrics to Devo such as logging traces or metrics.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"standard_senders","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"In charge of delivering pulled events to Devo.","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Sender statistics","type":"text"}]},{"type":"paragraph","content":[{"text":"Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"b4ce6589-85aa-4583-b31b-545216de40ff"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-neutral, #F4F5F7)","rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Logging trace","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"var(--ds-background-neutral, #F4F5F7)","rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Number of available senders: 1","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Displays the number of concurrent senders available for the given Sender Service.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"sender manager internal queue size: 0","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Displays the items available in the internal sender queue.","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.","type":"text"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Standard - Total number of messages sent: 57, messages sent since \"2023-01-10 16:09:16.116750+00:00\": 0 (elapsed 0.000 seconds","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[365.0]},"content":[{"type":"paragraph","content":[{"text":"Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"44 events were sent to Devo since the collector started.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The last checkpoint timestamp was ","type":"text"},{"text":"2023-01-10 16:09:16.116750+00:00","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"21 events where sent to Devo between the last UTC checkpoint and now.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Those 21 events required ","type":"text"},{"text":"0.00 seconds","type":"text","marks":[{"type":"code"}]},{"text":" to be delivered.","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"By default these traces will be shown every 10 minutes.","type":"text"}]}]}]}]}]}]}],"version":1}

Browser not supported