...
| Info |
|---|
More information Refer to the Vendor setup section to know more about these configurations. |
...
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
|---|
Service events | The different available services in AWS usually generate some information related to their internal behaviors, such as "a virtual machine has been started", "a new file has been created in an S3 bucket" or "an AWS lambda function has been invoked" and this kind of event can be triggered by no human interaction. The service events are managed by the CloudWatch Events service (CWE), recently AWS has created a new service called Amazon EventBridge that tends to replace the CWE service. The findings detected by AWS Security Hub are also managed by CloudWatch Events (CWE). | ReceiveMessage
ReceiveMessage - Amazon Simple Queue Service | Generic events: service-events-all
Security Hub events: sqs-cloudwatch-consumer
| Generic events: Security Hub events: | -
|
Audit events | This kind of event is more specific because they are triggered by a human interaction no matter the different ways used: API, web interaction, or even the CLI console. The audit events are managed by the CloudTrail service. There are two ways to read Audit events: API: using CloudTrail API. This way is slower, but it can retrieve data back in time. S3+SQS: forwarding CloudTrail data to an S3 bucket and reading from there through a SQS queue. This way is much faster, but it only can retrieve elements since the creation of the S3+SQS pipeline.
| Via API: LookupEvents
LookupEvents - AWS CloudTrail Via S3+SQS: ReceiveMessage
ReceiveMessage - Amazon Simple Queue Service | audit-events-all
| | -
|
Metrics | According to the standard definition, this kind of information is usually generated at the same moment is requested because it is usually a query about the status of a service (all things inside AWS are considered services). AWS makes something slightly different because what is doing is to generate metrics information every N time slots, such as 1 min, 5 min, 30 min, 1h, etc., even if no one makes a request (also is possible to have information every X seconds but this would require extra costs). The metrics are managed by the CloudWatch Metrics service (CWM). | ListMetrics
ListMetrics - Amazon CloudWatch After listing the metrics, GetMetricData and GetMetricStatistics are also called. GetMetricData - Amazon CloudWatch GetMetricStatistics - Amazon CloudWatch | metrics-all
| cloud.aws.cloudwatch.metrics
| -
|
Logs | Logs could be defined as information with a non-fixed structure that is sent to one of the available “logging” services, these services are CloudWatch Logs and S3. There are some very customizable services, such as AWS Lambda, or even any developed application which is deployed inside an AWS virtual machine (EC2), that can generate custom log information, this kind of information is managed by the CloudWatch Logs service (CWL) and also by the S3 service. There are also some other services that can generate logs with a fixed structure, such as VPC Flow Logs or CloudFront Logs. These kinds of services require one special way of collecting their data. | DescribeLogStreams
DescribeLogStreams - Amazon CloudWatch Logs | Logs can be: Managed by Cloudwatch: This is a custom service that is activated using service custom_service and including the type logs into the types parameter in the config file. Not managed by Cloudwatch: Use non-cloudwatch-logs service and include the required type (flowlogs for VPC Flow Logs and/or cloudfrontlogs for CloudFront Logs) into the types parameter in the config file.
| | -
|
AWS GuardDuty | AWS GuardDuty is a managed threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon S3. Data Sources: GuardDuty ingests and processes data from AWS CloudTrail logs, VPC Flow Logs, and DNS logs
Findings: When a potential threat is detected, GuardDuty generates a finding. These findings provide details about the activity, including the affected resources, type of threat, and suggested remediation actions. We are using API to get findings of guardduty GuardDuty service. | What is Amazon GuardDuty? - Amazon GuardDuty | aws-guardduty
| | v1.10.0
|
Cisco Umbrella [Non-AWS service] | Cisco Umbrella is a cloud-driven Secure Internet Gateway (SIG) that leverages insights gained through the analysis of various logs, including DNS logs, IP logs, and Proxy logs, to provide a first line of defense. DNS logs record all DNS queries that are made through the Cisco Umbrella DNS resolvers. These logs contain data about the DNS queries originating from your network, requested domain names and the IP address of the requester. IP logs capture all IP-based communications that occur through the network. These logs store details such as the source and destination IP addresses, ports and protocols used. Proxy logs are generated when users access web resources through the Cisco Umbrella intelligent proxy. They contain detailed information on the web traffic including the URL accessed, the method of access (GET, POST, etc.), the response status, etc | Via S3+SQS: ReceiveMessage
ReceiveMessage - Amazon Simple Queue Service | cisco-umbrella
| sig.cisco.umbrella.dns
sig.cisco.umbrella.ip
sig.cisco.umbrella.proxy
| v1.6.0
|
...
This section is intended to explain how to proceed with specific actions for services.
...
Service events (all services)
This service could be considered a general AWS event puller. It reads events from all the AWS services, which are managed by CloudWatch.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
If auto_event_type parameter is not set or is set to false, the events are going to be ingested into the table cloud.aws.cloudwatch.events If auto_event_type parameter is set to true, the events are going to be ingested into the table cloud.aws.cloudwatch.{event_type}
|
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsCloudwatchEventsPullerSetup(aws,aws#abc123,service-events-all#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsCloudwatchEventsPullerSetup(aws,aws#abc123,service-events-all#predefined,us-east-2) -> Creating user session
INFO InputProcess::AwsCloudwatchEventsPullerSetup(aws,aws#abc123,service-events-all#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudwatchEventsPullerSetup(aws,aws#abc123,service-events-all#predefined,us-east-2) -> Setup for module <AwsCloudwatchEventsPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Starting a new pulling from "dc-aws-cloudwatch-test-1.fifo" queue at "2022-09-23T07:44:54.589769+00:00"
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Received 198 response(s), received 1973 message(s), generated 1973 message(s), detected_event_types: ["ssm", "s3", "sts", "backup", "kms", "tag", "config", "logs", "cloudtrail"], avg_time_per_source_message: 335.170 ms
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Starting a new pulling from "dc-aws-cloudwatch-test-1.fifo" queue at "2022-09-23T07:55:55.546142+00:00"
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Received 1 response(s), received 0 message(s), generated 0 message(s), detected_event_types: [], avg_time_per_source_message: 437.862 ms
INFO InputProcess::AwsCloudwatchEventsPuller(aws,abc123,service-events-all,predefined,us-east-2) -> Data collection completed. Elapsed time: 0.438 seconds. Waiting for 59.562 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message: | Code Block |
|---|
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1655983326.290848): Number of requests performed: 2; Number of events received: 52; Number of duplicated events filtered out: 0; Number of events generated and sent: 52 (from 52 unflattened events); Average of events per second: 92.99414315733. |
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
...
Service events (
...
Security Hub)
This service reads Cloudtrail audit events via API.
There are two ways to read Cloudtrail events: via API or via S3+SQS.
API: It is slower, but can read past events.
S3+SQS: It is much faster, but can only read events since the creation of the queue.
This service makes use of the AWS API to get the data.
...
| title | Devo categorization and destination |
|---|
...
is used to read specifically Security Hub events, which need to be processed in a different way.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
Using this service, all the Security Hub events are going to be ingested into the table cloud.aws.cloudtrailsecurityhub.events. If auto_event_type parameter is set to true, the events are going to be ingested into the table cloud.aws.cloudtrail.{event_type}. findings
|
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsCloudtrailApiPullerSetupAwsCloudwatchEventsPullerSetup(aws,aws#abc123,auditservice-events-all#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsCloudtrailApiPullerSetupAwsCloudwatchEventsPullerSetup(aws,aws#abc123,auditservice-events-all#predefined,us-east-2) -> Creating user session
INFO InputProcess::AwsCloudtrailApiPullerSetupAwsCloudwatchEventsPullerSetup(aws,aws#abc123,auditservice-events-all#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudtrailApiPullerSetupAwsCloudwatchEventsPullerSetup(aws,aws#abc123,auditservice-events-all#predefined,us-east-2) -> Setup for module <AwsCloudtrailApiPuller><AwsCloudwatchEventsPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action.| Code Block |
|---|
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudwatchEventsPuller(aws,abc123,auditsqs-eventscloudwatch-allconsumer,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudwatchEventsPuller(aws,abc123,auditsqs-eventscloudwatch-allconsumer,predefined,us-east-2) -> Starting a new pulling from "['all_sources']cloudwatch-test.fifo" sourcequeue at "2022-09-23T08:5611:2250.366820440225+00:00"
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudwatchEventsPuller(aws,abc123,auditsqs-eventscloudwatch-allconsumer,predefined,us-east-2) -> UsingReceived 15 minutes as "gap until now", start_date: "2022-09-12T12:34:56.123456+00:00", end_date: "2022-09-23T08:41:22.366820+00:00", time_slot_in_hours: "1"1 response(s), received 0 message(s), generated 0 message(s), detected_event_types: [], avg_time_per_source_message: 519.301 ms
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudwatchEventsPuller(aws,abc123,auditsqs-eventscloudwatch-allconsumer,predefined,us-east-2) -> Total number ofData collection completed. Elapsed time: slots to be processed: 261
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Number of processed time slots so far: 100
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Number of processed time slots so far: 200
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Received 1315 response(s), messages (total/dropped/other_region/duplicated/generated): 124/6149/0/0/113, tag template used: "0.520 seconds. Waiting for 59.480 second(s) until the next one |
| Info |
|---|
The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
Audit events (via API)
This service reads Cloudtrail audit events via API.
There are two ways to read Cloudtrail events: via API or via S3+SQS.
API: It is slower, but can read past events.
S3+SQS: It is much faster, but can only read events since the creation of the queue.
This service makes use of the AWS API to get the data.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
If auto_event_type parameter is not set or is set to false, the events are going to be ingested into the table cloud.aws.cloudtrail.events. If auto_event_type parameter is set to true, the events are going to be ingested into the table cloud.aws.cloudtrail.{event_type}.
123456789012.us-east-8.1.prod-1", avg_time_per_source_message: 708.624 ms
|
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudtrailApiPullerSetup(aws,abc123aws#abc123,audit-events-allall#predefined,predefined,us-east-2) -> ElapsedSession time:cannot 931expire.842 seconds. Last retrieval took too much time, no wait will be applied in this loop
Using user/profile authentication.
INFO InputProcess::AwsCloudtrailApiPullerAwsCloudtrailApiPullerSetup(aws,abc123aws#abc123,audit-events-all,predefinedall#predefined,us-east-2) -> DataCreating collectionuser completed.session
ElapsedINFO time: 2.717 seconds. Waiting for 57.283 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message:InputProcess::AwsCloudtrailApiPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudtrailApiPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Setup for module <AwsCloudtrailApiPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> ReceivedStarting 4 response(s), messages (total/dropped/other_region/duplicated/generated): 186/8/0/1/177, tag template used: "cloud.aws.cloudtrail.{event_type}.123456789012.us-west-8.1.prod-1", avg_time_per_source_message: 678.952 ms |
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
|
Audit events (via S3 + SQS)
This service reads Cloudtrail audit events via the S3+SQS pipeline.
There are two ways to read Cloudtrail events: via API or via S3+SQS.
API: It is slower, but can read past events.
S3+SQS: It is much faster, but can only read events since the creation of the queue.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
If auto_event_type parameter is not set or is set to false, the events are going to be ingested into the table cloud.aws.cloudtrail.events. If auto_event_type parameter is set to true, the events are going to be ingested into the table cloud.aws.cloudtrail.{event_type}. |
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: |
Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
| Code Block |
|---|
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefineddata collection every 60 seconds
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Starting a new pulling from "['all_sources']" source at "2022-09-23T08:56:22.366820+00:00"
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Using 15 minutes as "gap until now", start_date: "2022-09-12T12:34:56.123456+00:00", end_date: "2022-09-23T08:41:22.366820+00:00", time_slot_in_hours: "1"
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Total number of time slots to be processed: 261
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Number of processed time slots so far: 100
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Number of processed time slots so far: 200
...
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Creating user sessionReceived 1315 response(s), messages (total/dropped/other_region/duplicated/generated): 124/6149/0/0/113, tag template used: "cloud.aws.cloudtrail.{event_type}.123456789012.us-east-8.1.prod-1", avg_time_per_source_message: 708.624 ms
INFO InputProcess::AwsSqsS3GenericPullerSetupAwsCloudtrailApiPuller(aws,aws#abc123abc123,audit-events-all#predefinedall,predefined,us-east-2) -> NewElapsed AWS session started.
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Setup for module <AwsSqsS3CloudTrailPuller> has been successfully executed |
Puller output
A successful initial run has the following output messages for the puller module:
| Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
time: 931.842 seconds. Last retrieval took too much time, no wait will be applied in this loop
INFO InputProcess::AwsSqsS3CloudTrailPullerAwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Data Startingcollection datacompleted. collectionElapsed every 60time: 2.717 seconds. INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1797, total_bytes: 3830368 (60.43562 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0Waiting for 57.283 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
| Code Block |
|---|
INFO InputProcess::AwsCloudtrailApiPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> ConsumedReceived messages: 17974 response(s), messages (60.436958 seconds) => 29 msg/sec
INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1652, total_bytes: 3555837 (60.311803 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Consumed messages: 1652 messages (60.313064 seconds) => 27 msg/sec
INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1949, total_bytes: 4277470 (60.187779 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Consumed messages: 1949 messages (60.187248 seconds) => 32 msg/sec
... |
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
| Info |
|---|
The @devo_pulling_id value is injected into each event to allow grouping total/dropped/other_region/duplicated/generated): 186/8/0/1/177, tag template used: "cloud.aws.cloudtrail.{event_type}.123456789012.us-west-8.1.prod-1", avg_time_per_source_message: 678.952 ms |
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded on in that Pull action in LoxcopeDevo’s search window. |
...
Audit events (via S3 + SQS)
This service could be considered a general AWS metric puller. It reads metrics from all the AWS services that generate them. Those metrics are also managed by Cloudwatch.This service makes use of the AWS API to get the datareads Cloudtrail audit events via the S3+SQS pipeline.
There are two ways to read Cloudtrail events: via API or via S3+SQS.
API: It is slower, but can read past events.
S3+SQS: It is much faster, but can only read events since the creation of the queue.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
| All cloudwatchmetrics. |
| Expand |
|---|
| title | Verify data collection |
|---|
|
| Once the |
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and |
managing the token expiration when needed.Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. | Setup outputA successful managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Creating user session
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#abc123,audit-events-all#predefined,us-east-2) -> Setup for module <AwsSqsS3CloudTrailPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsSqsS3CloudTrailPuller(aws,abc123,audit-events-all,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1797, total_bytes: 3830368 (60.43562 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Consumed messages: 1797 messages (60.436958 seconds) => 29 msg/sec
INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1652, total_bytes: 3555837 (60.311803 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Consumed messages: 1652 messages (60.313064 seconds) => 27 msg/sec
INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 1949, total_bytes: 4277470 (60.187779 seconds)
INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Consumed messages: 1949 messages (60.187248 seconds) => 32 msg/sec
... |
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message: | Info |
|---|
The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
Metrics (All metrics)
This service could be considered a general AWS metric puller. It reads metrics from all the AWS services that generate them. Those metrics are also managed by Cloudwatch.
This service makes use of the AWS API to get the data.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
All the events are going to be ingested into the table cloud.aws.cloudwatch.metrics. |
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> Creating user session
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> Setup for module <AwsCloudwatchMetricPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Starting a new pulling from "['AWS/EC2', 'AWS/EC2Spot']" namespaces at "2022-09-23T14:49:36.266007+00:00"
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Time range: "2022-09-23T14:48:00Z" > "2022-09-23T14:49:00Z"
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Received 3 response(s), generated 17 message(s), tag used: "cloud.aws.cloudwatch.metrics.936082584952.us-east-2.1", avg_time_per_source_message: 393.845 ms
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Applied an offset to wait, retrieval_offset: -36.266007 seconds
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Data collection completed. Elapsed time: 1.182 seconds. Waiting for 22.552 second(s) until the next one |
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message: | Info |
|---|
The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in the Data Search area of Devo. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
AWS-GuardDuty (Via API)
This service reads GuardDuty events via API. This service is not scalable because of it use of GuardDuty APIs. We use this service for “low” data due to the API limitation, otherwise we should use AWS_SQS_IF
This service makes use of the AWS API to get the data only if data is low
The events are going to be ingested into the table cloud.aws.guardduty.findings
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
2024-052024-05-31T18:40:27.031 INFO InputProcess::AwsGuardDutyApiPullerSetup(aws,aws#121214,aws-guardduty#predefined,ap-southeast-1) -> Session cannot expire. Using user/profile authentication.
2024-05-31T18:40:28.486 INFO InputProcess::AwsGuardDutyApiPullerSetup(aws,aws#121214,aws-guardduty#predefined,ap-southeast-1) -> Creating user session
2024-05-31T18:40:28.487 INFO InputProcess::AwsGuardDutyApiPullerSetup(aws,aws#121214,aws-guardduty#predefined,ap-southeast-1) -> New AWS session started.
2024-05-31T18:40:29.779 INFO InputProcess::AwsGuardDutyApiPullerSetup(aws,aws#121214,aws-guardduty#predefined,ap-southeast-1) -> Setup for module <AwsGuardDutyApiPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the setup puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.2024-05-31T13:12:08.535895316Z 2024-05-31T13:12:08.535 INFO InputProcess::AwsCloudwatchMetricPullerSetupAwsGuardDutyApiPuller(aws,aws#123,ec2#predefined,us-east-21023001,aws-guardduty,predefined,ap-southeast-1) -> Starting data Creatingcollection userevery session60 INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudwatchMetricPullerSetup(aws,aws#123,ec2#predefined,us-east-2seconds
2024-05-31T13:12:11.648725368Z 2024-05-31T13:12:11.648 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> SetupCreated fora module <AwsCloudwatchMetricPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Starting a new pulling from "['AWS/EC2', 'AWS/EC2Spot']" namespaces at "2022-09-23T14:49:36.266007+00:00"
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Time range: "2022-09-23T14:48:00Z" > "2022-09-23T14:49:00Z"
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Received 3 response(s), generated 17 message(s), tag used: "cloud.aws.cloudwatch.metrics.936082584952.us-east-2.1", avg_time_per_source_message: 393.845 ms
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2) -> Applied an offset to wait, retrieval_offset: -36.266007 seconds
INFO InputProcess::AwsCloudwatchMetricPuller(aws,123,ec2,predefined,us-east-2sender: {"name": "DevoSender(standard_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/etc/devo/keys/ca.d/chain.crt", "cert_path": "/etc/devo/keys/devo.crt", "key_path": "/etc/devo/keys/devo.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "collector-723d1da04d3a87c4-6d89bfcf9-6xzms", session_id: "139892643371760"
2024-05-31T13:12:22.238716958Z 2024-05-31T13:12:22.229 INFO InputProcess::AwsGuardDutyApiPuller(aws,1023001,aws-guardduty,predefined,ap-southeast-1) -> Total event received: 584, and sent: 584
2024-05-31T13:12:22.238732449Z 2024-05-31T13:12:22.230 INFO InputProcess::AwsGuardDutyApiPuller(aws,1023001,aws-guardduty,predefined,ap-southeast-1) -> Data collection completed. Elapsed time: 13.697 seconds. Waiting for 46.303 second(s) until the next on |
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message: | Code Block |
|---|
2024-05-31T13:12:22.238716958Z 2024-05-31T13:12:22.229 INFO InputProcess::AwsGuardDutyApiPuller(aws,1023001,aws-guardduty,predefined,ap-southeast-1) -> Total event received: 584, and sent: 584
2024-05-31T13:12:22.238732449Z 2024-05-31T13:12:22.230 INFO InputProcess::AwsGuardDutyApiPuller(aws,1023001,aws-guardduty,predefined,ap-southeast-1) -> Data collection completed. Elapsed time: 113.182697 seconds. Waiting for 2246.552303 second(s) until the next one |
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message: | Info |
|---|
The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcopethe Data Search area of Devo. |
|
Non Cloudwatch Logs
This service reads logs from some AWS services, but those logs are not managed by Cloudwatch. These logs are stored in an S3 bucket and read through an SQS queue, so it is using an S3+SQS pipeline.
...
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#123,non-cloudwatch-logs#predefined,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#123,non-cloudwatch-logs#predefined,us-east-2) -> Creating user session
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#123,non-cloudwatch-logs#predefined,us-east-2) -> New AWS session started. |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsSqsS3GenericPullerSetup(aws,aws#123,non-cloudwatch-logs#predefined,us-east-2) -> Setup for module <AwsSqsS3VpcFlowlogsPuller> has been successfully executed
INFO InputProcess::AwsSqsS3VpcFlowlogsPuller(aws,123,non-cloudwatch-logs,predefined,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsSqsS3VpcFlowlogsPuller(aws,123,non-cloudwatch-logs,predefined,us-east-2) -> Received 2 response(s), messages (fromSQS/generated): 0/0, discarded files: 0, avg_time_per_source_message: 169.711 ms
INFO InputProcess::AwsSqsS3VpcFlowlogsPuller(aws,123,non-cloudwatch-logs,predefined,us-east-2) -> Data collection completed. Elapsed time: 0.340 seconds. Waiting for 59.660 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message: | Code Block |
|---|
INFO InputProcess::AwsSqsS3VpcFlowlogsPuller(aws,123,non-cloudwatch-logs,predefined,us-east-2) -> Received 2 response(s), messages (fromSQS/generated): 0/0, discarded files: 0, avg_time_per_source_message: 169.711 ms |
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
Custom Logs
This service reads logs from some AWS services and these logs are managed by Cloudwatch. Cloudwatch creates log groups to store the different log sources, so it is required to use a custom puller in order to read from different log groups at the same time. This service makes use of the AWS API to get the data.
...
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::AwsCloudwatchLogsPullerSetup(aws,aws#123,cwl_1#custom,us-east-2) -> Session cannot expire. Using user/profile authentication.
INFO InputProcess::AwsCloudwatchLogsPullerSetup(aws,aws#123,cwl_1#custom,us-east-2) -> Creating user session
INFO InputProcess::AwsCloudwatchLogsPullerSetup(aws,aws#123,cwl_1#custom,us-east-2) -> New AWS session started.
INFO InputProcess::AwsCloudwatchLogsPullerSetup(aws,aws#123,cwl_1#custom,us-east-2) -> Setup for module <AwsCloudwatchLogsPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::AwsCloudwatchLogsPuller(aws,123,cwl_1,custom,us-east-2) -> Starting data collection every 60 seconds
INFO InputProcess::AwsCloudwatchLogsPuller(aws,123,cwl_1,custom,us-east-2) -> Starting a new pulling from "/aws/events/devo-cloudwatch-test-1" at "2022-09-23T15:08:18.132865+00:00"
INFO InputProcess::AwsCloudwatchLogsPuller(aws,123,cwl_1,custom,us-east-2) -> Optimized first retrieval approach for high number of log streams with medium size |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
Cisco Umbrella (via S3+SQS)
This service reads logs from a Cisco Umbrella managed bucket via the S3+SQS pipeline. Cisco provides a way to deposit logging data into a S3 bucket.
| Expand |
|---|
| title | Devo categorization and destination |
|---|
|
There are three types of events: dnslogs, iplogs and proxylogs. Cisco stores them in different paths depending on the event type. The collector will ingest them towards the following tables: dnslogs: sig.cisco.umbrella.dns iplogs: sig.cisco.umbrella.ip proxylogs: sig.cisco.umbrella.proxy In case Cisco starts sending other type of events to s3, they will go to: sig.cisco.umbrella.unknown
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector does not use any kind of persistent storage. |
Collector operations
This section is intended to explain how to proceed with the specific operations of this collector.
...
