Document toolboxDocument toolbox

CrowdStrike API resources collector

Owned by Juan Tomás Alonso Nieto (Deactivated)
Last updated: Nov 25, 2024 by Borja Moro Moreno
21 min read
[ 1 Overview ] [ 2 Devo Collector Features ] [ 3 Data source description ] [ 4 Accepted Authentication Methods ] [ 5 Vendor setup ] [ 6 Run the collector ] [ 7 Collector services detail ] [ 8 Troubleshooting ] [ 9 Collector operations ] [ 10 Change log ] [ 11 Version migration ]

Overview

The CrowdStrike Falcon platform is a powerful solution that includes EDR (Endpoint Detection and Response), next-generation anti-virus, and device control for endpoints. It also provides a whole host of other operational capabilities across IT operations and security including Threat Intelligence.

The CrowdStrike API is managed from the CrowdStrike Falcon UI by the Falcon Administrator. From there, multiple API clients can be defined along with their required scope.

Devo Collector Features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

  • Not allowed

Running environments

  • Collector Server

  • On Premise

Populated Devo events

  • Table

Flattening pre-processing

  • No

Allowed source events obfuscation

  • No

Data source description

Available from v1.0.0

Data Source

Subtype

Service

Table

Data Source

Subtype

Service

Table

Hosts

-

hosts

edr.crowdstrike.falconstreaming.agents

Description

Hosts are endpoints that run the Falcon sensor. You can get information and details about these agents.

End point

  1. Listing: {base_url}/devices/queries/devices/v1

  2. Details: {base_url}/devices/entities/devices/v2

Check the {base_url} in the config parameters details for further information.

Incidents

-

incidents

edr.crowdstrike.falconstreaming.incidents

Description

Incidents are events that occur in an organization which can represent a cybersecurity threat or an attack.

End point

  1. Listing: {base_url}/incidents/queries/incidents/v1

  2. Details: {base_url}/incidents/entities/incidents/GET/v1

Check the {base_url} in the config parameters details for further information.

Spotlight
Vulnerabilities

-

vulnerabilities

  • table: edr.crowdstrike.falconstreaming.vulnerabilities

  • alias: edr.crowdstrike.falcon_spotlight.vulnerabilities

Description

Vulnerabilities are known security risks in an operating system, application, hardware, firmware, or other part of a computing stack.

End point

  1. Listing: {base_url}/spotlight/queries/vulnerabilities/v1

  2. Details: {base_url}/spotlight/entities/vulnerabilities/v2

Check the {base_url} in the config parameters details for further information.

Behaviors

-

behaviors

edr.crowdstrike.falconstreaming.behaviors

Description

Behaviors are patterns of data transmissions in a network that are out of the norm, used to detect anomalies before cyber attacks occur.

End point

  1. Listing: {base_url}/incidents/queries/behaviors/v1

  2. Details: {base_url}/incidents/entities/behaviors/GET/v1

Check the {base_url} in the config parameters details for further information.

File Vantage

-

filevantage

edr.crowdstrike.falcon_filevantage.change

Description

Collect data about changes to files, folders, and registries with Falcon FileVantage APIs. Store this data to help you meet certain compliance recommendations and requirements as listed in the Sarbanes-Oxley Act, National Institute for Standards and Technology (NIST), Health Insurance Portability and Accountability Act (HIPAA), and others.

End point

  1. Listing: {base_url}/filevantage/queries/changes/v2

  2. Details: {base_url}/filevantage/entities/changes/v21

Check the {base_url} in the config parameters details for further information.

For more information on how the events are parsed, visit our page.

Available from v1.3.0

Data Source

Subtype

Service

Table

Data Source

Subtype

Service

Table

Event Stream (eStream)

AuthActivity AuditEvent

estream

edr.crowdstrike.falconstreaming.auth_activity

IncidentSummaryEvent

estream

edr.crowdstrike.falconstreaming.incident_summary

RemoteResponseSessionStartEvent RemoteResponseSessionEndEvent

estream

edr.crowdstrike.falconstreaming.remote_response_session

CustomerIOCEvent

estream

edr.crowdstrike.falconstreaming.customer_ioc

Event_ExternalAPIEvent

estream

edr.crowdstrike.falconstreaming.external_api

DetectionSummaryEvent

deprecated by crowdstrike

estream

edr.crowdstrike.falconstreaming.detection_summary

use epp detection summary See v1.11.0

UserActivityAuditEvent

estream

Depending on the event's event.ServiceName property (in lowercase):

  • groups → edr.crowdstrike.falconstreaming.user_activity_groups

  • devices → edr.crowdstrike.falconstreaming.user_activity_devices

  • detections → edr.crowdstrike.falconstreaming.user_activity_detections

  • quarantined_files → edr.crowdstrike.falconstreaming.user_activity_quarantined_files

  • ip_whitelist → edr.crowdstrike.falconstreaming.user_activity_ip_whitelist

  • prevention_policy → edr.crowdstrike.falconstreaming.user_activity_prevention_policy

  • sensor_update_policy → edr.crowdstrike.falconstreaming.user_activity_sensor_update_policy

  • device_control_policy → edr.crowdstrike.falconstreaming.user_activity_device_control_policy

Description

The Streaming API provides several types of events.

End point

The endpoints are dynamically generated by following this (simplified) approach:

  1. Once an authentication token has been obtained, a request to {base_url}/sensors/entities/datafeed/v2 is performed to obtain the "Data Feeds".

    1. Check the {base_url} in the config parameters details for further information.

  2. Each Data Feed will contain a URL and a session token. A request to each of these URLs (along with their corresponding token) will return a streaming response in which every non-empty line represents a different event.

    1. Every Data Feed will also contain a "refresh stream" URL, which is accessed every less than 30 minutes.

    2. All the Data Feeds are processed in parallel. The amount of available Data Feeds depend on the CrowdStrike account's configuration.

For more information on how the events are parsed, visit our page.

Available from v1.10.0

Data Source

Subtype

Service

Table

Data Source

Subtype

Service

Table

Alerts

-

alerts

edr.crowdstrike.falconstreaming.alert

Description

Alerts are events that occur in an organization which can represent a cybersecurity threat or an attack.

End point

  1. Listing: {base_url}/alerts/queries/alerts/v2

  2. Details: {base_url}/alerts/entities/alerts/GET/v2
    Check the {base_url} in the config parameters details for further information.

For more information on how the events are parsed, visit our page.

Available from v1.11.0

Data Source

Subtype

Service

Table

Data Source

Subtype

Service

Table

Event Stream (eStream)

EPPDetectionSummaryEvent

estream

edr.crowdstrike.falconstreaming.epp_detection_summary

Description

Alerts are events that occur in an organization which can represent a cybersecurity threat or an attack.

End point

The endpoints are dynamically generated by following this (simplified) approach:

  1. Once an authentication token has been obtained, a request to {base_url}/sensors/entities/datafeed/v2 is performed to obtain the "Data Feeds".

    1. Check the {base_url} in the config parameters details for further information.

  2. Each Data Feed will contain a URL and a session token. A request to each of these URLs (along with their corresponding token) will return a streaming response in which every non-empty line represents a different event.

    1. Every Data Feed will also contain a "refresh stream" URL, which is accessed every less than 30 minutes.

    2. All the Data Feeds are processed in parallel. The amount of available Data Feeds depend on the CrowdStrike account's configuration.

For more information on how the events are parsed, visit our page.

Accepted Authentication Methods

Authentication method

Details

Authentication method

Details

user/pass

You will need your client_id_value, which acts as a user, and secret_key_value, which acts as a password, to connect to the API and execute the API request.

Treat Your Secret Key Like A Password

The security of your application is tied to the security of your secret key. Secure it as you would any sensitive credential. Don't share it with unauthorized individuals or email it to anyone under any circumstances.

Vendor setup

In order to configure the Devo | CrowdStrike API Resources collector, you need to create an API client that will be used to authenticate API requests.

  1. After getting your CrowdStrike Falcon Cloud credentials, log into the CrowdStrike Falcon Cloud dashboard.

  2. Click the three dots in the left menu bar.

  3. Click API Clients and Keys. This will open a page to create an API client.

  1. Click Add API Client at the top right corner. Enter a CLIENT NAME and DESCRIPTION

  1. Then, enable the API scopes for your new API client. Click the required Read permissions for each scope and click ADD to create the client.

  1. Finally, copy the Client ID and Client Secret shown on the next screen. You will need these values to configure the collector.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Please check the section Data Source Description to learn about the target tables for each service.

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the reset_persistence_auth_value to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file.

Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Configuration errors

Error Type

Error Id

Error Message

Cause

Solution

Error Type

Error Id

Error Message

Cause

Solution

InitVariablesError

1-2

Invalid content detected in the configuration

The module_properties setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

3-5

Invalid content detected in the configuration

The base_url setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

6-7

Invalid content detected in the configuration

The override_base_url setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

8-9

Invalid content detected in the configuration

The base_tag setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

10-11

Invalid content detected in the configuration

The user_agent setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

12-13

Invalid content detected in the configuration

The endpoint setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

14-15

Invalid content detected in the configuration

The auth setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

16-17

Invalid content detected in the configuration

The event_list setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

18-19

Invalid content detected in the configuration

The details settings need to have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

20-22

Invalid content detected in the configuration

The logs_limit_in_items setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

23-24

Invalid content detected in the configuration

The credentials setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

25-26

Invalid content detected in the configuration

The client_id setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

27-28

Invalid content detected in the configuration

The secret_key setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

29-31

Invalid content detected in the configuration

The start_timestamp_in_epoch_seconds setting does not have the right format.

Check the documentation and update the configuration accordingly

InitVariablesError

32-33

Invalid content detected in the configuration

The unique_identifier setting does not have the right format.

Check the documentation and update the configuration accordingly

SetupError

100

Required credentials are invalid

Required credentials are invalid

Include the proper credentials in the configuration

SetupError

101

Service not found

A declared service is not valid

Include the proper service name in the configuration

SetupError

102-103

The token has no access

The generated token cannot access a service list.

Enable the service in the Crowdstrike configuration

SetupError

104-105

The token has no access

The generated token cannot access service details.

Enable the service in the Crowdstrike configuration

Runtime errors

Error Type

Error Id

Error Message

Cause

Solution

Error Type

Error Id

Error Message

Cause

Solution

PrePullError

200

Error before pulling data

The start time is is newer than the current date

Update the configuration

PullError

300-312

Error pulling data

Error pulling data from the service

Review the error and act accordingly if required.

ApiError

400-403

API error

The API returned an error

Review the error and act accordingly if required.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module:

2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null} 2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json" 2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> "\etc\devo\job" does not exists 2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json" 2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> "\etc\devo\collector" does not exists 2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "C:\git\collectors2\devo-collector-<name>\config\config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False} 2023-01-10T15:22:57.171 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: False

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:

2023-01-10T15:23:00.788 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread 2023-01-10T15:23:00.789 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds) 2023-01-10T15:23:00.790 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread 2023-01-10T15:23:00.842 INFO OutputProcess::MainThread -> global_status: {"output_process": {"process_id": 18804, "process_status": "running", "thread_counter": 21, "thread_names": ["MainThread", "pydevd.Writer", "pydevd.Reader", "pydevd.CommandThread", "pydevd.CheckAliveThread", "DevoSender(standard_senders,devo_sender_0)", "DevoSenderManagerMonitor(standard_senders,devo_1)", "DevoSenderManager(standard_senders,manager,devo_1)", "OutputStandardConsumer(standard_senders_consumer_0)",

Sender services

The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:

Logging trace

Description

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

Sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:

  • 44 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.

  • 21 events were sent to Devo between the last UTC checkpoint and now.

  • Those 21 events required 0.007 seconds to be delivered.

    By default these traces will be shown every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service

Sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds

Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:

  • 44 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2023-01-10 16:09:16.116750+00:00.

  • 21 events were sent to Devo between the last UTC checkpoint and now.

  • Those 21 events required 0.00 seconds to be delivered.

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB) INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)

Change log

Release

Released on

Release type

Recommendations

Release

Released on

Release type

Recommendations

v1.11.0

Oct 28, 2024

New FEATUREs
IMPROVEMENTS

Recommended version

v1.10.0

Oct 11, 2024

New FEATUREs

Upgrade

v1.9.1

Sep 26, 2024

IMPROVEMENTS

Upgrade

v1.9.0

Sep 16, 2024

IMPROVEMENTS

Upgrade

v1.8.0

Jul 24, 2024

IMPROVEMENTS
BUG FIXING

Upgrade

v1.7.0

May 17, 2024

IMPROVEMENTS
BUG FIXING

Upgrade

v1.6.0

Mar 26, 2024

IMPROVEMENTS

Upgrade

v1.4.3

Nov 27, 2023

IMPROVEMENTS

Upgrade

v1.4.2

Jan 20, 2023

IMPROVEMENTS

Upgrade

v1.4.0

Sep 15, 2022

IMPROVEMENTS
BUG FIXING

Upgrade

v1.3.1

Sep 15, 2022

IMPROVEMENTS

Upgrade

 

v1.3.0

Sep 9, 2022

IMPROVEMENTS
New FEATUREs

Upgrade

 

v1.2.0

Jul 7, 2022

IMPROVEMENTS

Upgrade

v1.1.0

Apr 8, 2022

IMPROVEMENTS
VULNS

Upgrade

v1.0.0

Dec 16, 2021

New FEATUREs

-

Version migration