/
Firewalla MSP collector

Firewalla MSP collector

Overview

Firewalla MSP is a Managed Security Portal designed for security and infosec professionals to easily manage multiple Firewalla boxes from anywhere.

Devo collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

not allowed

Running environments

collector server

Populated Devo events

table

Flattening preprocessing

no

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Device

Get list of device

/v1/device/list

devices

firewall.firewalla.msp.device

v1.0.0

Device detail

Get device details

/v1/device/{gid}/{mac}

devices

firewall.firewalla.msp.device_detail

v1.0.0

Flows

Get flows of the firewall MSP

/v2/flows

flows

firewall.firewalla.msp.flow

v1.0.0

Alarm

Get list of alarm

/v2/alarms

alarms

firewall.firewalla.msp.alarm

v1.0.0

Alarm Detail

Get Alarm details

/v2/alarms/{gid}/{aid}

alarms

firewall.firewalla.msp.alarm_detail

v1.0.0

Box

Get the list of boxes

/v2/boxes

boxes

firewall.firewalla.msp.box

v1.0.0

For more information on how the events are parsed, visit our page.

Flattening preprocessing

Data source

Collector service

Optional

Flattening details

Device

devices

yes

not required

Device detail

devices

yes

not required

Flows

flows

yes

not required

Alarm

alarms

yes

not required

Alarm Detail

alarms

yes

not required

Box

boxes

yes

not required

Device

devices

yes

not required

Accepted Authentication Methods

Authentication method

api_token

msp_domain

Authentication method

api_token

msp_domain

api_token

REQUIRED

REQUIRED

msp_domain

REQUIRED

REQUIRED



Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

api_token

The api_token for firewalla_msp API

msp_domain

The msp_domain for firewalla_msp API. ex: https://{msp_domain}.firewalla.net

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

This collector can only make 30 requests per hour for each service.

Common Setup outpur for all services

A successful run has the following output messages for the setup module:

2024-12-17T15:56:52.446 INFO InputProcess::MainThread -> FirewallaMSPPullerSetup(firewalla_msp#123451,devices#predefined) -> Starting thread 2024-12-17T15:56:52.447 INFO InputProcess::MainThread -> FirewallaMSPDevicePuller(firewalla_msp#123451,devices#predefined) - Starting thread 2024-12-17T15:56:52.447 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-12-17T15:56:52.447 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-firewalla-msp/state/df8895fef2a509cbd87fcc9850dc0c81" 2024-12-17T15:56:52.451 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-firewalla-msp/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz 2024-12-17T15:56:52.452 WARNING InputProcess::FirewallaMSPDevicePuller(firewalla_msp#123451,devices#predefined) -> Waiting until setup will be executed 2024-12-17T15:56:52.454 INFO InputProcess::MainThread -> InputMetricsThread -> Started thread for updating metrics values (update_period=10.0) 2024-12-17T15:56:52.456 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-12-17T15:56:52.456 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-firewalla-msp/state/865a79c1b99ad39b22becc235c9732cb" 2024-12-17T15:56:52.461 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-firewalla-msp/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz 2024-12-17T15:56:52.466 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-12-17T15:56:52.466 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-firewalla-msp/state/e784c94a6fe28de5937e50674407d82e" 2024-12-17T15:56:52.477 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-firewalla-msp/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz 2024-12-17T15:56:52.480 INFO InputProcess::MainThread -> [GC] global: 26.6% -> 26.6%, process: RSS(74.55MiB -> 75.05MiB), VMS(825.46MiB -> 825.46MiB) 2024-12-17T15:56:52.481 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "vendor_requests" created: "Number of requests received from the vendor API", unit: "requests" 2024-12-17T15:56:52.482 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-12-17T15:56:52.482 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_received" created: "Number of messages received from the vendor API", unit: "1" 2024-12-17T15:56:52.482 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-firewalla-msp/state/4ff7b345dc444ac050cf75f93e5dcb3b" 2024-12-17T15:56:52.483 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_removed" created: "Number of messages removed by the collector", unit: "1" 2024-12-17T15:56:52.483 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_filtered" created: "Number of messages filtered by the collector", unit: "1" 2024-12-17T15:56:52.483 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_counter" created: "Number of messages enqueued", unit: "1" 2024-12-17T15:56:52.484 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_bytes" created: "Number of bytes enqueued", unit: "1" 2024-12-17T15:56:52.484 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_counter" created: "Number of messages enqueued", unit: "1" 2024-12-17T15:56:52.484 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_bytes" created: "Number of messages enqueued", unit: "1" 2024-12-17T15:56:52.484 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_counter" created: "Number of messages enqueued in the queue", unit: "1" 2024-12-17T15:56:52.485 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_bytes" created: "Number of messages enqueued in the queue", unit: "1" 2024-12-17T15:56:52.485 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Gauge "module_global_status" created: "Global status of current module", unit: "1" 2024-12-17T15:56:52.487 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-firewalla-msp/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz 2024-12-17T15:56:52.491 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-12-17T15:56:52.491 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-firewalla-msp/state/10dd360c86621afd5a28a029a0dddcf6" 2024-12-17T15:56:52.493 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread 2024-12-17T15:56:52.494 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds) 2024-12-17T15:56:52.495 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread 2024-12-17T15:56:52.495 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.496 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.496 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread 2024-12-17T15:56:52.498 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds) 2024-12-17T15:56:52.500 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread 2024-12-17T15:56:52.502 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.503 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.504 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.504 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-12-17T15:56:52.504 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread 2024-12-17T15:56:52.505 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.506 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.506 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-12-17T15:56:52.508 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.508 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds) 2024-12-17T15:56:52.508 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-12-17T15:56:52.508 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-12-17T15:56:52.509 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread 2024-12-17T15:56:52.509 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.510 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-17T15:56:52.514 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.514 INFO OutputProcess::MainThread -> OutputMetricsThread -> Started thread for updating metrics values (update_period=10.0) 2024-12-17T15:56:52.515 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-12-17T15:56:52.515 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:52.517 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-12-17T15:56:52.520 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_counter" created: "Number of messages sent to the defined output", unit: "1" 2024-12-17T15:56:52.551 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_bytes" created: "Number of bytes sent to the defined output", unit: "1" 2024-12-17T15:56:52.552 INFO OutputProcess::MainThread -> [GC] global: 26.7% -> 26.8%, process: RSS(74.80MiB -> 75.18MiB), VMS(1.37GiB -> 1.37GiB) 2024-12-17T15:56:53.207 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-firewalla-msp/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-firewalla-msp/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-firewalla-msp/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "137769423454016" 2024-12-17T15:56:53.209 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-17T15:56:53.635 INFO InputProcess::FirewallaMSPPullerSetup(firewalla_msp#123451,devices#predefined) -> Setup for module <FirewallaMSPDevicePuller> has been successfully executed


Boxes

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-12-17T16:08:10.210 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> Pull Started 2024-12-17T16:08:11.507 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1734431890199):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 1.543. 2024-12-17T16:08:11.508 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> Sent 2 events to Devo 2024-12-17T16:08:11.508 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1734431890199):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 1.541. 2024-12-17T16:08:11.509 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1734431890199):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 1.541. 2024-12-17T16:08:11.509 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> The data is up to date!

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-12-17T16:08:11.509 INFO InputProcess::FirewallaMSPBoxPuller(firewalla_msp#123451,boxes#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1734431890199):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 1.541.

devices

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

flows

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

alarms

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Restart the persistance

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the start_time_in_utc parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.


Note: If the error is Failed to establish a new connection: [Errno 110] Connection timed out please check the credentials if they are provided correctly.

Error type

Error ID

Error message

Cause

Solution

InitVariablesError

1

"start_time_in_utc" setting from "services" section should be in the following format - YYYY-MM-DDThh:mm:ssZ

param 'start_time_in_utc' is not in correct format

start_time_in_utc should be in correct format

InitVariablesError

2

"start_time_in_utc" {start_time} is in the future

param 'start_time_in_utc' is not in correct format

start_time_in_utc should be current time or in the past

InitVariablesError

3

start_time_in_utc" is not defined. It should be defined

param 'start_time_in_utc' is not defined

start_time_in_utc should be defined

SetupError

100

HTTP Error occurred while checking the server health for firewalla_msp: {error message}

Credentials error {unauthorized} or mismatch b/w credentials or firewalla_msp server error

Check the credentials or contact developer with required message

SetupError

101

Some error occurred while checking the server health for firewalla_msp. Error detailsl {e}.

Error on the firewalla msp server.

Contact the developer with exact error message.

PullError

300

Error related to HTTP, occurred while retrieving events from firewalla_msp server{summery} , {details}

This error happens when the collector tries to fetch the data from API.

In this error you will find the HTTP error code as well as the summary and details.

PullError

301

Error while checking the server health for firewalla_msp. Error: {details}

Some exceptions occurred while making the API request.

Contact the developer with exact error message.

PullError

302

HTTP Error occurred while checking the server health for firewalla_msp: {error message}

Error while fetching sender_ip_address data

Get the HTTP error code and contact the developer

PullError

303

Some error occurred while checking the server health for firewalla_msp. Error details {e}

Exception while making the API request sender_ip_address

In this error you will find the HTTP error code as well as the summary and details.

PullError

304

HTTP Error occurred while checking the server health for firewalla_msp: {error message}

Error while fetching sender_data data

Contact the developer with required message

PullError

305

Some error occurred while checking the server health for firewalla_msp. Error details {e}

Some exceptions occurred while making the API request.

Contact the developer with required message

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Change log

Release

Released on

Release type

Details

Recommendations

Release

Released on

Release type

Details

Recommendations

v1.0.0

Jan 9, 2025

NEW Feature



Initial Release

Recommended Version