Execute Playbooks with Webhook

As an alternative to setting up playbooks to run on a schedule, you can schedule a playbook to run when an external application, such as Splunk, makes a POST request to a Webhook URL.
You can also supply additional data in a request body when making requests to the Webhook address. The playbook then has access to the supplied data through the loadEventsFromExecutionContext() operator.

Set Up a Stream for a Webhook Trigger

To set up a playbook for a Webhook trigger:

1. Edit an existing stream.

2. On the Edit Stream page, edit the stream name and batch length and select the On demand with webhook option.

3. The Webhook URL will be displayed. Copy the URL and save it.

4. Click Save.

The Webhook URL is now available for use. To execute the stream, a POST call must be made to the Webhook URL. The following is an example curl command that includes a POST call to the Webhook URL.

The command output includes a request ID.

Example response with the requestId

The request ID can be used to fetch the state of the stream trigger using the following API call.

The following is an example response when the stream is scheduled and batch is created.

``` {json}{ "result": { "requestId": "27f5f7ee-74ca-42e4-a226-81ecc0580fe2", "expiryAt": 1569149800083, "createdAt": 1569148300083, "batchId":"batch-1822643", "state": "scheduled" } }

If the stream cannot be scheduled due to the expiration limit, the stream is timed out (timedout). In this case, the client must retry to trigger the stream.

Example response when the invocation rate limit is exceeded

When the stream is scheduled, the following API call can be used to see the stream execution status. Use the batchId from the previous stream request API response.

The following is a sample response.

Sample response

When stream execution results are available, the status transitions to ready. The following is an example with the ready status.

Sample response

To obtain the remaining results, fetch the next pages.

The following table lists the status values.

Query Parameters to the Webhook

The optional intervalStart and intervalEnd parameters provide the time range for stream execution. The following rules apply to the time range settings.

• intervalStart and intervalEnd times must be given in Unix epoch time in milliseconds.

• If both a start and end time are provided, both are used.

• If only a start time is provided, the end time is the start time plus the batch Interval time.

• If only the end time is provided, the start time is the end time minus the batch Interval time.

• If no start or end time is provided, the start time is the current time minus the batch Interval time and the end time is the current time.

curl -XPOST --header "Content-Type: application/json" -d '{}' https://xyz.logichub.com/api/trigger/stream-5234/717e995a-6fdd-42b7-a2c2-ce3e9e613d63?intervalStart=1570468818221&intervalEnd=1570468848221

Supplying Additional Data to the Webhook

To supply additional data to the Webhook, edit the playbook. Insert a step with the operator called loadEventsFromExecutionContext().

The operator’s argument is a parent table that is used with Webhook execution and provides a body JSON with additional data. The JSON code from the parent table can help test the playbook while authoring without needing to set up a running stream. It can also provide additional data to the playbook if the playbook itself does not contain all of the desired data.
When the playbook runs using Webhook functionality, the output of loadEventsFromExecutionContext() contains the body parameters that are sent using the Webhook. The parameters are formatted as follows with the resulting table as shown below.

curl -XPOST --header "Content-Type: application/json" -d '{"rows": [{"key1": "peter_test95", "splunk_notable_id":"splunkid", "lhub_id": "10"}, {"key1": "vj_test", "lhub_id": "11"}]}' https://xyz.logichub.com/api/trigger/stream-5234/717e995a-6fdd-42b7-a2c2-ce3e9e613d63

If the provided body contains the key rows with the values as an array of key-value pairs, then the loadEventsFromExecutionContext() operator parses the data and formats it into a table as follows.
If the provided body isn’t in this format, the loadEventsFromExecutionContext() operator loads the provided body contents as a string in its raw structure.

``` {json}{ "rows":[ { "key1":"peter_test95", "splunk_notable_id":"splunkid", "lhub_id":"10" }, }

If there are more requests which exceed what is specified by the above properties, then these will be queued. By default, the queue length is 20. Following property controls the queue length.

If the queue is full, further requests will be rejected. The webhook API will return an HTTP error "TOO_MANY_REQUESTS" with HTTP status code "429".

The queue is drained at the same rate as specified by the rate limit configuration that is, by default five requests in 5 minutes interval. There is another optimization that kicks in for the queued requests; queued requests can be merged into fewer requests as explained in the section below.

This queue is an in-memory queue and is designed to absorb occasional spikes. Therefore, any application restart will lose the queue data. If requests take less time to be processed then a larger queue will be fine, else you can increase the parallelism.

Merging/Coalescing of Stream Requests

Coalescing of request is an optimization that works in the following cases:

1. There are many requests in a short span of time

2. Only one execution of stream can handle all the required logic of all incoming requests

All requests that are queued are candidates for merging when they are picked from the queue.
Requests will be merged only if they satisfy the following criteria:

1. Requests are from the same stream

2. Requests have the same parameter (on demand stream can take params as well)

3. Time interval gap is less than an hour. Refer to
   (lhub.stream.rateLimitterMaxTimeRangeGap=3600000)

The following requests will merge:

Merged request

Turn Off Merging/Coalesce of Requests

Coalescing of request is an optimization that works well in the following cases:

1. There are many requests in a short span of time

2. Only one execution of stream can handle all the required logic of all incoming requests

If your on demand streams do not work due to coalescing into fewer requests, you can turn off coalescing by setting the following properties:

• Either Set Queue sIze parameter to 1
  lhub.stream.rateLimitterQueueSize=1

Or, set the time interval property to 0
lhub.stream.rateLimitterMaxTimeRangeGap=0

There may be a slight performance penalty, as the requests that could have been merged will now run independently.

Recommendations in Case Merging/Coalesce of Requests is Turned Off

1. First turn off merging by following
   lhub.stream.rateLimitterMaxTimeRangeGap=0

2. Find out your concurrency
   execution_time = Average Batch Execution Time (in minutes)
   n = Number of webhook requests per minute
   concurrency = n * execution_time

3. Change following concurrency properties
   a. Increase your parallel webhook requests run count
   lhub.stream.rateLimitterPermits= concurrency

b. Change refill time to average batch execution time
lhub.stream.rateLimitterRefillTime = execution_time (in milliseconds)

1. Change queue size
   If rate limit is exceeded requests are queued, queue size can be controlled via
   lhub.stream.rateLimitterQueueSize=20

This is useful in cases when there are occasional spikes. Requests are rejected if the queue is full.

Properties Reference