Page Comparison

...

• Switch to server mode

• Manage browser memory

• Limit concurrent queries

• Use a brief time range when building a new query

• Follow the order of operations

• Consider cardinality when grouping events

• Reduce the number of fields in a table

Switch to server mode

When you access the search window after running a search, your browser is in charge of processing some of your query operations in addition to the server. In order to prevent browser exhaustion, switching to server mode is recommended when dealing with queries that require computationally heavy calculations. For the same reason, it is also recommended before grouping your data when dealing with tables that contain fields with a very large number of distinct values (that is, high cardinality and/or variability). In cases other than the aforementioned, it is recommended to continue running queries using the default search window mode.

...

Manage browser memory

Restart your browser to free up memory.

Minimize the number of open tabs to maximize available memory.

Limit concurrent queries

As a general rule, you should minimize the number of concurrent queries in order to maximize available memory.

If you need to have multiple, large queries open, create a second session by opening another browser window in incognito mode to better handle the memory requirements.

Use a brief time range when building a new query

Query-building can sometimes involve quite a bit of trial and error with the operations you apply to the data. So before starting to build and refine your query, apply a briefer time period and make sure that real-time event flow is off. This provides better performance because there will be fewer events to apply the operations to when you apply filters, create fields, and so on. Once you are satisfied with your query, you can set the time range and real-time event flow as you require. 

Follow the order of query operations

All filters should be applied early in the query, and certainly before grouping and aggregation. This reduces the memory and computation required for the later operations. The following describes the recommended order of operations:

1. Filters

2. Create fields (data enrichment)

3. Filters of new fields

4. Groups

5. Aggregations

Consider cardinality when grouping events

Avoid grouping by fields with a very large number of different values (high cardinality). This can be resource-intensive and produce results that are harder to read and analyze. Here are some tips for grouping by fields with high cardinality:

• Consider applying a filter to the field before grouping to limit the cardinality.

• If the field contains numeric values, enrich the data with a new field that identifies a numeric range to which the event belongs, then group by the numeric range instead of the individual values. 

Reduce the number of fields in a table

If you use a Finder to open a data table, you can pre-select only the fields with data that is of interest to you. This reduces the amount of data that your browser needs to load into memory. Here's how. 

...

• Toggle Execution Information

• View Selected Events

• Time Interval History

Toggle Execution Info

Click the gear icon on the toolbar and select Tools → Execution info. This menu gives you useful information about the current query and can tell you:

• How many rows the query has in total, and how many have been loaded so far.

• How much memory is currently being used, and what is the maximum memory you will be able to use.

View selected events

Sometimes it can be visually difficult to examine an event's data, especially when the number of fields necessitates a horizontal scrollbar. This is when this tool comes in really handy.

...

This window lists each event's fields and values on its own page so that you can thoroughly examine the events one at a time. The Rich views toggle is activated by default and when activated, correctly reads and displays fields with values in JSON format. Use the Type field to see the type of data of each Event. Image data is shown directly as an image. You can also copy an event's data to paste it elsewhere, or download the event in CSV, JSON, or TXT formats.

Time Interval History

When building and running queries to generate a periodic report, you may be working in multiple data tables concurrently and applying the same time range to each table.

...

Bonus tips

Sparse verses dense

Suppose you have to search for a specific tree in both a sparse forest and a dense forest. The spare forest will have a small number of trees and it will be really easy to spot a special tree. However, the dense forest will have too many trees that could possibly match the one you're looking for; you would have to manually inspect every tree.

This concept directly relates to the frequency of values and the number of events in your searches. 

Ordering of clauses

The order of clauses is important to achieve optimal performance in your queries. See the following example, where 99% of the logs in the table include the term "INFO". This query:

from application.log.log where toktains(raw, "INFO"), service="test"

...

Be careful when using the Not operation

The Not (not) operation does not use the token index, so it is recommended to push Not clauses down your query and apply additional filter operations before.

Use logical operators in the proper order

When using the logical operators And (and), Or (or) and Not (not), it is important to place them properly in the query to get the required results. Always keep in mind the De Morgan's law:

not (A or B) → (not A) and (not B)
not (A and B) → (not A) or (not B)

...