Devo SOAR - approximateLabelLookup

Score an event table by looking at similar events from another event table.

Example:
tableA is a table that has already been processed and scored.
tableB is an event table to be scored.

The two tables have some overlapping content (some similar events) but are not the same

Instead of processing all of tableB, use the scores (or labels) that are already determined for the parts that are similar to tableA. With this method, only the portions of tableB that are not similar to tableA must be processed and scored. It is not necessary to repeat the processing for the similar portions of the tables.

Operator Usage in Easy Mode

Click + on the parent node.
Enter the Approximate Label Lookup operator in the search field and select the operator from the Results to open the operator form.
In the Reference table drop-down, enter or select a reference table.
Click Add More to add the list of field names in the reference table to measure similarity.
In the Ref Label field, enter the label field name in the reference table.
In the Score Table drop-down, enter or select a table to be scored.
Click Add More to add the list of columns in the to-be-labeled table, same order as in the reference field.
Click Run to view the result.
Click Save to add the operator to the playbook.
Click Cancel to discard the operator form.

Usage Details

``` {text}approximateLabelLookup(tableA,listOfColumnsFromTableA, scoreColumn, tableB, listOfColumnsFromTableB)

**Input**  
`[tableA](http://google.com)`: reference (lookup) table  
`listOfColumnsFromTableA`: list of column names from `tableA` that will be used as a feature to measure similarity.e.g. ["bytes_in","bytes_out"]. Column values should be numeric.  
`scoreColumn`: lookup score or label from `tableA`  
`tableB`: event table to be scored (approximate label)  
`listOfColumnsFromTableB`: list of column names from `tableB` that will be used as a feature to measure similarity between `tableA` and `tableB`. Ordering of columns names are important, should same order as in listOfColumnsFromTableA

**Output**  
tableB + addition "lhub_lookup_label"

## Example

We want to find a score for `tableB` by looking similar events from `tableA`, where by "similar" we mean: 

`tableA.col1` is similar to `tableB.col1`, and `tableA.col2` is similar to `tableB.col2` (but not same).

**TableA**

&lt;style&gt;
  th {
    border: 1px solid #cccccc;
    background-color: #eeeeee;
    padding: 8px 5px 8px 5px;
    text-align: left
  }
&lt;/style&gt;

&lt;div&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;id&lt;/th&gt;
&lt;th&gt;col1&lt;/th&gt;
  &lt;th&gt;col2&lt;/th&gt;
  &lt;th&gt;score&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;u1&lt;/td&gt;&lt;td&gt;11&lt;/td&gt;&lt;td&gt;12&lt;/td&gt;&lt;td&gt;1.0&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;u2&lt;/td&gt;&lt;td&gt;21&lt;/td&gt;&lt;td&gt;22&lt;/td&gt;&lt;td&gt;5.0&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;
&lt;td&gt;u3&lt;/td&gt;&lt;td&gt;31&lt;/td&gt;&lt;td&gt;32&lt;/td&gt;&lt;td&gt;10.0&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/tr&gt;
&lt;/table&gt;&lt;/div&gt;

**tableB:**

&lt;style&gt;
  th {
    border: 1px solid #cccccc;
    background-color: #eeeeee;
    padding: 8px 5px 8px 5px;
    text-align: left
  }
&lt;/style&gt;

&lt;div&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;id&lt;/th&gt;
&lt;th&gt;col1&lt;/th&gt;
  &lt;th&gt;col2&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
  &lt;tr&gt;&lt;td&gt;x1&lt;/td&gt;&lt;td&gt;11&lt;/td&gt;&lt;td&gt;11&lt;/td&gt;&lt;/tr&gt;
    &lt;tr&gt;&lt;td&gt;x2&lt;/td&gt;&lt;td&gt;20&lt;/td&gt;&lt;td&gt;20&lt;/td&gt;&lt;/tr&gt;
  &lt;tr&gt;&lt;td&gt;x3&lt;/td&gt;&lt;td&gt;50&lt;/td&gt;&lt;td&gt;50&lt;/td&gt;&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/tr&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;style&gt;table.blueTable {
  border: 1px solid #1C6EA4;
  background-color: #FFFFFF;
  width: 100%;
  text-align: left;
  border-collapse: collapse;
}
table.blueTable td, table.blueTable th {
  border: 1px solid #AAAAAA;
  padding: 3px 2px;
}
table.blueTable tbody td {
  font-size: 13px;
}
table.blueTable thead {
  background: #E0E0E0;
  background: -moz-linear-gradient(top, #e8e8e8 0%, #e3e3e3 66%, #E0E0E0 100%);
  background: -webkit-linear-gradient(top, #e8e8e8 0%, #e3e3e3 66%, #E0E0E0 100%);
  background: linear-gradient(to bottom, #e8e8e8 0%, #e3e3e3 66%, #E0E0E0 100%);
  border-bottom: 1px solid #444444;
}
table.blueTable thead th {
  font-size: 15px;
  font-weight: bold;
  color: #424242;
  border-left: 1px solid #D0E4F5;
}
table.blueTable thead th:first-child {
  border-left: none;
}

table.blueTable tfoot td {
  font-size: 14px;
}
table.blueTable tfoot .links {
  text-align: right;
}
table.blueTable tfoot .links a{
  display: inline-block;
  background: #1C6EA4;
  color: #FFFFFF;
  padding: 2px 8px;
  border-radius: 5px;
}&lt;/style&gt;

LQL command
``` {sql}
approximateLabelLookup(tableA, ["col1","col2"], "score", tableB, ["col1", "col2"])

Output

id	col1	col2	lhub_lookup_label
x1	11	11	1.0
x2	20	20	5.0
x3	50	50	null

"x3" is not scored, since we didn't find similar event from tableA. "u3" in tableA is closer than others, but it is not within 10% difference range.