Document toolboxDocument toolbox

NSS feeds for web logs

Owned by Juan Tomás Alonso Nieto (Deactivated)
May 23, 2022

Only for NSS web server.

A large number of filters or complex filters, such as string search, might impact the performance of the NSS.

To configure a feed for web logs:

  1. Go to Administration → Nanolog Streaming Service.
  2. On the NSS Feeds tab, click Add NSS Feed. The Add NSS Feed window appears.

  3. On the Add NSS Feed window, enter the following information:

    FieldInformation
    Feed NameEnter or edit the name of the feed. Each feed is a connection between NSS and your Devo Relay.
    NSS TypeSelect which type of feed you are configuring. NSS for Web is selected by default.
    NSS ServerChoose an NSS from the list.
    StatusThe NSS feed is Enabled by default. Click Disabled if you want to activate it later.
    SIEM Destination Type

    The type of destination. Choose between:

    • SIEM IP Address - Enter the IP address of the Devo Relay to which the logs are streamed. 

    • FQDN - (optional) Enter the destination for the TCP connection to which the logs are streamed. This allows failover from one IP to the other without manual intervention, but rather relying on updating the DNS entry. NSS will re-resolve the FQDN only when the existing connection goes down. This feature cannot be used for DNS-based load balancing.

    SIEM TCP PortEnter the port number of the Devo Relay to which the logs are streamed. Ensure that the Devo Relay is configured to accept the feed from the NSS. If you are using the proposed TCP configuration, type 13004. 
    Log TypeChoose Web Logs.
    SIEM Rate Limit (Events per Second)Leave as unrestricted, unless you need to throttle the output stream due to licensing or other constraints. A limit that is too low for the traffic volume will cause log loss.
    Feed Output TypeChoose Custom.
    Feed Escape Character

    The Zscaler service hex encodes all non-printable ASCII characters that are in URLs when it sends the logs to the NSS. Any URL character that is less than 0x21, or above 0x7E, will be encoded as %HH. This ensures that your Devo Relay will be able to parse the URLs in case they contain non-printable characters. For example, a \n char in a URL is encoded as %0A, and a space is encoded as %20. In this field, you can specify additional characters that you would like to encode. For example, type a comma (,) to encode it as %2C. This is useful if you are using this character as your delimiter and would like to ensure it does not cause erroneous delimitation. Note that the service encodes characters in URLs, host names, and referer URLs only. If custom encoding was done for a record, the %s{eedone} field will be YES for that record.


    Feed Output Format

    Copy and paste the following output format:

    \{"time":"%s{time}","tz":"%s{tz}","ss":%d{ss},"mm":%d{mm},"hh":%d{hh},"dd":%d{dd},"mth":%d{mth},"mon":"%s{mon}","yyyy":%d{yyyy},"day":"%s{day}","epochtime":%d{epochtime},"dept":"%s{dept}","login":"%s{login}","ologin":"%s{ologin}","throttlereqsize":%d{throttlereqsize},"throttlerespsize":%d{throttlerespsize},"bwthrottle":"%s{bwthrottle}","bwclassname":"%s{bwclassname}","bwrulename":"%s{bwrulename}","appname":"%s{appname}","appclass":"%s{appclass}","module":"%s{module}","bamd5":"%s{bamd5}","dlpdict":"%s{dlpdict}","dlpdicthitcount":"%s{dlpdicthitcount}","dlpeng":"%s{dlpeng}","dlpidentifier":%d{dlpidentifier},"dlpmd5":"%s{dlpmd5}","fileclass":"%s{fileclass}","filetype":"%s{filetype}","filesubtype":"%s{filesubtype}","filename":"%s{filename}","unscannable":"%s{unscannable}","reqdatasize":%d{reqdatasize},"reqhdrsize":%d{reqhdrsize},"reqsize":%d{reqsize},"respdatasize":%d{respdatasize},"resphdrsize":%d{resphdrsize},"respsize":%d{respsize},"totalsize":%d{totalsize},"reqmethod":"%s{reqmethod}","reqversion":"%s{reqversion}","respcode":"%s{respcode}","respversion":"%s{respversion}","referer":"%s{referer}","uaclass":"%s{uaclass}","ua":"%s{ua}","ua_token":"%s{ua_token}","host":"%s{host}","ehost":"%s{ehost}","eurl":"%s{eurl}","ereferer":"%s{ereferer}","contenttype":"%s{contenttype}","refererhost":"%s{refererhost}","erefererpath":"%s{erefererpath}","eurlpath":"%s{eurlpath}","erefererhost":"%s{erefererhost}","url":"%s{url}","df_hostname":"%s{df_hostname}","mobappname":"%s{mobappname}","mobappcat":"%s{mobappcat}","mobdevtype":"%s{mobdevtype}","cip":"%s{cip}","cintip":"%s{cintip}","sip":"%s{sip}","proto":"%s{proto}","trafficredirectmethod":"%s{trafficredirectmethod}","location":"%s{location}","rulelabel":"%s{rulelabel}","ruletype":"%s{ruletype}","reason":"%s{reason}","action":"%s{action}","ssldecrypted":"%s{ssldecrypted}","clientsslcipher":"%s{clientsslcipher}","clienttlsversion":"%s{clienttlsversion}","clientsslsessreuse":"%s{clientsslsessreuse}","srvsslcipher":"%s{srvsslcipher}","srvtlsversion":"%s{srvtlsversion}","srvocspresult":"%s{srvocspresult}","srvcertchainvalpass":"%s{srvcertchainvalpass}","srvwildcardcert":"%s{srvwildcardcert}","serversslsessreuse":"%s{serversslsessreuse}","srvcertvalidationtype":"%s{srvcertvalidationtype}","srvcertvalidityperiod":"%s{srvcertvalidityperiod}","riskscore":%d{riskscore},"threatname":"%s{threatname}","malwareclass":"%s{malwareclass}","malwarecat":"%s{malwarecat}","urlclass":"%s{urlclass}","urlsupercat":"%s{urlsupercat}","urlcat":"%s{urlcat}","devicehostname":"%s{devicehostname}","devicemodel":"%s{devicemodel}","devicename":"%s{devicename}","deviceostype":"%s{deviceostype}","deviceosversion":"%s{deviceosversion}","deviceowner":"%s{deviceowner}","deviceappversion":"%s{deviceappversion}","ztunnelversion":"%s{ztunnelversion}","recordid":%d{recordid},"productversion":"%s{productversion}","nsssvcip":"%s{nsssvcip}","eedone":"%s{eedone}"\}\n
    User ObfuscationYou can enable user obfuscation. When you do, it displays a random string instead of the user names. If this is enabled, the login field in Feed Format Output automatically changes to ologin field, which outputs the obfuscated login name. Choose Disable to display the user names.
    TimezoneBy default, this is set to the organization's time zone. The time zone you set applies to the time field in the output file. The time zone automatically adjusts to changes in daylight savings in the specific time zone. The configured time zone can be output to the logs as a separate field. The list of time zones is derived from the IANA Time Zone Database. Direct GMT offsets can also be specified.
    Duplicate LogsTo ensure that no logs are skipped during any downtime, specify the number of minutes that the NSS will send duplicate logs. Zscaler recommends setting the number to 60. This allows the NSS to send one-hour logs to the Devo Relay after the connection between the NSS and Devo Relay recovers.
  4. Click Save and activate the change.

Available filters

Action

  • Policy Action: Use this filter to limit the logs to transactions that were either allowed or blocked. Transactions wherein the service displayed a Caution page are considered blocked transactions; if users proceeded with the transactions, they are considered allowed.

  • Policy Reason: Use this filter to limit the logs based on the policy that the Zscaler service applied. These are the policy reason strings that are in transaction drilldown. They indicate which policy caused a block, or if allowed, the conditions under which they were allowed, such as Allowed due to override and Internet Access cautioned. Multiple selections are allowed.

Who

  • Users: Use this filter to limit the logs to specific users who generated transactions. You can search for users by user name or email address. There is no limit on the number of users that you can select. Users that are deleted after they are selected appear with a strikethrough line.

  • Departments: Use this filter to limit the logs to specific departments that generated transactions. You can search for departments. There is no limit on the number of departments that you can select. Departments that are deleted after they are selected appear with a strikethrough line.

From Where

  • Locations: Use this filter to limit the logs to specific locations from which transactions were generated. You can search for locations. There is no limit on the number of locations that you can select. Locations that are deleted after they are selected appear with a strikethrough line.

  • Client IP Addresses: Use this filter to limit the logs based on a client’s private IP address. You can enter:

    • An IP address (for example, 198.51.100.100)

    • A range of IP addresses (for example, 192.0.2.1-192.0.2.10)

    • An IP address with a netmask (for example, 203.0.113.0/24)

You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

  • Public IP Addresses: Use this filter to limit the logs based on a client’s public IP address. The internal IP address is available if traffic forwarding is forwarded to the service through a GRE or VPN tunnel or from the XFF header. If the internal IP address is not available, the value will be same as the client IP address. You can enter:

    • An IP address (for example, 198.51.100.100)

    • A range of IP addresses (for example, 192.0.2.1-192.0.2.10)

    • An IP address with a netmask (for example, 203.0.113.0/24)

You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

Transaction

  • Direction: Use this filter to limit the logs to either inbound or outbound traffic.

  • User Agents: Use this filter to limit the logs to transactions associated with the user-agent string that the browser included in its GET request. Choose from the list of predefined user-agent strings or enter custom user-agent strings. Multiple selections are allowed.

  • Custom User Agent Strings: Use this filter to limit the logs to specific user-agent strings. A user-agent string contains browser and system information that the destination server can use to provide appropriate content.

  • Protocol Types: Use this filter to limit the logs to specific protocols. Supported protocols are HTTP, HTTPS, and FTP. Multiple selections are allowed.

  • Request Methods: Use this filter to limit the logs based on the HTTP request method obtained from the client request. Multiple selections are allowed.

  • Response Codes: Use this filter to limit the logs based on the HTTP response code obtained from the server or generated by the ZIA Public Service Edge. Multiple selections are allowed.

  • Request Sizes: Use this filter to limit the logs based on HTTP request size. Enter either a specific size or a range with a dash. By default, the service uses bytes, but you can also specify KB, MB, GB, or TB. For example: 10KB-1MB, 200. You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

  • Response Sizes: Use this filter to limit the logs based on HTTP response size. Enter either a specific size or a range with a dash. By default, the service uses bytes, but you can also specify KB, MB, GB, or TB. For example: 10KB-1MB, 200. You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

  • Transaction Sizes: Use this filter to limit the logs based on transaction size, which is the header and body request or response size, or the request and response size. Enter either a specific size or a range with a dash. By default, the service uses bytes, but you can also specify KB, MB, GB, or TB. For example: 10KB-1MB, 200. You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

  • Referer URLs: Use this filter to limit the logs based on the Referer URL in the HTTP header. You can use wildcards based on the rules:

    • *string -> Suffix matching match URLs ending with ‘string’

    • String* -> Prefix matching match URLs beginning with ‘string’

    • *string* -> Substring matching match URLs containing ‘string’

    • String -> Exact matching match URLs that are exactly ‘string’

Multiple strings are allowed. Enter one string per line, then click Add Items. String search is not case-sensitive. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

To Where

  • URL Filter Type: Use this filter to limit the logs based on URLs in HTTP Requests. You can specify either a Hostname or the Full URL. You can use wildcards based on the rules:

    • String -> Exact matching  match URLs that are exactly ‘string’

    • *string* -> Substring matching match URLs containing ‘string’

    • String* -> Prefix matching match URLs beginning with ‘string’

    • *string -> Suffix matching match URLs ending with ‘string’

  • Host Names: Use this filter to limit the logs based on specific host names. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

  • URL Classes: Use this filter to limit the logs to specific URL classes. Select those that you would like to include. Multiple selections are allowed.

  • URL Super Categories: Use this filter to limit the logs to specific URL super categories. Select those that you would like to include. Multiple selections are allowed.

  • URL Categories: Use this filter to limit the logs to specific URL categories. Select those that you would like to include. Multiple selections are allowed.

  • Server IP Addresses: Use this filter to limit the logs based on the destination server’s IP address. You can enter:

    • An IP address (for example, 198.51.100.100)

    • A range of IP addresses (for example, 192.0.2.1-192.0.2.10)

    • An IP address with a netmask (for example, 203.0.113.0/24)

You can enter multiple entries. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

    • Cloud Application Classes: Use this filter to limit the logs to the selected cloud application classes. Multiple selections are allowed.

    • Cloud Applications: Use this filter to limit the logs to selected cloud applications. Multiple selections are allowed.

Security

  • Malware Classes: Use this filter to limit the logs based on malware class or name. Multiple selections are allowed.

  • Malware Names: Use this filter to limit the logs based on specific malware or viruses that were detected. You can specify multiple malware or virus names. Use the Search function to search for either.

    • Use the guidelines:

      • *string -> Suffix matching match malware names ending with ‘string’

      • String* -> Prefix matching match malware names beginning with ‘string’

      • *string* -> Substring matching match malware names containing ‘string’

      • String -> Exact matching match malware names that are exactly ‘string’
        Multiple strings are allowed. Enter one string per line. String search is not case-sensitive.

  • Advanced Threats: Use this filter to limit the logs based on the types of advanced threats that were detected. Multiple selections are allowed.

  • Threat Names: Use this filter to limit the logs based on specific threats that were detected. You can specify multiple threat names. Use the Search function to search for either.

    • Use the guidelines:

      • *string -> Suffix matching match threat names ending with ‘string’

      • String* -> Prefix matching match threat names beginning with ‘string’

      • *string* -> Substring matching match threat names containing ‘string’

      • String -> Exact matching match threat names that are exactly ‘string’
        Multiple strings are allowed. Enter one string per line. String search is not case-sensitive.

For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

    • Suspicious Content: Use this filter to limit the logs based on the Page Risk Index score of a transaction. Enter either a single value or a range of values, between 0 and 100. Multiple values are allowed. Hit Enter after each entry, then click Add Items. For item lists, you can view up to 500 items on a page; filter the list by searching for a word, phrase, or number contained in an item; and remove all items from the list (Remove All) or only items from a specific page (Remove Page). If you select Remove All or Remove Page, a confirmation window will appear.

File Type

  • File Type Categories: Use this filter to limit the logs based on the file type categories detected from the content. Multiple selections are allowed.

  • File Types: Use this filter to limit the logs based on the file type detected from the content. Multiple selections are allowed.

DLP

  • DLP Engines: Use this filter to limit the logs to transactions in which data leakage was detected based on specific DLP engines. Multiple selections are allowed.

  • DLP Dictionaries: Use this filter to limit the logs to transactions in which data leakage was detected based on specific DLP dictionaries. Multiple selections are allowed.