The Threat Intelligence Framework provides a modular input (Threat Intelligence Downloads) that handles the majority of configurations typically needed for downloading intelligence files & data. To access this modular input, you simply need to create a stanza in your Inputs.conf file called “threatlist”.
A very basic example is as follows:
[threatlist://myfirstintel]
delim_regex =
description = My First Threat Tofu
disabled = true
extract_regex = ^(\S+)\t+(\S+)\t+\S+\t+\S+\t*(\S*)
fields = ip:"$1-$2",description:"$3"
ignore_regex = (^#|^\s*$|^Start)
type = threatlist
url = http://home.dishwishy.com/threatofu.txt
This represents an example of “ip_intel” (IP address and domain) threat intelligence. Keep in mind that this is not CSV based so we will create the extraction regexs. Each stanza key provides the following functionality:
url: the download location for the intelligence data. Note that both GET and POST are supported (see appendix).
type: this can be a custom name, and is typically an arbitrary value. There are additional types that will be discussed later.
ignore_regex: regular expression that is used to ignore specific lines of text, for example in the case of commented lines that you do not want Threat Framework to look at. In this case it ignores any lines that start with the “#” character and other anchored elements (for this example, each item is pipe separated).
fields: This maps the specific threat intelligence fields (specific to the area of intelligence) to the extracted fields. The values prefixed with “$” reference the capture groups from the “extract_regex” key.
extract_regex: This is the regular expression used to extract the specific values to match with the Threat Framework expected values. This is used if there is not a consistent delimiter in the source file. NOTE: In the above sample, the data is tab separated, but is not consistent enough to use the “delim_regex”.
disabled: enabled or disabled – this can be set via the Splunk GUI. It’s best practice to leave as “disabled” and allow the end-user to enable it. Note: This process will create a new “local” directory with its own inputs.conf that will then show as enabled.
description: A description of what the intel and and what it does (shows in the mod-input)
delim_regex: This can be used in lieu of the “extract_regex” key if your intel is well formed/already delimited such as CSV files. This then auto extracts to the “$” prefixed values.
NOTE: the following key defaults for the “threatlist” stanza:
delim_regex - (defaults to ",") – NOTE: you need to
extract_regex - (no default)
fields - (defaults to "description:$1,ip:$2")
ignore_regex - (defaults to "(^#|^\s*$)")
skip_header_lines - (defaults to 0)
To understand how this is then parsed and ingested into Splunk ES, please see the following diagram. NOTE: This is not comma-separated (CSV) source data, so we are relying on the “extract_regex” to create the capture groups; which are highlighted in red, green and blue.
The capture groups denoted in red, blue and green are mapped in left to right order to the field mappings. In other words, $1 = red, $2 = blue, $3 = green. Those tokens are then used to map to the fields that exist in the Threat Intelligence Framework for both KVStore and CSV backed lookups.
