Splunk Enterprise Security: Is there a way to accept JSON as a threat intelligence download?

[panovattack]

Is there a way to accept a JSON as a threat intelligence download? I have a threat intelligence vendor that only provides JSON over their API.

[kchamplin_splun]

Currently there is not a JSON parser built into the Threat Intelligence Framework in Splunk ES. That said, if the Threat Intel provider has an app or TA and you can get the the data into a Splunk index, you can use a saved search to push it into either a KVStore lookup or CSV lookup that the Threat Intelligence Framework already monitors.

As an example, the iSight Partners intelligence feed is JSON based, and their app sorts out the download and save to index part of the problem. To get this integrated into the Threat Intelligence framework of ES you can simply create a saved search similar to the following.

index="isightpartners" domain !=null | eval description="iSight Partners Intel" |table domain,description | outputlookup append=T local_domain_intel

The local_domain_intel is a csv based lookup that is written to $SPLUNK_HOME/etc/apps/DA-ESS-ThreatIntelligence/lookups/local_domain_intel.csv

That specifically covers domain intel, but the rules apply across all threat categories (also the "description" field I created with eval since it's option for all intel and generally speaking is likely not in the source JSON). Here's a list of the associated "fields" you will want to write to for each category of intel - note that these also have other associated csv backed look files in the same directory.

email_intel:
    src_user
    subject

file_intel:
    file_hash
    file_name

http_intel:
    http_referrer
    http_user_agent
    url

ip_intel (by domain)
    domain

ip_intel (by ip)
    ip

process_intel
    process
    process_file_name

registry_intel
    registry_path
    registry_value_name
    registry_value_text

service_intel
    service
    service_file_hash
    service_dll_file_hash

user_intel
    user

certififate_intel
    certificate_issuer
    certificate_issuer_email
    certificate_issuer_organization
    certificate_issuer_unit
    certificate_serial
    certificate_subject
    certificate_subject_email
    certificate_subject_organization
    certificate_subject_unit

So if your JSON threat intel also contained info with file names and hashes, the aforementioned search could be tweaked for that case as well.

index="isightpartners" domain !=null | eval description="iSight Partners Intel" |table description, file_hash, file_name | outputlookup append=T local_file_intel

Please note that the fields in this case just happened to already have the right naming convention, you can always use eval to rename the raw, JSON-based intel to the right field names (as listed in the table above).

If you don't mind sharing, what is the threat intel provider you are using?

[panovattack]

Thanks kcchamplin. The iSightPartner API now allows you pull STIX files directly to the monitored threat intelligence directories and Splunk picks them up very well (helps with getting the file based indicators into splunk).

The intelligence provider is ThreatGrid. Their API is currently JSON only...Right now I am toying with the idea of downloading the JSON files, converting them to CSV and having a monitored directory pick them up. I don't see a ThreatGrid App (TA or otherwise). Did I miss it?

[kchamplin_splun]

Hey panovattack - very good to know regarding STIX direct download for iSight. As far as an app or TA for ThreatGrid, that would be awesome, but you're right there's not one currently available on Splunkbase. As far as rolling your own, you're on the right track for sure - you should be able to append to the previously listed files, there's also a custom inputs.conf stanza you can use:

[threatlist://my_custom_json_intel]
delim_regex = ,
description = JSON_based_intel_test
disabled = false
fields = domain:"$1",description:"$2"
ignore_regex = (^#|^\s*$|^Start)
type = threatlist
url = lookup://ip_intel_domain_lookup

Note that the fields again correspond to the ones outlined in my first post, and given its CSV, the "delim_regex" can remain a comma. Then you just use the "fields" stanza line to then map (in left to right order of your CSV file) the values to each intel category/component.

Since its using a lookup:// for the URL, you'll want to have the lookup configured in your transforms.conf

[ip_intel_domain_lookup]
filename = ip_intel_domain.csv

You can of course decide how your actual lookup file is formatted (if you decide that comma-separated is out of fashion, make sure to change your delim-regex to whatever your delimited is).