Splunk Enterprise Security

Why is Threat Intelligence not correctly parsing and automatically setting to STIX parsing?

RickvdIJ
Explorer

Hi all,

Within Splunk ES I've configured a test threat intelligence feed with the following settings:

New > Line oriented

  • Name: Binary Defense Banlist
  • type: network
  • url: https://www.binarydefense.com/banlist.txt
  • weight: 60
  • interval: 43200
  • Max Age: -30d
  • Max Size: 52428800
  • Checked Threat Intelligence
  • File parser: line
  • Delimiting regular exp: 
  • Extracting regex: ^(\d.+)$
  • Ignoring regex: (^#|^\s*$)
  • fields: ip$1,description:BinaryDefense_banlist
  • skip header lines: 0
  • No encoding, no user agent, sinkhole checked.

Some global parse modifier settings:

  • Certificate attribute breakout = checked
  • IDNA encode domains = unchecked
  • Parse domain from URL = unchecked

In debug mode I see that the file is downloaded and then it says:

<timestamp> INFO pid=1050977 tid:MainThread file=get_parser.oy:_detect_file_type:139 | stanza"binary Defense Banlist" status="Automatically detected STIX parsing for file_path /opt/splunk/var/lib/splunk/modinputs/threatlist/Binary Defense Banlist"

It goes on to parse the file and get the records. However, the records contain HTML elements like <'\div> and <\iframe> as url value. This is strange since it's just a .txt file. Moreover, why is it parsing it like a STIX document when I explicitly stated that the File parser = line?

This happens with other threat feeds as well. I've checked with a colleague at another client and with the exact same settings his works and mine doesn't.

 

Am I missing something? Do you know where else I can look to troubleshoot?

 

Some figures:

Splunk: 8.2.9

ES: 7.0.1

Single search head, behind proxy

Labels (2)
0 Karma
1 Solution

RickvdIJ
Explorer

It looks like a proxy issue where the proxy is returning a blockpage. This explains why Splunk ES is seeing html elements as a result. Still in investigation.

View solution in original post

0 Karma

lblystone
Splunk Employee
Splunk Employee

Did you add the proxy options to the intel download settings to see if that remediated it?

 

Additionally, I have noticed that sometimes the proxy will block the intel downloads as it thinks the list of malicious URLs/domains/IPs is itself malicious. I had to whitelist links coming from my Splunk instance on the proxy to remediate this. 

0 Karma

RickvdIJ
Explorer

It looks like a proxy issue where the proxy is returning a blockpage. This explains why Splunk ES is seeing html elements as a result. Still in investigation.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...