Splunk Enterprise Security

Splunk Enterprise Security: Why am I unable to find Threat Intellegence data after successful URL download?


Greetings and thanks for the looking at this question. I have a Splunk server in an air-gapped environment and I'm trying to get threat intelligence data from an external, Internet facing client to an internal URL server and then have Splunk Enterprise Security (ES) perform a URL download of the intelligence file. The external client downloads the feed and then pushes the feed to the URL server (I can push XML, JSON, CSV files).

I am able to get the intelligence file to the URL server from the external client and I set up a Threat Intelligence stanza within Data inputs » Threat Intelligence Downloads to download the file via URL. I am also able to verify that the intelligence file is successfully downloaded; here are the logs for the download:

Threat Intelligence Logs for URL download:

12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:run:373 | status="continuing" msg="Processing stanza" name="threatlist://custom_ioc"

12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:run:381 | status="retrieved_checkpoint_data" stanza="custom_ioc" last_run="1492973638.25"

12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:download_csv:279 | status="CSV download starting" stanza="custom_ioc"

12:53:58.246 AM 
2017-04-23 23:53:58,246 INFO pid=40446 tid=MainThread file=protocols.py:buildOpener:127 | Proxy server will not be used (check your proxy_server, proxy_port, and proxy_user settings if this is incorrect).

12:53:58.958 AM 
2017-04-23 23:53:58,958 INFO pid=40446 tid=MainThread file=threatlist.py:download_csv:310 | stanza="custom_ioc" retries_remaining="3" status="threat list downloaded" file="/indexes/hot/modinputs/threatlist/custom_ioc_2Gwqd.txt" bytes="17882720" url="https://urlserver/ioc/misp.csv.all.ADMIN.csv"

Then after the file is downloaded, I am getting the following error:

12:54:41.689 AM 2017-04-23 23:54:41,689 ERROR pid=40824 tid=MainThread file=threat_intelligence_manager.py:process_files:513 | status="Exception when processing file." filename="custom_ioc.csv" 
Traceback (most recent call last): 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 511, in process_files self.process_file(fullpath, last_run) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 252, in process_file self.process(filename, parser, typ, last_run) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 386, in process for metadata, intel in parser.parse(self._kvstore_limits): 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/parsers/csv_parser.py", line 402, in parse parser = CSVParserConfiguration(self.filename, self._stanza, self._collection_spec) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/parsers/csv_parser.py", line 95, in __init__ raise ValueError('Parser does not extract a field that can be mapped to a threat intelligence collection.') 
ValueError: Parser does not extract a field that can be mapped to a threat intelligence collection.

So it appears I have a parsing error, but I am not sure where to look next for that. I know I'm missing something, but am not sure what. Any assistance that can be given to point me in the right direction would be appreciated. Thank you.

Path Finder

Hi there, I realize I'm about a year late to the party, but in case this is still an issue, or if someone else comes to this from googling the parsing error as i did i wanted to still comment.

The ES parser cannot handle white-space, so if there's any white-space in the file you'll get this error.

I opened my file in vim, after trying to escape all the things, quote the descriptions, etc, and what finally got it to import was running :%s/\ //gafter replacing my spaces in the description with underscores.

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...