Splunk Enterprise Security: Why am I unable to find Threat Intellegence data after successful URL download?

[jonathangrant74]

Greetings and thanks for the looking at this question. I have a Splunk server in an air-gapped environment and I'm trying to get threat intelligence data from an external, Internet facing client to an internal URL server and then have Splunk Enterprise Security (ES) perform a URL download of the intelligence file. The external client downloads the feed and then pushes the feed to the URL server (I can push XML, JSON, CSV files).

I am able to get the intelligence file to the URL server from the external client and I set up a Threat Intelligence stanza within Data inputs » Threat Intelligence Downloads to download the file via URL. I am also able to verify that the intelligence file is successfully downloaded; here are the logs for the download:

Threat Intelligence Logs for URL download:

4/24/17
12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:run:373 | status="continuing" msg="Processing stanza" name="threatlist://custom_ioc"

4/24/17
12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:run:381 | status="retrieved_checkpoint_data" stanza="custom_ioc" last_run="1492973638.25"

4/24/17
12:53:58.245 AM 
2017-04-23 23:53:58,245 INFO pid=40446 tid=MainThread file=threatlist.py:download_csv:279 | status="CSV download starting" stanza="custom_ioc"

4/24/17
12:53:58.246 AM 
2017-04-23 23:53:58,246 INFO pid=40446 tid=MainThread file=protocols.py:buildOpener:127 | Proxy server will not be used (check your proxy_server, proxy_port, and proxy_user settings if this is incorrect).

4/24/17
12:53:58.958 AM 
2017-04-23 23:53:58,958 INFO pid=40446 tid=MainThread file=threatlist.py:download_csv:310 | stanza="custom_ioc" retries_remaining="3" status="threat list downloaded" file="/indexes/hot/modinputs/threatlist/custom_ioc_2Gwqd.txt" bytes="17882720" url="https://urlserver/ioc/misp.csv.all.ADMIN.csv"

Then after the file is downloaded, I am getting the following error:

4/24/17
12:54:41.689 AM 2017-04-23 23:54:41,689 ERROR pid=40824 tid=MainThread file=threat_intelligence_manager.py:process_files:513 | status="Exception when processing file." filename="custom_ioc.csv" 
Traceback (most recent call last): 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 511, in process_files self.process_file(fullpath, last_run) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 252, in process_file self.process(filename, parser, typ, last_run) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/threat_intelligence_manager.py", line 386, in process for metadata, intel in parser.parse(self._kvstore_limits): 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/parsers/csv_parser.py", line 402, in parse parser = CSVParserConfiguration(self.filename, self._stanza, self._collection_spec) 
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/parsers/csv_parser.py", line 95, in __init__ raise ValueError('Parser does not extract a field that can be mapped to a threat intelligence collection.') 
ValueError: Parser does not extract a field that can be mapped to a threat intelligence collection.

So it appears I have a parsing error, but I am not sure where to look next for that. I know I'm missing something, but am not sure what. Any assistance that can be given to point me in the right direction would be appreciated. Thank you.

[scruse]

Hi there, I realize I'm about a year late to the party, but in case this is still an issue, or if someone else comes to this from googling the parsing error as i did i wanted to still comment.

The ES parser cannot handle white-space, so if there's any white-space in the file you'll get this error.

I opened my file in vim, after trying to escape all the things, quote the descriptions, etc, and what finally got it to import was running :%s/\ //gafter replacing my spaces in the description with underscores.