ES - Threat Intelligence - FS-ISAC Feeds

[ajhsjahdpgjhapi]

Attempting to ingest feeds from FS-ISAC into ES.
I can see in splunk that a file is created:
2018-06-19 17:01:28,107 INFO pid=23553 tid=MainThread file=stix_parser.py:preprocess:154 | msg="Finished parsing STIX documents" filename="/opt/splunk/etc/apps/SA-ThreatIntelligence/local/data/threat_intel/fsisac_filehash_TAXII_filehash_2018-06-19T17-01-22.135143.xml" success="0" failed="0"

ls -lah /opt/splunk/etc/apps/SA-ThreatIntelligence/local/data/threat_intel/
total 12K
drwx--x---. 2 splunk splunk 4.0K Jun 19 16:56 .
drwx--x---. 3 splunk splunk 25 Oct 17 2016 ..
-rw-------. 1 splunk splunk 483 Jun 19 16:56 fsisac_all_high_TAXII_all_high_2018-06-19T16-56-22.004935.xml
-rw-------. 1 splunk splunk 483 Jun 19 16:56 fsisac_filehash_TAXII_filehash_2018-06-19T16-56-21.863297.xml

Below are the contents of the file:
2018-06-19T21:01:06.060327+00:00

2018-06-19T21:01:06.101416+00:00

Whats strange is the file is quickly deleted and ever poll, splunk re-creates the file, then deletes it again. I never see any of the threat intelligence, I've disabled all other feeds in an attempt to get this to work and I don't see anything on the "Threat Intelligence > Threat Activity" dashboard.

I've:
1. Created multiple feeds on analysis.fsisac[dot]com
2. Created multiple Threat Intelligence Downloads in an attempt to get data from any of them (see inputs below):

I don't see any errors associated with feeds.

Status of fsisac threatintel_internal_logs:

eventtype=threatintel_internal_logs fsisac | stats count by status
status count
TAXII feed polling starting 5450
continuing 5450
retrieved_checkpoint_data 5300
Retrieved document from TAXII feed 4307
no_checkpoint_data 150
Detected updated threatlist stanzas - ALL lookup gen searches will be executed 5

inputs for fsisac:

[threatlist://fsisac]
description = FS-ISAC threat intel
index = _internal
initial_delay = 300
interval = 300
max_age = -1y
post_args = collection="Default" earliest="-1y" taxii_username="redacted" taxii_password="redacted" cert_file="redacted.crt" key_file="redacted.key"
retries = 3
retry_interval = 60
source = ModularInput:Threatlist
sourcetype = ModularInput:Threatlist
target = threatlist
timeout = 30
type = taxii
url = https://analysis.fsisac.com/
weight = 50

[threatlist://fsisac_2]
delim_regex = ,
description = FS-ISAC threat intel
ignore_regex = (^#|^\s*$)
index = _internal
initial_delay = 300
interval = 300
max_age = -1y
post_args = taxii_username="redacted" taxii_password="redacted" cert_file="redacted.crt" key_file="redacted.key"
retries = 3
retry_interval = 60
skip_header_lines = 0
source = fsisac
sourcetype = ModularInput:Threatlist
target = threatlist
timeout = 30
type = taxii
url = https://analysis.fsisac.com/taxii-discovery-service
weight = 50

[threatlist://fsisac_all_high]
delim_regex = ,
description = FS-ISAC threat intel
ignore_regex = (^#|^\s*$)
index = _internal
initial_delay = 300
interval = 300
post_args = collection="all_high" taxii_username="redacted" taxii_password="redacted" cert_file="redacted.crt" key_file="redacted.key"
retries = 3
retry_interval = 60
skip_header_lines = 0
source = fsisac_filehash
sourcetype = ModularInput:Threatlist
target = threatlist
timeout = 30
type = taxii
url = https://analysis.fsisac.com/
weight = 50

[threatlist://fsisac_filehash]
delim_regex = ,
description = FS-ISAC threat intel
disabled = 0
ignore_regex = (^#|^\s*$)
index = _internal
initial_delay = 300
interval = 300
post_args = collection="filehash" taxii_username="redacted" taxii_password="redacted" cert_file="redacted.crt" key_file="redacted.key"
retries = 3
retry_interval = 60
skip_header_lines = 0
source = fsisac_2
sourcetype = ModularInput:Threatlist
target = threatlist
timeout = 30
type = taxii
url = https://analysis.fsisac.com/
weight = 50

[pappjr]

These settings worked for me:

NOTE: Make sure you put your .crt and .key file issued by FS-ISAC in the auth folder of the app directory you create the input inside of (e.g. /opt/splunk/etc/apps/DA-ESS-ThreatIngelligence/auth). You can verify you're connecting successfully by reviewing the threat intel download logs (index=_internal sourcetype=threatintel:download)

[threatlist://fs-isac-default]
delim_regex = ,
description = FS-ISAC system.Default feed
ignore_regex = (^#|^\s*$)
interval = 43200
is_threatintel = 1
max_age = -30d
post_args = collection="system.Default" earliest="-1y" taxii_username="<your_provided_username>" taxii_password="<your_password>" cert_file="<your_cert.crt>" key_file="<your_key.key>"
retries = 3
retry_interval = 60
sinkhole = 0
skip_header_lines = 0
timeout = 30
type = taxii
url = https://analysis.fsisac.com/taxii-discovery-service
weight = 1

[stevenbukovic]

These generally worked for me as well, but I would note that some of the stanza's are invalid if you are not on a more current version of Splunk/ES combo. I would recommend starting out without sinkhole and is_threatintel otherwise the taxii polling won't even start. I found this out by restarting the splunk service and paying attention to the error streams that show up in the startup output. Errors will look like the following:

Invalid key in stanza
[threatlist://fs-isac-default] in
/opt/splunk/etc/apps/SA-ThreatIntelligence/local/inputs.conf, line 47: is_threatintel (value: 1).
Invalid key in stanza
[threatlist://fs-isac-default] in
/opt/splunk/etc/apps/SA-ThreatIntelligence/local/inputs.conf, line 52: sinkhole (value:0).

I overlooked the original note on .crt and .key files need to be placed in the auth/ folder within the app.

I found the following conf talk from 2017 as well which may help fill in some details for folks. Though it doesn't mention ISAC data specifically could be a good primer for others.

https://conf.splunk.com/files/2017/slides/enterprise-security-biology-dissecting-the-splunk-enterpri...

[stevenbukovic]

Any updates on this thread?

[mad4wknds]

"What's strange is the file is quickly deleted and ever poll, Splunk re-creates the file, then deletes it again. I never see any of the threat intelligence."

Under Threat Intelligence Management you can remove the sinkhole policy that deletes the files