Splunk Enterprise Security: After upgrading from Splunk 6.3.3 to 6.5.0, why are these threat lists failing to download?

[ttchorz]

After moving to Splunk 6.5 from Splunk 6.3.3, the following threat intelligence sources fail to download. Splunk ES was upgraded to 4.5

I checked the server has internet access. I also excluded corresponding URLs from the proxy.

msg="A threat intelligence download has failed" stanza="hailataxii_malware" status="Retrieved document from TAXII feed

msg="A threat intelligence download has failed" stanza="mozilla_public_suffix_list" status="threat list download failed after multiple retries

[ttchorz]

This has been noted as a bag in ES 4.5 version. the bag has been fixed in the 4.5.1 version.

Thanks jwelch for your help on that!

[jwelch_splunk]

Hi,
The first message you have there is actually a success, but is a result of SOLNESS-10559. This can be safely suppressed:
/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/local
Create a file called inputs.conf if one does not already exist. If it does just add:

[configuration_check://confcheck_failed_threat_download]
suppress = (Retrieved document from TAXII feed)

A restart will be required.

Cause: a search string is off here in confcheck_failed_threat_download.py

As far as your second error, I would need more information to try and work out the issue. Perhaps the third party site was unavailable at the time the download was attempted. Is the second error still happening for you?

[ttchorz]

mozilla_public_suffix_list is set to update every 3 hours and it keeps failing every time it tries to update.

I checked the site and it is accessible -

https://publicsuffix.org/list/effective_tld_names.dat

What other information would you like to see to help diagnose?

[jwelch_splunk]

I would do a wget from the Splunk SH itself, and see what the behavior is:

 wget https://publicsuffix.org/list/effective_tld_names.dat

Also is this windows or Linux?

[ttchorz]

I am running Windows 😞

I will provide the output once I figure out an alternative to wget on Windows

[ttchorz]

There seems to be a problem with the certificate trust. It cold be something to do with TLS 1.2

Any ways to bypass the certificate verification in Splunk?

C:\Program Files (x86)\GnuWin32\bin>wget https://publicsuffix.org/list/effective
_tld_names.dat
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2016-10-27 13:20:47--  https://publicsuffix.org/list/effective_tld_names.dat
Resolving publicsuffix.org... 63.245.213.24
Connecting to publicsuffix.org|63.245.213.24|:443... connected.
ERROR: cannot verify publicsuffix.org's certificate, issued by `/C=US/O=DigiCert
 Inc/CN=DigiCert SHA2 Secure Server CA':
  Unable to locally verify the issuer's authority.
ERROR: certificate common name `static-san.mozilla.org' doesn't match requested
host name `publicsuffix.org'.
To connect to publicsuffix.org insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

[jwelch_splunk]

When you do this wget command, it has nothing to do with splunk. Splunk is not in the picture here.

[ttchorz]

Understood. When used wget --no-check-certificate a file was downloaded with no issues.

C:\Program Files (x86)\GnuWin32\bin>wget --no-check-certificate https://publicsu
ffix.org/list/effective_tld_names.dat
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
--2016-10-27 14:06:13--  https://publicsuffix.org/list/effective_tld_names.dat
Resolving publicsuffix.org... 63.245.213.24
Connecting to publicsuffix.org|63.245.213.24|:443... connected.
WARNING: cannot verify publicsuffix.org's certificate, issued by `/C=US/O=DigiCe
rt Inc/CN=DigiCert SHA2 Secure Server CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `static-san.mozilla.org' doesn't match requeste
d host name `publicsuffix.org'.
HTTP request sent, awaiting response... 200 OK
Length: 190800 (186K) [text/plain]
Saving to: `effective_tld_names.dat'

100%[======================================>] 190,800      614K/s   in 0.3s

2016-10-27 14:06:13 (614 KB/s) - `effective_tld_names.dat' saved [190800/190800]

[goodsellt]

Do you have the addresses of the CA servers removed from the proxy as well? It seems possible that the proxy is messing with the data inside of the cert validation request.

I had a similar issue with the Splunk App for AWS (receiving invalid cert response, even though checking the cert with openssl showed it was valid) and I had to force Splunk to ignore the proxy environment variable to correct the problem?