Error when trying to configure Splunk Security Essentials?

[jcas]

I am getting the following errors when trying to Discover Content.

I have tried searching online and am not finding any good reasons for this. I also tried downgrading app versions and got the same errors. We just deployed Splunk recently and have some servers reporting in. Splunk itself has no errors or issues that it is reporting.

[bibibo]

In my case I disabled PYTHONHTTPSVERIFY on splunk-launch.conf and this fixed my issue.

In case it helps, a positive Karma is appreciated.

[Christoph_vW · Accepted solution]

Proper Solution:

edit your /etc/systemd/system/Splunkd.service

in the [Service] section add the following two lines:

Environment=REQUESTS_CA_BUNDLE=/etc/ssl/ca-bundle.pem
Environment=SSL_CERT_FILE=/etc/ssl/ca-bundle.pem

Replace /etc/ssl/ca-bundle.pem with the path to your CA bundle with your own certificate (or keep the path and add your ca certificates to the linux os truststore)

Python standard libs (httplib,urlib3) will use the CA trust bundle specified in SSL_CERT_FILE and the requests library will use REQUESTS_CA_BUNDLE.

One problem left: Splunk will often connect by IP adress instead of using proper hostnames.

For Security Essentials you might have two options (I verified the first):
-include IP 127.0.0.1 in the certificate of the Search Head
or
-in web.conf set mgmtHostPort =<SPLUNK-SEARCH-HEAD-FQDN>:8089
(Security Essentials will read this property in bin/sse_id_enrichment.py and will use it for the connection)

[jcas]

I recently upgraded Splunk and Security Essentials. After the upgrade I followed @Christoph_vW's instructions and I am not seeing any errors.

[jcas]

Does anyone have any thoughts on what the issue might be? Any other information I can provide to help diagnose?

[scelikok]

Hi @jcas,

"Splunk Security Essentials" and "Splunk Enterprise Security Content Update" apps are using some shared resources. If you have "Splunk Enterprise Security Content Update" installed, you can try updating it to the latest version.

[PickleRick]

Also as with any Splunk-supplied apps, the error due to the app's internals is quite unlikely (not impossible though of course) so if you have any proxy in place (like reverse-proxy in case of SH cluster) try and see whether the same happens if you connect directly to a single search-head. Sometimes proxies insert some headers which disrupt proper connectivity.

[jcas]

Hi @scelikok and @PickleRick,

I don't have the "Splunk Enterprise Security Content Update" app and we have a pretty basic setup. One search head and one indexer. One thing I was seeing some information about was self signed certs potentially causing this error. We have a DNS entry that we are using and have a certificate in place that is signed by our internal CA. Again everything is working as expected in Splunk with no certificate errors, but SSE seems to not work. 

When I go to the Data Inventory Overview I get this error, but I thought it was maybe because I just wasn't able to add any content because that page is broken as well.

Please let me know what other details I can provide to help diagnose.

Thanks!

[PickleRick]

There is most obviously some problem with certs.

Unfortunately, custom commands get spawned as separate processes with separate settings and each python program can handle some settings, especially certificate handling, differently.

Since this one uses urllib I expect it uses whatever urllib uses by default but I have no idea which one it is.

You could search for "splunk python custom ca" or "splunk python self-signed error" but I can't point you to exact solution.

[jcas]

Thanks for your reply @PickleRick. Can you provide any direction on where in the SSE config files I would need to add "verify_certificate": False to stop the app from checking the cert? I am aware this isn't ideal but I am not finding any information on making Splunk respect my CA cert that I have added to the OS ca cert file.

Thanks,

[jcas]

I think I found some of the code in SSE that is making the URL request.  Would anyone be able to help me identify where I could pass a "verify=false" flag?

base_url = "https://" + getConfKeyValue('web', 'settings', 'mgmtHostPort')

request = six.moves.urllib.request.Request(base_url + '/services/pullJSON?config=data_inventory',
headers={'Authorization': ('Splunk %s' % self.sessionKey)})
search_results = six.moves.urllib.request.urlopen(request)

[PickleRick]

As far as I remember, that was a parameter for urllib.request.Request(). But that's not a very pretty solution. Besides, if you fiddle with the script, your changes will get overwritten next time you update the app, remember that. And if the original script changes, you'll have to manually synchronize those changes with your version.

So it would be probably better to find where the python script is getting certs from. Unfortunately, I don't know that.

[jcas]

@PickleRick In case you are interested this is the only solution I found for my issue.

So short story ... There is an issue with python3 that it can't/won't respect self signed certs, there is no workaround to make it recognize your CA.

You can create a unique SSL context in each of the SSE config files that don't check SSL

Steps: (reference:https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error?page=1&t...)

cd /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

grep -r "urlopen(request)" /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

In each config file that has a urlopen command you need to modify the following:

At the top of the file:

import ssl
context = ssl._create_unverified_context()

At each urlopen line (search in vi using /urlopen) add context=context like this:

urlopen(request, context=context)

Re-run the grep command to make sure you have modified everything.

As you noted above with all these edits we won't be able to update SSE unless we then go through this process again after updating, but unless Python changes how they do SSL verification we don't really have a choice. Thanks again for your assistance in pointing me in the right direction.

[PickleRick]

Hmm... That sounds hardly believable as every RootCA in the end is a self-signed CA...

[jcas]

They have a pre-defined list of acceptable CAs that you can't modify. I believe it is a list approved or created by Mozilla.

[PickleRick]

The "predefined list" has to be stored somewhere 😉 so even if nothing else worked, it should be possible to add to this store or replace it. The problem is with finding it I assume (or configuring python to use another one)

And why it should have anything to do with Mozilla? o_O

Neither python nor Splunk are Mozilla's products.

[jcas]

Yeah definitely seems weird. Maybe because Firefox is open source? Just guessing though 🙂