All Apps and Add-ons

Error when trying to configure Splunk Security Essentials?

jcas
Explorer

I am getting the following errors when trying to Discover Content.

jcas_0-1676926086090.png

I have tried searching online and am not finding any good reasons for this. I also tried downgrading app versions and got the same errors. We just deployed Splunk recently and have some servers reporting in. Splunk itself has no errors or issues that it is reporting.

Labels (1)
0 Karma
1 Solution

jcas
Explorer

@PickleRick In case you are interested this is the only solution I found for my issue.

So short story ... There is an issue with python3 that it can't/won't respect self signed certs, there is no workaround to make it recognize your CA.

You can create a unique SSL context in each of the SSE config files that don't check SSL

Steps: (reference:https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error?page=1&t...)

cd /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

grep -r "urlopen(request)" /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

In each config file that has a urlopen command you need to modify the following:

At the top of the file:

import ssl
context = ssl._create_unverified_context()

At each urlopen line (search in vi using /urlopen) add context=context like this:

urlopen(request, context=context)

Re-run the grep command to make sure you have modified everything.

As you noted above with all these edits we won't be able to update SSE unless we then go through this process again after updating, but unless Python changes how they do SSL verification we don't really have a choice. Thanks again for your assistance in pointing me in the right direction.

 

 

 

View solution in original post

0 Karma

Christoph_vW
Loves-to-Learn Lots

Proper Solution:

edit your /etc/systemd/system/Splunkd.service

in the [Service] section add the following two lines:

Environment=REQUESTS_CA_BUNDLE=/etc/ssl/ca-bundle.pem
Environment=SSL_CERT_FILE=/etc/ssl/ca-bundle.pem

Replace /etc/ssl/ca-bundle.pem with the path to your CA bundle with your own certificate (or keep the path and add your ca certificates to the linux os truststore)

 

Python standard libs (httplib,urlib3) will use the CA trust bundle specified in SSL_CERT_FILE and the requests library will use REQUESTS_CA_BUNDLE.

 

One problem left: Splunk will often connect by IP adress instead of using proper hostnames.

For Security Essentials you might have two options (I verified the first):
-include IP 127.0.0.1 in the certificate of the Search Head
or
-in web.conf set mgmtHostPort =<SPLUNK-SEARCH-HEAD-FQDN>:8089
(Security Essentials will read this property in bin/sse_id_enrichment.py and will use it for the connection)

 

 

0 Karma

jcas
Explorer

Does anyone have any thoughts on what the issue might be? Any other information I can provide to help diagnose?

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @jcas,

"Splunk Security Essentials" and "Splunk Enterprise Security Content Update" apps are using some shared resources. If you have "Splunk Enterprise Security Content Update" installed, you can try updating it to the latest version.

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Also as with any Splunk-supplied apps, the error due to the app's internals is quite unlikely (not impossible though of course) so if you have any proxy in place (like reverse-proxy in case of SH cluster) try and see whether the same happens if you connect directly to a single search-head. Sometimes proxies insert some headers which disrupt proper connectivity.

0 Karma

jcas
Explorer

Hi @scelikok and @PickleRick,

I don't have the "Splunk Enterprise Security Content Update" app and we have a pretty basic setup. One search head and one indexer. One thing I was seeing some information about was self signed certs potentially causing this error. We have a DNS entry that we are using and have a certificate in place that is signed by our internal CA. Again everything is working as expected in Splunk with no certificate errors, but SSE seems to not work. 

When I go to the Data Inventory Overview I get this error, but I thought it was maybe because I just wasn't able to add any content because that page is broken as well.

jcas_0-1676983081141.png

Please let me know what other details I can provide to help diagnose.

Thanks!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

There is most obviously some problem with certs.

Unfortunately, custom commands get spawned as separate processes with separate settings and each python program can handle some settings, especially certificate handling, differently.

Since this one uses urllib I expect it uses whatever urllib uses by default but I have no idea which one it is.

You could search for "splunk python custom ca" or "splunk python self-signed error" but I can't point you to exact solution.

0 Karma

jcas
Explorer

Thanks for your reply @PickleRick. Can you provide any direction on where in the SSE config files I would need to add "verify_certificate": False to stop the app from checking the cert? I am aware this isn't ideal but I am not finding any information on making Splunk respect my CA cert that I have added to the OS ca cert file.

Thanks,

0 Karma

jcas
Explorer

I think I found some of the code in SSE that is making the URL request.  Would anyone be able to help me identify where I could pass a "verify=false" flag?

base_url = "https://" + getConfKeyValue('web', 'settings', 'mgmtHostPort')

request = six.moves.urllib.request.Request(base_url + '/services/pullJSON?config=data_inventory',
headers={'Authorization': ('Splunk %s' % self.sessionKey)})
search_results = six.moves.urllib.request.urlopen(request)

0 Karma

PickleRick
SplunkTrust
SplunkTrust

As far as I remember, that was a parameter for urllib.request.Request(). But that's not a very pretty solution. Besides, if you fiddle with the script, your changes will get overwritten next time you update the app, remember that. And if the original script changes, you'll have to manually synchronize those changes with your version.

So it would be probably better to find where the python script is getting certs from. Unfortunately, I don't know that.

0 Karma

jcas
Explorer

@PickleRick In case you are interested this is the only solution I found for my issue.

So short story ... There is an issue with python3 that it can't/won't respect self signed certs, there is no workaround to make it recognize your CA.

You can create a unique SSL context in each of the SSE config files that don't check SSL

Steps: (reference:https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error?page=1&t...)

cd /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

grep -r "urlopen(request)" /opt/splunk/etc/apps/Splunk_Security_Essentials/bin

In each config file that has a urlopen command you need to modify the following:

At the top of the file:

import ssl
context = ssl._create_unverified_context()

At each urlopen line (search in vi using /urlopen) add context=context like this:

urlopen(request, context=context)

Re-run the grep command to make sure you have modified everything.

As you noted above with all these edits we won't be able to update SSE unless we then go through this process again after updating, but unless Python changes how they do SSL verification we don't really have a choice. Thanks again for your assistance in pointing me in the right direction.

 

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Hmm... That sounds hardly believable as every RootCA in the end is a self-signed CA...

0 Karma

jcas
Explorer

They have a pre-defined list of acceptable CAs that you can't modify. I believe it is a list approved or created by Mozilla.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The "predefined list" has to be stored somewhere 😉 so even if nothing else worked, it should be possible to add to this store or replace it. The problem is with finding it I assume (or configuring python to use another one)

And why it should have anything to do with Mozilla? o_O

Neither python nor Splunk are Mozilla's products.

0 Karma

jcas
Explorer

Yeah definitely seems weird. Maybe because Firefox is open source? Just guessing though 🙂

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...