topic Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9? in All Apps and Add-ons

Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

brdr — Tue, 14 Feb 2023 22:19:01 GMT

We upgraded our Splunk Enterprise from v8.2.5 to v9.0.1. When we did, it broke the Add-on for Microsoft 365. Every time a connection is made to microsoft we see this SSL error:

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))

Has anyone run into this before?

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 03:52:16 GMT

Same here interested in any feedback

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 04:59:30 GMT

if you were using a ssl certificate then check it is still there like validity etc. and make sure it is still owned by splunk user

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 05:04:41 GMT

I can confirm that the cert is still correct & owned by Splunk user

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 05:07:01 GMT

What type of cert is it ?internal CA signed or splunk self signed ones?

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 05:08:31 GMT

Our CA signed cert

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 05:08:52 GMT

if its a CA signed cert then are your o365 inputs still using the same cert. The path in inputs where it requests for ssl info is that correct?

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 05:10:58 GMT

As i was saying check if the inputs have the correct path to cert and error is very specific

"verify failed: self signed certificate in certificate chain"

inputs are detecting a self signed cert instead of CA one.

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 05:14:25 GMT

check the cert with this :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in file.cer

make sure your $SPLUNK_HOME is set correctly or just set the path replacing $SPLUNK_HOME

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 05:15:04 GMT

if its a CA signed cert then are your o365 inputs still using the same cert. - Yes it is had verification

The path in inputs where it requests for ssl info is that correct? - This is the app
Splunk Add-on for Microsoft Office 365 | Splunkbase

The places I have found that cacert.pem is specified in the app are
.../3rdparty/certifi/cacert.pem
.../lib/certifi/cacert.pem

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 05:23:07 GMT

I can verify that the cert in the apps locations is valid

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 05:23:15 GMT

The cert will not be in app.

it is always under $SPLUNK_HOME/etc/auth

and you point the app in that direction

check your cert

check the cert with this :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in file.cer

make sure your $SPLUNK_HOME is set correctly or just set the path replacing $SPLUNK_HOME

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

Bazza_12 — Fri, 10 Mar 2023 06:33:35 GMT

Can I ask what changed in v9 that enforced this? As the error directly correlates with our upgrade date

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

SinghK — Fri, 10 Mar 2023 07:55:01 GMT

I have not found any problems with v9 at all and i have o365 data integration. so cant comment on that you need to check logs. if still cant find anything open a case with support.

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

jp_at_hb — Mon, 08 May 2023 20:32:43 GMT

I had the same issues at first and have concluded that something changed with the python libraries between 8.x and 9.x. I have not done an intense review.. YET. But This is what I found.

CA certs in the following locations are ignored.

3rdparty/certifi/cacert.pem
lib/certifi/cacert.pem

The first just never gets accessed anywhere in the code.

[splunk_ta_o365]$ find . -name \*.py | xargs -ifname grep -iH 3rdparty fname
./bin/splunk_ta_o365_rest_handlers.py:# Adding 3rdparty folder in sys.path for using future module

And the second does not even get checked. Since the certifi lib is not getting included. I created this test to see. By using one of the existing o365 mod inputs.

from splunk_ta_o365_bootstrap import setup_python_path, run_module import sys if __name__ == "__main__": try: setup_python_path() import certifi import requests print('PATH = ' + ":".join(sys.path)) print('CAcert = '+ certifi.where()) print('Checking connection to MS...') test = requests.get('https://login.microsoftonline.com') print('Connection to MS.') except requests.exceptions.SSLError as err: print('SSL Error. ' + err)

Run it as below. If it fails, it means that you do not have the needed certs.

[splunk_ta_o365]$ splunk cmd python3 bin/testssl.py
PATH = /opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/lib:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/lib/python37.zip:/opt/splunk/lib/python3.7:/opt/splunk/lib/python3.7/lib-dynload:/opt/splunk/lib/python3.7/site-packages:/opt/splunk/lib/python3.7/site-packages/bottle-0.12.19-py3.7.egg
CAcert = /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
Checking connection to MS...
Connection to MS.

It is the system CA cert file that is getting read. If you add your proxy certs, etc to the bottom of it. It should work.

I have not looked for the exact reason as of yet, nor filed a ticket yet. This was just a work around and NOT the way to do this.

I hope it helps.
Cheers.

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

brdr — Tue, 22 Aug 2023 18:06:59 GMT

Where in the inputs will you find (or set) the path to the cert?

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

jp_at_hb — Tue, 22 Aug 2023 18:23:17 GMT

correct under the o365 app. The o365 is not picking it up. Even if you force sys.path to the app dir it still ignores it and chooses the system one.

certifi IMO needs to also have support for other certs paths. As it clearly states it does not. Yet everyone still uses it 🙂

When you use internal CAs out to M$. you need to this to work. especially when you are behind something like a zscaler, where they wont just pass that traffic through

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

jp_at_hb — Tue, 22 Aug 2023 18:24:04 GMT

correct. but NOT CA certs.

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

brdr — Tue, 22 Aug 2023 19:11:02 GMT

Thank you for posting the solution!! When we upgraded from v8.2.5 to v9.0.1 of the enterprise this was the only app that didn't work post upgrade. We have half dozen other apps that didn't require this work around with using a different cacert.pem.

Re: Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

brdr — Thu, 24 Aug 2023 11:09:04 GMT

NOTE: When you add a tenant through o365 TA gui, the cert (cacert.pem) that it references is in lib/certifi directory.