Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

[brdr]

We upgraded our Splunk Enterprise from v8.2.5 to v9.0.1.  When we did, it broke the Add-on for Microsoft 365. Every time a connection is made to microsoft we see this SSL error:

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))

Has anyone run into this before?

[brdr]

NOTE: When you add a tenant through o365 TA gui, the cert (cacert.pem) that it references is in lib/certifi directory. 

[Bazza_12]

So that is a lack of consistency and poor app coding. 

What we found as a work around was that if you append your site certs to /opt/splunk/etc/auth/cacert.pem the app works fine 

If you still use the standard /opt/splunk/etc/auth/server.pem when it expires you will likely face issues with generating a new server.pem (as it uses /opt/splunk/etc/auth/cacert.pem) with complaints about bundle size 

Work around (to the work around) was copy /opt/splunk/etc/auth/cacert.pem to /opt/splunk/etc/auth/cacert.pem.bk before adding your site bundle. You can then revert to this to generate server.pem

[kvm]

@Bazza_12  could you please clarify the part "append your site certs", is this referring to the contents under "splunk_ta_o365/lib/certifi/cacert.pem" ?

[Bazza_12]

The app uses /{splunk_home}/etc/auth/cacert.pem rather than any certifi library cacert.pem

[brdr]

The app is using: /{splunk_home}/splunk/lib/python3.7/site-packages/certifi/cacert.pem which is the issue. The app is not using /{splunk_home}/etc/auth/cacert.pem rather than any certifi library cacert.pem

[Bazza_12]

The official docs trouble shooting page begs to differ 

https://docs.splunk.com/Documentation/AddOns/released/MSO365/Troubleshooting#:~:text=Azure%20Active%...

[brdr]

The section of the troubleshooting guide your refer to is wrong in fixing this app's issue with respect to the certificates.  That section refers to splunk server side authentication not the app. 

[jp_at_hb · Accepted solution]

I had the same issues at first and have concluded that something changed with the python libraries between 8.x and 9.x.  I have not done an intense review.. YET. But This is what I found.

CA certs in the following locations are ignored.

3rdparty/certifi/cacert.pem
lib/certifi/cacert.pem

The first just never gets accessed anywhere in the code.

[splunk_ta_o365]$ find . -name \*.py | xargs -ifname grep -iH 3rdparty fname
./bin/splunk_ta_o365_rest_handlers.py:# Adding 3rdparty folder in sys.path for using future module

And the second does not even get checked. Since the certifi lib is not getting included. I created this test to see. By using one of the existing o365 mod inputs. 

from splunk_ta_o365_bootstrap import setup_python_path, run_module
import sys

if __name__ == "__main__":
    try:
        setup_python_path()
        import certifi
        import requests
        print('PATH = ' + ":".join(sys.path))
        print('CAcert = '+ certifi.where())
        print('Checking connection to MS...')
        test = requests.get('https://login.microsoftonline.com')
        print('Connection to MS.')
    except requests.exceptions.SSLError as err:
        print('SSL Error. ' + err)

Run it as below. If it fails, it means that you do not have the needed certs.

[splunk_ta_o365]$ splunk cmd python3 bin/testssl.py
PATH = /opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/lib:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/lib/python37.zip:/opt/splunk/lib/python3.7:/opt/splunk/lib/python3.7/lib-dynload:/opt/splunk/lib/python3.7/site-packages:/opt/splunk/lib/python3.7/site-packages/bottle-0.12.19-py3.7.egg
CAcert = /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
Checking connection to MS...
Connection to MS.

It is the system CA cert file that is getting read. If you add your proxy certs, etc to the bottom of it. It should work. 

I have not looked for the exact reason as of yet, nor filed a ticket yet. This was just a work around and NOT the way to do this. 

I hope it helps. 
Cheers.

[brdr]

Thank you for posting the solution!! When we upgraded from v8.2.5 to v9.0.1 of the enterprise this was the only app that didn't work post upgrade. We have half dozen other apps that didn't require this work around with using a different cacert.pem. 

[SinghK]

if you were using a ssl certificate then check it is still there like validity etc. and make sure it is still owned by splunk user

[Bazza_12]

I can confirm that the cert is still correct & owned by Splunk user 

[SinghK]

What type of cert is it ?internal  CA signed or splunk self signed ones?

[SinghK]

if its a CA signed cert then are your o365 inputs still using the same cert. The path in inputs where it requests for ssl info is that correct?

[Bazza_12]

if its a CA signed cert then are your o365 inputs still using the same cert. - Yes it is had verification 

The path in inputs where it requests for ssl info is that correct? - This is the app 
Splunk Add-on for Microsoft Office 365 | Splunkbase

The places I have found that cacert.pem is specified in the app are 
.../3rdparty/certifi/cacert.pem
.../lib/certifi/cacert.pem

[jp_at_hb]

correct under the o365 app. The o365 is not picking it up. Even if you force sys.path to the app dir it still ignores it and chooses the system one. 

certifi IMO needs to also have support for other certs paths. As it clearly states it does not. Yet everyone still uses it 🙂

When you use internal CAs out to M$. you need to this to work. especially when you are behind something like a zscaler, where they wont just pass that traffic through

[SinghK]

The cert will not be in app.

it is always under $SPLUNK_HOME/etc/auth

and you point the app in that direction

check your cert 

check the cert with this :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in file.cer

make sure your $SPLUNK_HOME is set correctly or just set the path replacing $SPLUNK_HOME

[jp_at_hb]

correct. but NOT CA certs.

[brdr]

Where in the inputs will you find (or set) the path to the cert?

[jp_at_hb]

its not in the inputs.

And you cannot set it.

its in one of two places

• Under the applications called cacert.pem
• or in the Splunk system dirs. /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem  😞

The first one is getting ignored after we upgraded to 9

[Bazza_12]

Can I ask what changed in v9 that enforced this? As the error directly correlates with our upgrade date