All Apps and Add-ons

Splunk Add-on for Microsoft - SSL Issue when upgrading to enterprise v9?

brdr
Contributor

We upgraded our Splunk Enterprise from v8.2.5 to v9.0.1.  When we did, it broke the Add-on for Microsoft 365. Every time a connection is made to microsoft we see this SSL error:

SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))

Has anyone run into this before?

 

 

Labels (1)
Tags (1)
1 Solution

jp_at_hb
Explorer

I had the same issues at first and have concluded that something changed with the python libraries between 8.x and 9.x.  I have not done an intense review.. YET. But This is what I found.

CA certs in the following locations are ignored.

3rdparty/certifi/cacert.pem
lib/certifi/cacert.pem

The first just never gets accessed anywhere in the code.

[splunk_ta_o365]$ find . -name \*.py | xargs -ifname grep -iH 3rdparty fname
./bin/splunk_ta_o365_rest_handlers.py:# Adding 3rdparty folder in sys.path for using future module

And the second does not even get checked. Since the certifi lib is not getting included. I created this test to see. By using one of the existing o365 mod inputs. 

from splunk_ta_o365_bootstrap import setup_python_path, run_module
import sys

if __name__ == "__main__":
    try:
        setup_python_path()
        import certifi
        import requests
        print('PATH = ' + ":".join(sys.path))
        print('CAcert = '+ certifi.where())
        print('Checking connection to MS...')
        test = requests.get('https://login.microsoftonline.com')
        print('Connection to MS.')
    except requests.exceptions.SSLError as err:
        print('SSL Error. ' + err)

 

Run it as below. If it fails, it means that you do not have the needed certs.

[splunk_ta_o365]$ splunk cmd python3 bin/testssl.py
PATH = /opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/lib:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/lib/python37.zip:/opt/splunk/lib/python3.7:/opt/splunk/lib/python3.7/lib-dynload:/opt/splunk/lib/python3.7/site-packages:/opt/splunk/lib/python3.7/site-packages/bottle-0.12.19-py3.7.egg
CAcert = /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
Checking connection to MS...
Connection to MS.

It is the system CA cert file that is getting read. If you add your proxy certs, etc to the bottom of it. It should work. 

I have not looked for the exact reason as of yet, nor filed a ticket yet. This was just a work around and NOT the way to do this. 

I hope it helps. 
Cheers.

View solution in original post

brdr
Contributor

NOTE: When you add a tenant through o365 TA gui, the cert (cacert.pem) that it references is in lib/certifi directory. 

0 Karma

Bazza_12
Explorer

So that is a lack of consistency and poor app coding. 

What we found as a work around was that if you append your site certs to /opt/splunk/etc/auth/cacert.pem the app works fine 

If you still use the standard /opt/splunk/etc/auth/server.pem when it expires you will likely face issues with generating a new server.pem (as it uses /opt/splunk/etc/auth/cacert.pem) with complaints about bundle size 

Work around (to the work around) was copy /opt/splunk/etc/auth/cacert.pem to /opt/splunk/etc/auth/cacert.pem.bk before adding your site bundle. You can then revert to this to generate server.pem

0 Karma

kvm
Explorer

@Bazza_12  could you please clarify the part "append your site certs", is this referring to the contents under "splunk_ta_o365/lib/certifi/cacert.pem" ?

0 Karma

Bazza_12
Explorer

The app uses /{splunk_home}/etc/auth/cacert.pem rather than any certifi library cacert.pem

0 Karma

brdr
Contributor

The app is using: /{splunk_home}/splunk/lib/python3.7/site-packages/certifi/cacert.pem which is the issue. The app is not using /{splunk_home}/etc/auth/cacert.pem rather than any certifi library cacert.pem

0 Karma

Bazza_12
Explorer
0 Karma

brdr
Contributor

The section of the troubleshooting guide your refer to is wrong in fixing this app's issue with respect to the certificates.  That section refers to splunk server side authentication not the app. 

0 Karma

jp_at_hb
Explorer

I had the same issues at first and have concluded that something changed with the python libraries between 8.x and 9.x.  I have not done an intense review.. YET. But This is what I found.

CA certs in the following locations are ignored.

3rdparty/certifi/cacert.pem
lib/certifi/cacert.pem

The first just never gets accessed anywhere in the code.

[splunk_ta_o365]$ find . -name \*.py | xargs -ifname grep -iH 3rdparty fname
./bin/splunk_ta_o365_rest_handlers.py:# Adding 3rdparty folder in sys.path for using future module

And the second does not even get checked. Since the certifi lib is not getting included. I created this test to see. By using one of the existing o365 mod inputs. 

from splunk_ta_o365_bootstrap import setup_python_path, run_module
import sys

if __name__ == "__main__":
    try:
        setup_python_path()
        import certifi
        import requests
        print('PATH = ' + ":".join(sys.path))
        print('CAcert = '+ certifi.where())
        print('Checking connection to MS...')
        test = requests.get('https://login.microsoftonline.com')
        print('Connection to MS.')
    except requests.exceptions.SSLError as err:
        print('SSL Error. ' + err)

 

Run it as below. If it fails, it means that you do not have the needed certs.

[splunk_ta_o365]$ splunk cmd python3 bin/testssl.py
PATH = /opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/lib:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/lib/python37.zip:/opt/splunk/lib/python3.7:/opt/splunk/lib/python3.7/lib-dynload:/opt/splunk/lib/python3.7/site-packages:/opt/splunk/lib/python3.7/site-packages/bottle-0.12.19-py3.7.egg
CAcert = /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
Checking connection to MS...
Connection to MS.

It is the system CA cert file that is getting read. If you add your proxy certs, etc to the bottom of it. It should work. 

I have not looked for the exact reason as of yet, nor filed a ticket yet. This was just a work around and NOT the way to do this. 

I hope it helps. 
Cheers.

brdr
Contributor

Thank you for posting the solution!! When we upgraded from v8.2.5 to v9.0.1 of the enterprise this was the only app that didn't work post upgrade. We have half dozen other apps that didn't require this work around with using a different cacert.pem. 

0 Karma

SinghK
Builder

if you were using a ssl certificate then check it is still there like validity etc. and make sure it is still owned by splunk user

0 Karma

Bazza_12
Explorer

I can confirm that the cert is still correct & owned by Splunk user 

0 Karma

SinghK
Builder

What type of cert is it ?internal  CA signed or splunk self signed ones?

0 Karma

SinghK
Builder

if its a CA signed cert then are your o365 inputs still using the same cert. The path in inputs where it requests for ssl info is that correct?

0 Karma

Bazza_12
Explorer

if its a CA signed cert then are your o365 inputs still using the same cert. - Yes it is had verification 

 

The path in inputs where it requests for ssl info is that correct? - This is the app 
Splunk Add-on for Microsoft Office 365 | Splunkbase

The places I have found that cacert.pem is specified in the app are 
.../3rdparty/certifi/cacert.pem
.../lib/certifi/cacert.pem

0 Karma

jp_at_hb
Explorer

correct under the o365 app. The o365 is not picking it up. Even if you force sys.path to the app dir it still ignores it and chooses the system one. 

certifi IMO needs to also have support for other certs paths. As it clearly states it does not. Yet everyone still uses it 🙂

When you use internal CAs out to M$. you need to this to work. especially when you are behind something like a zscaler, where they wont just pass that traffic through

0 Karma

SinghK
Builder

The cert will not be in app.

it is always under $SPLUNK_HOME/etc/auth

and you point the app in that direction

check your cert 

check the cert with this :

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in file.cer

make sure your $SPLUNK_HOME is set correctly or just set the path replacing $SPLUNK_HOME

0 Karma

jp_at_hb
Explorer

correct. but NOT CA certs.

0 Karma

brdr
Contributor

Where in the inputs will you find (or set) the path to the cert?

Tags (1)
0 Karma

jp_at_hb
Explorer

its not in the inputs.

And you cannot set it.

its in one of two places

  • Under the applications called cacert.pem
  • or in the Splunk system dirs. /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem  😞

The first one is getting ignored after we upgraded to 9

0 Karma

Bazza_12
Explorer

Can I ask what changed in v9 that enforced this? As the error directly correlates with our upgrade date 

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...