We upgraded our Splunk Enterprise from v8.2.5 to v9.0.1. When we did, it broke the Add-on for Microsoft 365. Every time a connection is made to microsoft we see this SSL error:
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1106)')))
Has anyone run into this before?
I had the same issues at first and have concluded that something changed with the python libraries between 8.x and 9.x. I have not done an intense review.. YET. But This is what I found.
CA certs in the following locations are ignored.
3rdparty/certifi/cacert.pem
lib/certifi/cacert.pem
The first just never gets accessed anywhere in the code.
[splunk_ta_o365]$ find . -name \*.py | xargs -ifname grep -iH 3rdparty fname
./bin/splunk_ta_o365_rest_handlers.py:# Adding 3rdparty folder in sys.path for using future module
And the second does not even get checked. Since the certifi lib is not getting included. I created this test to see. By using one of the existing o365 mod inputs.
from splunk_ta_o365_bootstrap import setup_python_path, run_module
import sys
if __name__ == "__main__":
try:
setup_python_path()
import certifi
import requests
print('PATH = ' + ":".join(sys.path))
print('CAcert = '+ certifi.where())
print('Checking connection to MS...')
test = requests.get('https://login.microsoftonline.com')
print('Connection to MS.')
except requests.exceptions.SSLError as err:
print('SSL Error. ' + err)
Run it as below. If it fails, it means that you do not have the needed certs.
[splunk_ta_o365]$ splunk cmd python3 bin/testssl.py
PATH = /opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/lib:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/etc/apps/splunk_ta_o365/bin:/opt/splunk/lib/python37.zip:/opt/splunk/lib/python3.7:/opt/splunk/lib/python3.7/lib-dynload:/opt/splunk/lib/python3.7/site-packages:/opt/splunk/lib/python3.7/site-packages/bottle-0.12.19-py3.7.egg
CAcert = /opt/splunk/lib/python3.7/site-packages/certifi/cacert.pem
Checking connection to MS...
Connection to MS.
It is the system CA cert file that is getting read. If you add your proxy certs, etc to the bottom of it. It should work.
I have not looked for the exact reason as of yet, nor filed a ticket yet. This was just a work around and NOT the way to do this.
I hope it helps.
Cheers.
I have not found any problems with v9 at all and i have o365 data integration. so cant comment on that you need to check logs. if still cant find anything open a case with support.
Our CA signed cert
As i was saying check if the inputs have the correct path to cert and error is very specific
"verify failed: self signed certificate in certificate chain"
inputs are detecting a self signed cert instead of CA one.
check the cert with this :
$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in file.cer
make sure your $SPLUNK_HOME is set correctly or just set the path replacing $SPLUNK_HOME
I can verify that the cert in the apps locations is valid
Same here interested in any feedback