All Apps and Add-ons

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676) when trying to use add-on.

TWiseOne
Path Finder

I have this add-on installed on a HFW, just installed the 0Gb ingestion license (to allow KVStore to run) but I am now getting SSL errors?!?

I can't see any SSL configuration element in the app/docs so am not sure where to start.

Full Error Message:

2017-10-16 09:36:03,597 ERROR pid=869 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace.py", line 72, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/input_module_ms_o365_message_trace.py", line 57, in collect_events
r = requests.get(microsoft_trace_url, auth=requests.auth.HTTPBasicAuth(global_microsoft_office_365_username, global_microsoft_office_365_password))
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)

Why is this failing? And where do I start to fix it?

Thanks.

0 Karma
1 Solution

TWiseOne
Path Finder

@garethatiag Kudos to you!!

The issue was SSL inspection.

I was advised that there was a rule to turn off SSL inspection on anything destined for office365.com but internally that was not the case. This is the problem when using 3rd parties to manage your perimeter.

All fixed now, thanks.

View solution in original post

janith
Engager

@TWiseOne  Just wanted to contribute to this discussion although it's bit old. I recently came up with the same error while working with the Splunk o365 add-on and the way I fixed it by editing the following file.

/opt/splunk/etc/apps/splunk_ta_o365/bin/3rdparty/certify/cacert.pem

I had to add the public key details of the Certificate used by the proxy server in it.

 

 

0 Karma

rgraham29975
New Member

Thanks! We don't do ask inspection. my browser has the same problem which states the security cert is not trusted. I can tell my browser to ignore, but how would i do that in Splunk?

0 Karma

TWiseOne
Path Finder

I need a bit more information to determine where this could be going wrong. I have not used the technology your TA queries so can you let me know:

  • Is the TA on a HFW or Search Head?
  • Is this On-Prem to On-Prem connection from the TA to the cisco API or On-Prem to Cloud API?
  • Does the TA make you create/import certificates for the inputs?
  • When you say your browser doesn't trust the certificate, what are you connecting to when you get that error as I suspect the 2 certificates are completely separate?
0 Karma

rgraham29975
New Member

Is the TA on a HFW or Search Head?
Both only have one server.

Is this On-Prem to On-Prem connection from the TA to the cisco API or On-Prem to Cloud API?
On-Prem only.

Does the TA make you create/import certificates for the inputs?
yes.

When you say your browser doesn't trust the certificate, what are you connecting to when you get that error as I suspect the 2 certificates are completely separate?
I connect to the Tetration server. According to setup I first generate an API key and download the JSON file. Then copy and paste the key and password into the splunk setup script.

0 Karma

rgraham29975
New Member

Hi TWiseOne,
I made the following change and I am able to collect inventory data. I am still getting a 403 on collecting sensor data. I believe that is a problem with credentials and will continue to look into it.

Here is the change I made to the python script:
rc = RestClient('https://' + url, api_key=apikey,api_secret=apisecret, verify=True) ß------ CHANGE TO verify=False

Ron

0 Karma

akshatj2
Path Finder

do you have the path for this python script?

0 Karma

rgraham29975
New Member

How was this fixed? Getting the same error message:
04-17-2018 08:19:20.584 -0700 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA_cisco-tetration/bin/collectTetration.py -inventory" requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)

0 Karma

thambisetty
Super Champion

Hi,

As mentioned in my previous comment, if you are using proxy for this connection, the connection will be inspected by proxy and there will be some certificate mismatch while making the connection to server. To avoid this, get certificate for your client machine (splunk server) from the CA that has issued certificate to Proxy server and install that certificate in your client machine certificate store.

I am happy to help you on this further if required.

————————————
If this helps, give a like below.
0 Karma

thambisetty
Super Champion

Then you need to disable ssl inspection on proxy as twiseone mentioned.

————————————
If this helps, give a like below.
0 Karma

lim2
Path Finder

I got the same SSLError when company's proxy was replaced with firewall rule to https://reports.office365.com. Tried the steps in https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error
When I added my company's public root ca to /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/requests/cacert.pem, SSL error was resolved.

0 Karma

TWiseOne
Path Finder

Hi,

I see you are using a different TA to the one which my original error was being reported on. However, the issue I had was due to a network device, between the Splunk Server and the cloud API, that was doing SSL Inspection on the connection when it wasn't meant to be.

Once a rule was added to not perform SSL inspection, the connection was established we had no further issues.

0 Karma

thambisetty
Super Champion

Hi,

If you have installed MSCS in linux problem is with Certificate that is installed in client machine( where MSCS is installed) .

Try to get Internal certificate for your client machine from where your proxy got Internal certifitcate. so that root CA will be same for the both client and proxy. your connection will be trusted you will get rid of from the ssl certificate error.

————————————
If this helps, give a like below.
0 Karma

TWiseOne
Path Finder

@garethatiag Kudos to you!!

The issue was SSL inspection.

I was advised that there was a rule to turn off SSL inspection on anything destined for office365.com but internally that was not the case. This is the problem when using 3rd parties to manage your perimeter.

All fixed now, thanks.

View solution in original post

gjanders
SplunkTrust
SplunkTrust

Have you configured the add on to retrieve data from the MS azure/office 365 website?
Are you using a proxy? And if so does your proxy do any man in the middle SSL changes?

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

TWiseOne
Path Finder

Thanks for your response.

There is no option for where to retrieve data from in the configuration page, only Account Name, Username & Password.

We previously had the other O365 app which successfully retrieved items but not mail tracking logs which is why we changed to this add-on. However it has been troublesome to get working.

I am not aware of any MIM SSL Changes that are done by a proxy but can ask.

0 Karma

gjanders
SplunkTrust
SplunkTrust

It appears to use https://reports.office365.com by default and it does have a debug level of logging that might provide more hints.

It does not appear to be an official Splunk supported app...
I also did not see a way to add a proxy in unless I missed it, so I'm assuming the server it's running on has a default proxy configured in Splunk or has direct internet access.

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

hardikJsheth
Motivator

The error is coming because of failed certificate verification. Can you check if add-on is making https call to url which is using self signed certificate?

If its a self signed certificate, then you either need to use verify=False in your python script when making api calls or put public key of certificate in your trust store.

0 Karma

TWiseOne
Path Finder

Thanks for the response.

This all just seems a bit long winded to get a simple add-on that should work OOTB to work as it professes to do.

I will look into it again but as @garethatiag stated it does't appear to be a Splunk supported app so not sure how I can get support from the developer.

0 Karma

TWiseOne
Path Finder

@garethatiag I have had the relevant team check for SSL inspection on the traffic from my HFW and it is turned off for all *office365.com sites so should be allowed through to verify.

I don't understand the certificate mechanism that this app is using, should I see a certificate download or should I have a certificate installed already on my HFW, and if so what Certificate is required and where is it placed on a *nix HFW?

The instructions on this app make it seem like it can just be installed and work but I am finding there are quite a few gotchas with no direct communication to the team responsible for making it!

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.