Why did our indexer stop receiving data from all forwarders last night with SSL error "certificate verify failed"?

[crunchit]

Hi all,

Splunk Enterprise 6.2.3 (264376).

Overnight, the indexer stopped receiving data from all of the forwarders. Up until that point, it was receiving data from them all fine without issues.

The splunkd.log on the forwarders shows the following error:

05-26-2016 09:48:15.956 +1000 WARN  DeploymentClient - Unable to send handshake message to deployment server. Error status is: not_connected
05-26-2016 09:48:22.644 +1000 ERROR TcpOutputFd - Connection to host=externalip:9996 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
05-26-2016 09:48:22.644 +1000 WARN  TcpOutputProc - Applying quarantine to idx=externalip:9996 numberOfFailures=2

In the excerpt above I have replaced my external IP with externalip.

We hadn't made any configuration changes before the issue occurred, but once it happened and I saw the error, I went ahead and replaced the default expiring certificates as per the recent email thinking this may have been the problem and restarted, but the issue is still happening.

I have tried updating the outputs.conf file on a forwarder to say sslVerifyServerCert = false but this didn't help, still got the same error.

I inherited this Splunk install when a colleague left a few months ago so I am still learning having never used Splunk before that, haven't been able to figure out what to try next.

Any assistance from someone would be great.

Regards,
William

[jkat54 · Accepted solution]

Maybe one of your certificates have expired.

http://docs.splunk.com/Documentation/Splunk/6.0/Security/ConfigureSplunkforwardingtousesignedcertifi...

Go through the above to try and find certs that may have expired.

Here's a way to get the expiration date:

/opt/splunk/bin/openssl x509 -enddate -noout -in /path/to/cert

[crunchit]

Hi jkat54,
Thanks for the suggestion, I thought I had checked them all but after you mentioned this again I went through a bit more meticulously and found a cert that expired yesterday! Thanks a lot for your assistance.
William

[jkat54]

I converted to an answer so you may mark it as such. Glad to help! See you around!