topic Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent? in Getting Data In

Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

wrangler2x — Tue, 16 Sep 2014 19:08:43 GMT

A forwarder just up and quit sending logs to my indexer one morning last week. I did not notice until Monday (yesterday) afternoon. I asked the admin there to restart splunkd and when that did not correct the issue, I asked him to send me the logs. I saw several things that don't look good. The first was the second of these two back-to-back forwarder splunkd log entries:

09-15-2014 16:30:31.998 -0700 WARN  DeploymentClient - Phonehome thread is now started.
09-15-2014 16:30:31.998 -0700 WARN  DeploymentClient - Unable to send handshake

The second was in the part of the forwarder splunkd logs where you normally see the TCP ports being initialized:

09-15-2014 16:30:33.165 -0700 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available

And then on the indexer I found this in the splunkd.log:

09-15-2014 15:12:40.669 -0700 ERROR TcpInputProc - Error encountered for connection from src=128.200.xxx.xxx:49266.error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
09-15-2014 15:18:17.960 -0700 ERROR TcpInputProc - Error encountered for connection from src=128.200.xxx.xxx:52373.error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

Any ideas why this has suddenly started happening?

Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

grijhwani — Tue, 16 Sep 2014 19:23:42 GMT

Perhaps the internal SSL certificate has expired. Off the top of my head that is what springs to mind. How long has the installation been in place?
See About securing Splunk with SSL

Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

wrangler2x — Tue, 16 Sep 2014 20:09:49 GMT

Good idea, but I just updated the SSL cert on my indexer and all my forwarders last May. They all have the same server cert, so it does not explain this one forwarder. I just got the forwarder release from the admin that runs it:

#cat /opt/splunk/etc/splunk.version 
VERSION=5.0.3
BUILD=163460
PRODUCT=splunk
PLATFORM=SunOS-sparcv9

I also had him to a ' splunk cmd btool outputs list --debug' and all I see in the email he sent me back was lines from /opt/splunk/etc/system/default/outputs.conf and nothing from the deployment app outputs.conf at all. That is very bizarre.

Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

wrangler2x — Mon, 28 Sep 2020 17:36:16 GMT

Well, it turns out that the /opt/splunk/etc/apps/OIT_OUTPUT_9998 went missing for whatever reason, and that contains the server.conf and outputs.conf in the default directory. The splunk forwarder apparently crashed at some point and those files were not there for it to read. The question is, why did it go away? That's a mystery. This is spec'd-out in the global section of the serverclass.conf file, so every forwarder described there should have a copy of it. #mystery

Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

wrangler2x — Mon, 28 Sep 2020 17:37:21 GMT

The mystery deepens. There is no evidence of crash files, and no evidence of system crash or reboot. There were no configuration changes made to the deployment application for either of these systems, so nothing to make splunk forwarder restart. Yet it had to have in order to notice the missing server.conf and outputs.conf files which were in /opt/splunk/etc/apps/OIT_OUTPUT_9998/default.

Re: Why are we getting "SSL clause is not found or servercert not provided" in forwarder splunkd.log, causing data not to be sent?

wrangler2x — Wed, 24 Sep 2014 17:11:53 GMT