mntbighker - did this answer actually help you? I sense sarcasm (good for you if it was).
At any rate, I'm having the same issue now. I tracked it back to expired certs - 3 years to the day of installing Splunk, all my forwarders have crapped out with the same errors you are seeing.
I have regenerated the $SPLUNK_HOME/etc/auth/server.pem on my master Splunk server using
splunk createssl server-cert
I am still getting the errors.
When we installed Splunk and the forwarders, all of this was generated automatically behind the scenes (or at least the majority).
Here's the problems I have with this issue:
I'm seeing tons of users on Splunk.com reporting this issue - some as old as 2008 at least
We received no warning from Splunk - this is kind of important - why isn't Splunk checking itself for this?
This is internal to the Splunk tool itself - why does it not auto-generate new certs (if you're using self signed certs anyway).
Why are there no clear documents on how to fix this? The forums are nice, but this is a problem that ALL of Splunk's users will encounter at some point
Why is Splunk not putting effort into making this better / fixing this? To get to this error, you had to be a paying customer for 3+ years. You should really want to keep us happy.
I want a patch, or a very clear path to fixing this. I have a dozen forwarders that have been silent for a week before anyone noticed.
... View more