Security

Expired SSL Cert?

mntbighker
Path Finder

It seems that on Aug. 15th my vanilla Splunk SSL cert expired:

09-07-2012 17:28:38.987 -0700 ERROR TcpInputProc - Error encountered for connection from src=xxx.xxx.xxx.xxx:3353. error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired

I have never wanted or needed to mess with my own cert. Our only requirement has been to encrypt over the wire. So all my log aggregation it seems came to a grinding halt 3 weeks ago. Is Splunk going to publish a process to fix this or will it be an excruciating manual process involving every host including the forwarders and the server? I'm running 4.3.3 on the server.

Tags (1)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Unless the rootCA has expired, you only need a new server certificate. Use splunk createssl server-cert to create a new one certificate to replace the one you are using. You don't say how you have configured anything, but presumably you're using the default server.pem on the server, and no certificates on the client. Of course if you did enable client certificate verification, those will have to be regenerated as well.

View solution in original post

jd260
Engager

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n ${server_name} -c ${server_name}.fqdn
Then cp ${server_name}.pem to server.pem

mweissha
Path Finder

This answer is far closer to an actually helpful response. This command, and looking at the help for splunk cli ./bin/splunk help createssl, was what eliminated my ssl errors. Thanks jd260!

0 Karma

reswob4
Builder

Thanks @jd260. If I had found this answer this morning, it would have saved me hours of work.

Why this isn't in the Splunk docs is a mystery.

0 Karma

edekker
Explorer

Thanks. This saved me a lot of time. I swore I had it noted down somewhere, but alas..

0 Karma

mntbighker
Path Finder

I have been asking them to support SSL enabled forwarders in the web GUI and NOTHING has improved in many versions. In fact it takes major effort to make them understand what I mean, so I must presume that most people never bother with SSL (weird). Anyway, if they have this general attitude, then this situation about no support on expiring certs does'nt not surprise me in the least.

0 Karma

mdaedalus
Explorer

mntbighker - did this answer actually help you? I sense sarcasm (good for you if it was).

At any rate, I'm having the same issue now. I tracked it back to expired certs - 3 years to the day of installing Splunk, all my forwarders have crapped out with the same errors you are seeing.

I have regenerated the $SPLUNK_HOME/etc/auth/server.pem on my master Splunk server using

splunk createssl server-cert

I am still getting the errors.

When we installed Splunk and the forwarders, all of this was generated automatically behind the scenes (or at least the majority).

Here's the problems I have with this issue:

  • I'm seeing tons of users on Splunk.com reporting this issue - some as old as 2008 at least
  • We received no warning from Splunk - this is kind of important - why isn't Splunk checking itself for this?
  • This is internal to the Splunk tool itself - why does it not auto-generate new certs (if you're using self signed certs anyway).
  • Why are there no clear documents on how to fix this? The forums are nice, but this is a problem that ALL of Splunk's users will encounter at some point
  • Why is Splunk not putting effort into making this better / fixing this? To get to this error, you had to be a paying customer for 3+ years. You should really want to keep us happy.

I want a patch, or a very clear path to fixing this. I have a dozen forwarders that have been silent for a week before anyone noticed.

mdaedalus
Explorer

This isn't an answer - it's a question (and continuation of the other question asked by the original poster).

I tried to convert this answer to a question and keep getting a 500 error from the web server.

gkanapathy
Splunk Employee
Splunk Employee

Unless the rootCA has expired, you only need a new server certificate. Use splunk createssl server-cert to create a new one certificate to replace the one you are using. You don't say how you have configured anything, but presumably you're using the default server.pem on the server, and no certificates on the client. Of course if you did enable client certificate verification, those will have to be regenerated as well.

mweissha
Path Finder

I downvoted this post because this is not enough of an answer. according to the official docs of the cli command [./bin/splunk help createssl] there are 2 flags that are required to be filled in (-d for directory of cert and -n for the name)

this answer does not also advise to backup your original cert or where to store it after you generate.

0 Karma

mntbighker
Path Finder

Thanks for the help

0 Karma

Michael
Contributor

heh, ya "thanks for the help". I'm looking for this answer, and the best I can find are half-answers from 2012.

My guess is not many people are even paying attention to this. In our case, the expired certs are setting off alerts with other IDS/IPS sensors, so we want to address it. Even the /splunk help createssl documentation sucks, including line formatting and spacing that's all jacked up -- signs that no one is actually putting any energy into improving this situation.

Folks, when someone asks how to do something, as long as it's not completely in left-field, please answer it completely, or not at all. Assume defaults if information is omitted (avoid: "well, you didn't say what O/S, or your server's name, or your blood-type..."). For example the "answer above" does not work, there are other parameters that are required, and yet, it's the "accepted answer". Gah! Also the link to RTFM that discusses certs in general terms, does NOT explain how to renew a cert. Gah! Don't try to up your "answer" count with links to docs that discuss the issue at 20,000 feet. It's a question, looking for an answer. Period.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...