Splunk default SSL certificate expired for SSL communication between forwarders to indexers. "certificate expired"

[Masa]

Splunk default SSL certificate expired for SSL communication between forwarders to indexers

We've been using Splunk for over three years since 2010. We use Splunk's default SSL certificate for communication between forwarders and indexers. Recently we've noticed that events from the forwarders were not indexed.

In the indexer's splunkd.log, we've noticed the following error message;

01-31-2013 10:51:13.557 -0600 ERROR TcpInputProc - Error encountered for connection from src=10.1.1.11:33343. error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired

It seems like I need a new certificate for communication between forwarders and indexers. How can I create a new certificate?

[splunkreal]

Hello, is it possible that Splunkforwarder still works if the cacert.pem on the indexer is expired and from different certificate authority? We have sslVerifyServerCert = false set on the fwd.

Thanks.

[Masa]

Basically, Splunk online doc (http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcert...) provide great information about how to generate a new certificate.

Splunk default SSL certificate is valid for three years. So, when Splunk InputTcpProc validates a SSL certificate, it recognized it as invalid certificate. As default, Splunk initial installation generate "three" years valid certificates. If a user has been using Splunk since 2010. It is time to run into an issue of certificate expiration and forwarders cannot send events to the indexers which is using the default certificate for splunktcp-ssl connections.

Under the following conditions, just to set up a new certificate, which is generated by Splunk default CA, into indexers is enough to resolve the issue. No need to deal with forwarder settings.

1. Forwarders -(splunktcp-ssl)-> Indexers
2. Both forwarders and indexers are using the same rootCA which is Splunk default CA
3. Forwarders does not verify Server certificate CommonName/Alt-Name
4. Indexers do not require client side certifiate

[splunkIT]

Updated docs link:
http://docs.splunk.com/Documentation/Splunk/6.0/Security/ConfigureSplunkforwardingtousesignedcertifi...

http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwith...

http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwith...

[princemagaisa]

I have now created ssl certificates, My Splunk SSL certificates expired after the normal 3 year period. I have generated new SSL certificates which worked well with the forwarders running in the Linux OS. These forward data directly to the Splunk index.

However, since the certificates expired, the Splunk index is still not receiving the data from the DB connect servers.

What could be the root of this problem? How can I get my DB Connect App to start putting data in Splunk index? 

this is what i found on my logs
09-06-2016 18:21:57.221 +0200 INFO TcpOutputProc - Connection to x.x.x.x:9997 closed. Connection closed by server.
09-06-2016 18:21:57.323 +0200 WARN TcpOutputFd - Connect to x.x.x.x.x:9997 failed. Connection refused
09-06-2016 18:21:57.323 +0200 ERROR TcpOutputFd - Connection to host=x.x.x.x.x:9997 failed
09-06-2016 18:21:57.323 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.x=9997 _numberOfFailures=2
09-06-2016 18:22:25.066 +0200 INFO TcpOutputProc - Removing quarantine from idx=x.x.x.x:9997
09-06-2016 18:22:25.067 +0200 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997
09-06-2016 21:07:45.408 +0200 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunk/var/log/splunk/dbx.log'.

x.x.x.x refers to indexer IP

Could this also spring from SSL Certificate issues since i did not apply the new certificates the DB Connect server?

DESPARATE, please help!