DB Connect App stopped putting data in Splunk index after I updated my SSL certificates

[princemagaisa]

My Splunk SSL certificates expired after the normal 3 year period. I have generated new SSL certificates which worked well with the forwarders running in the Linux OS. These forward data directly to the Splunk index.

However, since the certificates expired, the Splunk index is still not receiving the data from the DB connect servers.

What could be the root of this problem? How can I get my DB Connect App to start putting data in Splunk index? 

[vasanthmss]

check the below items,

1. splunk forwarder to indexer connection: if yes .... then look the internal logs of the forwarder you can find the issue else fix the connection problem.
2. if there is no issue with the forwarder indexer communication - check for the dbconnect app's validate the connection details.

post some more detail to answer further

[princemagaisa]

this is what i found on my logs
09-06-2016 18:21:57.221 +0200 INFO TcpOutputProc - Connection to x.x.x.x:9997 closed. Connection closed by server.
09-06-2016 18:21:57.323 +0200 WARN TcpOutputFd - Connect to x.x.x.x.x:9997 failed. Connection refused
09-06-2016 18:21:57.323 +0200 ERROR TcpOutputFd - Connection to host=x.x.x.x.x:9997 failed
09-06-2016 18:21:57.323 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.x=9997 _numberOfFailures=2
09-06-2016 18:22:25.066 +0200 INFO TcpOutputProc - Removing quarantine from idx=x.x.x.x:9997
09-06-2016 18:22:25.067 +0200 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997
09-06-2016 21:07:45.408 +0200 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunk/var/log/splunk/dbx.log'.

x.x.x.x refers to indexer IP

Could this also spring from SSL Certificate issues since i did not apply the new certificates the DB Connect server?

DESPARATE, please help!

[vasanthmss]

These logs are from forwarder ?? Seems like indexer and forwarder communication failed in 9997 port. forwarder unable to connect to indexer with 9997 port using SSL. Are you using 3rd party ssl / self sign ssl? anyhow could you please share the configs?

Check the communication by:
telnet

telnet x.x.x.x 997

These are the few steps you can proceed to debug.

1. remove your ssl and validate the connection.
2. if the step 1 works you have issue with your SSL configurations.

My wild guess is your configurations on SSL is applied in forwarder but not indexer. since you are forcing forwarder to use SSL to the indexer communication. Have you done anything in indexer??

http://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

above is some old wiki page.. still you can refer the configurations.

[princemagaisa]

i am in desparate need of an answer, please help!

[princemagaisa]

in desparate need of an answer

[princemagaisa]

this is what i found on my logs
09-06-2016 18:21:57.221 +0200 INFO TcpOutputProc - Connection to x.x.x.x:9997 closed. Connection closed by server.
09-06-2016 18:21:57.323 +0200 WARN TcpOutputFd - Connect to x.x.x.x.x:9997 failed. Connection refused
09-06-2016 18:21:57.323 +0200 ERROR TcpOutputFd - Connection to host=x.x.x.x.x:9997 failed
09-06-2016 18:21:57.323 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.x=9997 _numberOfFailures=2
09-06-2016 18:22:25.066 +0200 INFO TcpOutputProc - Removing quarantine from idx=x.x.x.x:9997
09-06-2016 18:22:25.067 +0200 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997
09-06-2016 21:07:45.408 +0200 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunk/var/log/splunk/dbx.log'.

x.x.x.x refers to indexer IP

Could it be SSL Certificate issues sinnce i did not apply the new certificates the DB Connect server