Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer.

[thomson12]

Hi Splunk Support,

I have encounter the issue on Splunk Forwarder have lost connection to Splunk Indexer after clean indexing data on Splunk Indexer.

Scenario: I have cleaned indexing data on Splunk Indexer and check on Splunk Forwarder by command [~/bin/splunk list forward-server -auth username:password]

Result: Splunk Forwarder lost connection as below.
root@env # ~/bin/splunk list forward-server -auth username:password

Active Splunk-2-Splunk Forwards:

    Splunk_Indexer01:9997

Configured but inactive Splunk-2-Splunk Forwards:

    Splunk_Indexer02:9997

Please help. Thank you.

[thomson12]

Thanks for advise, I found this message from Splunk forum "Running splunk list forward-server lists one of the servers under "Configured but inactive forwards:", but it is forwarding. (SPL-35461)" I was POC with my environment. Command "~/bin/splunk list forward-server -auth username:password" will display some Indexer inactive "Configured but inactive Splunk-2-Splunk Forwards" but forwarder still forwarding data to that server when I verify time from Search apps.

[bmacias84 · Accepted solution]

The splunk clean eventdata -index ##someidex### -f command is a stop the world event requiring you to stop and start splunkd.

Are you using the auto-load-balance feature on your fowarders? When the splunkd service starts responding the fowarder should pickit up on the next interval.

[tcpout]

[tcpout-server://server1:9997]
[tcpout-server://server2:9997]
[tcpout-server://server3:9997]

[tcpout:default-autolb-group]
autoLB = true
disabled = false
autoLBFrequency = 45
maxFailuresPerInterval = 3
secsInFailureInterval = 2
connectionTimeout = 10

server = server1:9997,server2:9997, server3:9997

[jbsplunk]

What does the TcpOutputProc component of splunkd in your splunkd.log show is happening with outgoing connections to the indexers? You should see some type of information on connections to both indexers.