topic Re: Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer. in Getting Data In

Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer.

thomson12 — Thu, 20 Sep 2012 10:20:26 GMT

Hi Splunk Support,

I have encounter the issue on Splunk Forwarder have lost connection to Splunk Indexer after clean indexing data on Splunk Indexer.

Scenario: I have cleaned indexing data on Splunk Indexer and check on Splunk Forwarder by command [~/bin/splunk list forward-server -auth username:password]

Result: Splunk Forwarder lost connection as below.
root@env # ~/bin/splunk list forward-server -auth username:password

Active Splunk-2-Splunk Forwards:

    Splunk_Indexer01:9997

Configured but inactive Splunk-2-Splunk Forwards:

    Splunk_Indexer02:9997

Please help. Thank you.

Re: Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer.

jbsplunk — Thu, 20 Sep 2012 21:47:51 GMT

What does the TcpOutputProc component of splunkd in your splunkd.log show is happening with outgoing connections to the indexers? You should see some type of information on connections to both indexers.

Re: Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer.

bmacias84 — Thu, 20 Sep 2012 22:23:24 GMT

The splunk clean eventdata -index ##someidex### -f command is a stop the world event requiring you to stop and start splunkd.

Are you using the auto-load-balance feature on your fowarders? When the splunkd service starts responding the fowarder should pickit up on the next interval.

[tcpout]

[tcpout-server://server1:9997]
[tcpout-server://server2:9997]
[tcpout-server://server3:9997]

[tcpout:default-autolb-group]
autoLB = true
disabled = false
autoLBFrequency = 45
maxFailuresPerInterval = 3
secsInFailureInterval = 2
connectionTimeout = 10

server = server1:9997,server2:9997, server3:9997

Re: Clear indexing data on Splunk Indexer impact to Splunk forwarder lost connection to Splunk Indexer.

thomson12 — Fri, 21 Sep 2012 02:25:24 GMT

Thanks for advise, I found this message from Splunk forum "Running splunk list forward-server lists one of the servers under "Configured but inactive forwards:", but it is forwarding. (SPL-35461)" I was POC with my environment. Command "~/bin/splunk list forward-server -auth username:password" will display some Indexer inactive "Configured but inactive Splunk-2-Splunk Forwards" but forwarder still forwarding data to that server when I verify time from Search apps.