What is the impact if a Splunk forwarder loses communication with the indexer?

[iurie_tuluc]

Hello,

Can you please elaborate about the Forwarder/Indexer communication? As I understand the communication is realized via TCP using some Splunk proprietary mechanism.

I’d like to understand what will happen if the Forwarder will lose the connectivity with the Indexer. How will the forwarder’s host will behave on some heavy load with a big log records generated, including the situation when the Indexer is unexpectedly unreachable, like i.e. all the packets from the Forwarder will be dropped on the Indexer. Will the communication reestablish automatically when the Indexer will become reachable again?

Am curious if such tests where done and what are the results.

Thanks,

Iurie

[somesoni2 · Accepted solution]

This is a very common (unwanted) scenario and you can find many discussion around that in Splunk community. Like this one

https://answers.splunk.com/answers/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-the...

In summary, Forwarders have queue with limited capacity and will start to drop events if that queue is full. If you've more than one indexers then setting up LB and Indexer acknowledgement can help you avoid in-flight data loss.
http://docs.splunk.com/Documentation/Splunk/6.4.2/Forwarding/Protectagainstlossofin-flightdata

[iurie_tuluc]

yes, thanks. But beside the risk of losing the logs, I am analyzing a potential impact over the forwarder's host. I.e. in a brutal scenario when the indexer's server will start dropping everything coming from the client using IPTABLES.