Forwarder resiliency

[mundus]

My understanding was that when a forwarder loses its connection to the central Splunk server, it will continue accepting data from other forwarders and it will queue those events until the Splunk server comes back online. One of our engineers is under the impression that if the central Splunk server is unavailable, forwarders stop accepting new data or just drop it on the floor.

Could somebody clarify this for me?

Thx.

Craig

[_d_]

You're partially right; if an intermediate forwarder loses its connection to an indexer it will accept data from other forwarders until its own queues get filled up and then it will start dropping them.

This is true even when an indexer's queues are filled up (ie. the problem propagates down the tiers all the way to the first level forwarder). This is precisely why our preferred, most reliable and recommended method of data input is to first persist a log file to disk then monitor it using a Universal Forwarder. It is capable - among other things - of keeping track of what parts of a file have been sent to the indexer therefore guaranteeing you to have all your data safely send to the indexer in case something happens upstream (the connection is dropped, parts of the network go down, a firewall comes up, the indexer goes down for an upgrade, an indexer's queues are filled up etc). Hope it clarifies the problem a bit.