Getting Data In

Forwarder resiliency

mundus
Path Finder

My understanding was that when a forwarder loses its connection to the central Splunk server, it will continue accepting data from other forwarders and it will queue those events until the Splunk server comes back online. One of our engineers is under the impression that if the central Splunk server is unavailable, forwarders stop accepting new data or just drop it on the floor.

Could somebody clarify this for me?

Thx.

Craig

Tags (1)

_d_
Splunk Employee
Splunk Employee

You're partially right; if an intermediate forwarder loses its connection to an indexer it will accept data from other forwarders until its own queues get filled up and then it will start dropping them.

This is true even when an indexer's queues are filled up (ie. the problem propagates down the tiers all the way to the first level forwarder). This is precisely why our preferred, most reliable and recommended method of data input is to first persist a log file to disk then monitor it using a Universal Forwarder. It is capable - among other things - of keeping track of what parts of a file have been sent to the indexer therefore guaranteeing you to have all your data safely send to the indexer in case something happens upstream (the connection is dropped, parts of the network go down, a firewall comes up, the indexer goes down for an upgrade, an indexer's queues are filled up etc). Hope it clarifies the problem a bit.

Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...