Getting Data In

Forwarder resiliency

mundus
Path Finder

My understanding was that when a forwarder loses its connection to the central Splunk server, it will continue accepting data from other forwarders and it will queue those events until the Splunk server comes back online. One of our engineers is under the impression that if the central Splunk server is unavailable, forwarders stop accepting new data or just drop it on the floor.

Could somebody clarify this for me?

Thx.

Craig

Tags (1)

_d_
Splunk Employee
Splunk Employee

You're partially right; if an intermediate forwarder loses its connection to an indexer it will accept data from other forwarders until its own queues get filled up and then it will start dropping them.

This is true even when an indexer's queues are filled up (ie. the problem propagates down the tiers all the way to the first level forwarder). This is precisely why our preferred, most reliable and recommended method of data input is to first persist a log file to disk then monitor it using a Universal Forwarder. It is capable - among other things - of keeping track of what parts of a file have been sent to the indexer therefore guaranteeing you to have all your data safely send to the indexer in case something happens upstream (the connection is dropped, parts of the network go down, a firewall comes up, the indexer goes down for an upgrade, an indexer's queues are filled up etc). Hope it clarifies the problem a bit.

Get Updates on the Splunk Community!

App Platform's 2025 Year in Review: A Year of Innovation, Growth, and Community

As we step into 2026, it’s the perfect moment to reflect on what an extraordinary year 2025 was for the Splunk ...

Operationalizing Entity Risk Score with Enterprise Security 8.3+

Overview Enterprise Security 8.3 introduces a powerful new feature called “Entity Risk Scoring” (ERS) for ...

Unlock Database Monitoring with Splunk Observability Cloud

  In today’s fast-paced digital landscape, even minor database slowdowns can disrupt user experiences and ...