Knowledge Management

Splunk Indexer / Forwarder Acknowledgement explained

hrawat
Splunk Employee
Splunk Employee

If you use Splunk Enterprise or SplunkCloud, you can guard against loss of data when forwarding by enabling the indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data that the receiver does not acknowledge as "received".

You enable indexer acknowledgment on the forwarder, in the outputs.conf file.
See how acknowledgement works.
See how acknowledgement failure is handled.


However if the forwarder is restarted/stopped while waiting for indexer acknowledgment, in most of the cases unacknowledged data is not resend upon forwarder restart/start. That is because indexer acknowledgment is just an agreement  between the output processor and the target server.

 There are 4 major input types:

  1. File inputs(monitor/batch mode)
  2. Modular inputs
  3. Network inputs(TCP/UDP)
  4. HTTP inputs(http receiver endpoints/http event collector)

Only the file input in the monitor mode can resend data if the forwarder is restarted/stopped while waiting for indexer acknowledgment.

hrawat_splunk_0-1723069901155.png

 

Acknowledgement is sent back to the forwarder after replication factor is met. That means for rep factor 3, source indexer waits on acknowledgement from 2 replication target indexers.

Inputs on forwarders are not aware of the indexer acknowledgement process.

Latency increases when the target server is not an indexer as the intermediate tier will wait for acknowledgement before acknowledging back to edge forwarder.

For more information, see the outputs.conf spec file.

Tags (2)

hrawat
Splunk Employee
Splunk Employee

gjanders
SplunkTrust
SplunkTrust

Persistent queue support for monitor inputs will be very useful once it's available.

0 Karma

gjanders
SplunkTrust
SplunkTrust

It may be worth adding that the acknowledgement option cannot protect against data loss a scenario where a forwarder is restarted while the remote endpoint is not available

To expand on this point, let's assume we have universal forwarder A, sending data to heavy forwarder B (and only HF B). (And then assume B connects to indexers)

If A is reading from a file and sending to B, if we shutdown B, and while B is unable to process data we restart A during this downtime of B, any "in memory" data is lost at this point as the memory buffer if flushed on shutdown.

The file monitor will re-read the portion of the file *after* the lost portion of the data.

 

This experiment is quite easy to setup in a development environment, the only point I'm adding is that (as advertised) the acknowledgement protects against intermediate data loss. It does not protect against data loss when the remote endpoint is down and the source is restarted.

Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...