Splunk Indexer / Forwarder Acknowledgement explained

[hrawat]

If you use Splunk Enterprise or SplunkCloud, you can guard against loss of data when forwarding by enabling the indexer acknowledgment capability. With indexer acknowledgment, the forwarder will resend any data that the receiver does not acknowledge as "received".

You enable indexer acknowledgment on the forwarder, in the outputs.conf file.
See how acknowledgement works.
See how acknowledgement failure is handled.

However if the forwarder is restarted/stopped while waiting for indexer acknowledgment, in most of the cases unacknowledged data is not resend upon forwarder restart/start. That is because indexer acknowledgment is just an agreement  between the output processor and the target server.

 There are 4 major input types:

1. File inputs(monitor/batch mode)
2. Modular inputs
3. Network inputs(TCP/UDP)
4. HTTP inputs(http receiver endpoints/http event collector)

Only the file input in the monitor mode can resend data if the forwarder is restarted/stopped while waiting for indexer acknowledgment.

Acknowledgement is sent back to the forwarder after replication factor is met. That means for rep factor 3, source indexer waits on acknowledgement from 2 replication target indexers.

Inputs on forwarders are not aware of the indexer acknowledgement process.

Latency increases when the target server is not an indexer as the intermediate tier will wait for acknowledgement before acknowledging back to edge forwarder.

For more information, see the outputs.conf spec file.

[hrawat]

You nailed it. You may want to check https://community.splunk.com/t5/Knowledge-Management/Splunk-Persistent-Queue/m-p/688223/highlight/tr...

[gjanders]

Persistent queue support for monitor inputs will be very useful once it's available.

[gjanders]

It may be worth adding that the acknowledgement option cannot protect against data loss a scenario where a forwarder is restarted while the remote endpoint is not available

To expand on this point, let's assume we have universal forwarder A, sending data to heavy forwarder B (and only HF B). (And then assume B connects to indexers)

If A is reading from a file and sending to B, if we shutdown B, and while B is unable to process data we restart A during this downtime of B, any "in memory" data is lost at this point as the memory buffer if flushed on shutdown.

The file monitor will re-read the portion of the file *after* the lost portion of the data.

This experiment is quite easy to setup in a development environment, the only point I'm adding is that (as advertised) the acknowledgement protects against intermediate data loss. It does not protect against data loss when the remote endpoint is down and the source is restarted.