topic Re: Does the indexer acknowledgement queue/list persist across Splunk restarts? in Getting Data In

Does the indexer acknowledgement queue/list persist across Splunk restarts?

Glenn — Fri, 20 Apr 2012 07:41:50 GMT

I'd like to user indexer acknowledgement in my HA setup when forwarding from a primary indexer which receives events from forwarders, to a secondary indexer (despite the horrible proliferation of duplicate events it can cause, but that's another issue).

I'd like to know whether the queue or list of unacknowledged events maintained on the primary indexer will persist if the primary indexer is restarted (while the secondary is still unavailable).

If it doesn't, we could easily lose the queue and have gaps in our secondary index, breaking HA.

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

yannK — Fri, 20 Apr 2012 14:46:17 GMT

If I am correct, the ACK=true option (on the forwarder) will cause the forwarder to wait for an acknowledgement from the indexer that the event has been written to disk.
So If the indexer goes down, the forwarder will retry.
As you can see, at the end it will not cause gaps, only accidental duplicates.

Edit :

In the case of a forwarder

About tailing :

File Tailing queues, splunk keeps track of the position it was reading a file, and if it is restarted, it will restart from that point.
Scripted inputs/network inputs (example syslog on port 514), a splunk instance will store thoses queues in memory, and cannot recover them. An easy workaround is to use a syslog-ng/rsyslog server to write the log to file, and have splunk monitor the files (equivalent to disk buffer).

About queues :

Memory queues, used by default. A forwarder crashing/shutdown while queueing data will lose the data waiting in the memory queue.
Persistent queues, you can configure your forwarders to keep part of the data queued on disk instead of memory, they will be resumed once restarted. see http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

About HA acknowledgement :

you can also add another level of security, by using acknowledgments, this is more costly in traffic and speed. http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Protectagainstlossofin-flightdata

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

Glenn — Fri, 20 Apr 2012 15:51:59 GMT

I should have been more clear. I mean, what happens to the queue on the forwarder, if it goes down while the indexer is already down. ie. does the forwarder's queue still have the same data after it is restarted?

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

yannK — Fri, 20 Apr 2012 17:33:37 GMT

edited above.

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

sdaniels — Fri, 20 Apr 2012 18:03:56 GMT

You could use persistent queues on the forwarder so that the data is there after restart, otherwise you'll lose that data in memory.

http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

Glenn — Fri, 20 Apr 2012 23:35:54 GMT

Looks good thanks. I think persistant queues is what I was looking for.

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

Glenn — Fri, 20 Apr 2012 23:36:45 GMT

Thanks, this is helpful.

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

cphair — Mon, 09 Jul 2012 17:37:50 GMT

@yannK, do you have specific measures of the cost of enabling HA acknowledgement beyond what's in the document you linked? I understand the memory usage on the forwarder side would increase, but I'd like to know the effect on the indexer side as well.

Re: Does the indexer acknowledgement queue/list persist across Splunk restarts?

youngsuh — Wed, 22 Sep 2021 20:36:47 GMT

Here is the updated link for community convenience:

Use persistent queues to help prevent data loss - Splunk Documentation

https://docs.splunk.com/Documentation/Splunk/8.2.2/Forwarding/Protectagainstlossofin-flightdata