topic Indexer goes down, Stop the Splunk-forwarder from sending in Getting Data In

Indexer goes down, Stop the Splunk-forwarder from sending

Dark_Ichigo — Mon, 25 Feb 2013 22:29:23 GMT

I have been doing some research into what to my Splunk Forwarder can happen if my Splunk indexer goes down, I have noticed that it may will continue to queue the data in memory until full, this isn't what I had in mind as it will fill up what I can send through the bandwidth connection between the forwarder and the indexer.

I have noticed that there is a useACK=true in outputs.conf on the forwarder side, ifset to true I can then set both readTimeout & writeTimeot parameters to stop when the is no acknowledgement from the Splunk indexer, I have no idea if indeed setting the acknowledgement functionality also includes me setting up how much i want to indexer to index before sending in another patch for indexing, does this have to be specified manually?, is the useACK=true the best option I have?

As I would rather have the Splunk Forwarder stop sending through anything once the Splunk indexer indeed goes down.

Re: Indexer goes down, Stop the Splunk-forwarder from sending

yannK — Mon, 25 Feb 2013 23:15:49 GMT

This is already the default behavior:
if the indexer refuses data (full or down) then the forwarders fill their memory queue up to 2MB (default), then pause the monitoring.

Using useACK will only be useful if you have an unreliable connection and want the indexers to send an acknowledgment to the forwarders who otherwise will resent the events.

Re: Indexer goes down, Stop the Splunk-forwarder from sending

Dark_Ichigo — Mon, 25 Feb 2013 23:20:49 GMT

Is there a way to modify the max queue in the case to higher or lower than the default 2MB to stop it from forwarding data?

Does it keep logging events in the logs after it stops as well, repeatedly that its failing to send?

Re: Indexer goes down, Stop the Splunk-forwarder from sending

martin_mueller — Tue, 26 Feb 2013 09:12:57 GMT

In outputs.conf:

[tcpout]
maxQueueSize = 1MB

Re: Indexer goes down, Stop the Splunk-forwarder from sending

Dark_Ichigo — Tue, 26 Feb 2013 22:41:29 GMT

Thanks, but I take it we need to set useACK=true first?

Re: Indexer goes down, Stop the Splunk-forwarder from sending

martin_mueller — Wed, 27 Feb 2013 07:34:56 GMT

I don't think so.

http://docs.splunk.com/Documentation/Splunk/latest/admin/outputsconf