What happens to my events at Splunk Light Forwarder when the Indexer goes down?

[the_wolverine]

I have a bunch of Lightweight Forwarders (LWF) forwarding to my central indexer. What happens to my events when there's a problem with the indexer and it can't index when my LWFs are trying to send to it?

[dskillman · Accepted solution]

The LWF will queue up events and try to resend. There is a maxQueue setting in outputs.conf that you can configure a larger queue. There are other settings you can tweak to cover your scenarios like dropping events or blocking if the queue fills. I would recommend spinning up another Splunk Indexer and use AutoLB and distributed search to limit losing connectivity to the indexing tier. You'll get better redundancy and better performance.

http://www.splunk.com/base/Documentation/4.1/Admin/Outputsconf

[Dan · Accepted solution]

If the output queue fills up, all preceding Splunk processors will block. This means if you're monitoring a file or directory, the tailing processor will block and stop moving the pointers into each file. Once the indexer is up and the output queue empties, the tailing processor will unblock and the pointer will eventually catch up. You shouldn't lose data, unless the outage is so long that the file gets rolled or deleted.

If you have network inputs, no such luck.

[Dan]

that being said, I also recommend multiple Splunk Indexers and AutoLB. You get auto-failover in bad situations and awesome performance all other times.

[Dan]

like all else, this is configurable. dropEventsOnQueueFull in outputs.conf

[dskillman · Accepted solution]

The LWF will queue up events and try to resend. There is a maxQueue setting in outputs.conf that you can configure a larger queue. There are other settings you can tweak to cover your scenarios like dropping events or blocking if the queue fills. I would recommend spinning up another Splunk Indexer and use AutoLB and distributed search to limit losing connectivity to the indexing tier. You'll get better redundancy and better performance.

http://www.splunk.com/base/Documentation/4.1/Admin/Outputsconf