Indexing Queues Blocked if Network Is Not Reachable, why?

[mzorzi]

My Indexer is receiving data from a Forwarder but also sending data to non Splunk device.
This external device became unreachable and, consequently, all the queues in my Indexer became blocked.

I mean, not just the tcpout_connections related the the device, but also all the aggqueue, indexqueue, parsingqueue. The tcpin_connections instead it was working fine, but a the end I couldn't see any data from my Search.

Is that the normal behavior ? I would expect Splunk to keep indexing even if it can't send data out, or, in other words, the two queue chains to be separated.

[gkanapathy · Accepted solution]

This is normal. However, you can set dropEventsOnQueueFull in outputs.conf to control this. Alternatively, you could forward using a non-blocking protocol, i.e., UDP.

[gkanapathy]

Correct, you'd set dropEventsOnQueueFull on the indexer outputs.conf, not the LWF.

[mzorzi]

Ok, to be super clear: In a situation like LFW->FWD->ExtDev, in the outputs.conf stanza which defines FWD->ExtDev I set dropEventsOnQueueFull to 1. So if the communication between FWD->ExtDev goes down, tcpout_connections in FWD get fulled so it starts dropping event, but communication LWD->FWD stays up and Splunk don't lose anything. That makes sense.

[gkanapathy]

You can just set dropEventsOnQueueFull to 1 or some other positive integer for the output if you don't want it to block. Is this what you're looking for?

[mzorzi]

It's normal, but it's not nice..any plan to improve it in the future?