My Indexer is receiving data from a Forwarder but also sending data to non Splunk device.
This external device became unreachable and, consequently, all the queues in my Indexer became blocked.
I mean, not just the tcpout_connections related the the device, but also all the aggqueue, indexqueue, parsingqueue. The tcpin_connections instead it was working fine, but a the end I couldn't see any data from my Search.
Is that the normal behavior ? I would expect Splunk to keep indexing even if it can't send data out, or, in other words, the two queue chains to be separated.
This is normal. However, you can set
dropEventsOnQueueFull in outputs.conf to control this. Alternatively, you could forward using a non-blocking protocol, i.e., UDP.
You can just set dropEventsOnQueueFull to
1 or some other positive integer for the output if you don't want it to block. Is this what you're looking for?
Ok, to be super clear: In a situation like LFW->FWD->ExtDev, in the outputs.conf stanza which defines FWD->ExtDev I set dropEventsOnQueueFull to 1. So if the communication between FWD->ExtDev goes down, tcpout_connections in FWD get fulled so it starts dropping event, but communication LWD->FWD stays up and Splunk don't lose anything. That makes sense.