topic Re: Indexing Queues Blocked if Network Is Not Reachable, why? in Getting Data In

Indexing Queues Blocked if Network Is Not Reachable, why?

mzorzi — Wed, 07 Jul 2010 20:43:38 GMT

My Indexer is receiving data from a Forwarder but also sending data to non Splunk device.
This external device became unreachable and, consequently, all the queues in my Indexer became blocked.

I mean, not just the tcpout_connections related the the device, but also all the aggqueue, indexqueue, parsingqueue. The tcpin_connections instead it was working fine, but a the end I couldn't see any data from my Search.

Is that the normal behavior ? I would expect Splunk to keep indexing even if it can't send data out, or, in other words, the two queue chains to be separated.

Re: Indexing Queues Blocked if Network Is Not Reachable, why?

gkanapathy — Wed, 07 Jul 2010 20:50:18 GMT

This is normal. However, you can set dropEventsOnQueueFull in outputs.conf to control this. Alternatively, you could forward using a non-blocking protocol, i.e., UDP.

Re: Indexing Queues Blocked if Network Is Not Reachable, why?

mzorzi — Wed, 07 Jul 2010 21:09:27 GMT

It's normal, but it's not nice..any plan to improve it in the future?

Re: Indexing Queues Blocked if Network Is Not Reachable, why?

gkanapathy — Thu, 08 Jul 2010 02:05:48 GMT

You can just set dropEventsOnQueueFull to 1 or some other positive integer for the output if you don't want it to block. Is this what you're looking for?

Re: Indexing Queues Blocked if Network Is Not Reachable, why?

mzorzi — Thu, 08 Jul 2010 17:08:01 GMT

Ok, to be super clear: In a situation like LFW->FWD->ExtDev, in the outputs.conf stanza which defines FWD->ExtDev I set dropEventsOnQueueFull to 1. So if the communication between FWD->ExtDev goes down, tcpout_connections in FWD get fulled so it starts dropping event, but communication LWD->FWD stays up and Splunk don't lose anything. That makes sense.

Re: Indexing Queues Blocked if Network Is Not Reachable, why?

gkanapathy — Thu, 08 Jul 2010 22:53:45 GMT

Correct, you'd set dropEventsOnQueueFull on the indexer outputs.conf, not the LWF.