Getting Data In

Indexing Queues Blocked if Network Is Not Reachable, why?

Splunk Employee
Splunk Employee

My Indexer is receiving data from a Forwarder but also sending data to non Splunk device.
This external device became unreachable and, consequently, all the queues in my Indexer became blocked.

I mean, not just the tcpout_connections related the the device, but also all the aggqueue, indexqueue, parsingqueue. The tcpin_connections instead it was working fine, but a the end I couldn't see any data from my Search.

Is that the normal behavior ? I would expect Splunk to keep indexing even if it can't send data out, or, in other words, the two queue chains to be separated.

Tags (3)
1 Solution

Splunk Employee
Splunk Employee

This is normal. However, you can set dropEventsOnQueueFull in outputs.conf to control this. Alternatively, you could forward using a non-blocking protocol, i.e., UDP.

View solution in original post

Splunk Employee
Splunk Employee

This is normal. However, you can set dropEventsOnQueueFull in outputs.conf to control this. Alternatively, you could forward using a non-blocking protocol, i.e., UDP.

View solution in original post

Splunk Employee
Splunk Employee

Correct, you'd set dropEventsOnQueueFull on the indexer outputs.conf, not the LWF.

0 Karma

Splunk Employee
Splunk Employee

Ok, to be super clear: In a situation like LFW->FWD->ExtDev, in the outputs.conf stanza which defines FWD->ExtDev I set dropEventsOnQueueFull to 1. So if the communication between FWD->ExtDev goes down, tcpout_connections in FWD get fulled so it starts dropping event, but communication LWD->FWD stays up and Splunk don't lose anything. That makes sense.

0 Karma

Splunk Employee
Splunk Employee

You can just set dropEventsOnQueueFull to 1 or some other positive integer for the output if you don't want it to block. Is this what you're looking for?

0 Karma

Splunk Employee
Splunk Employee

It's normal, but it's not nice..any plan to improve it in the future?

0 Karma