Splunk App for Stream : if indexing queue blocked or the network connection with Indexer is disconnected what happen?

[kwchang_splunk]

Dear stream experts,

In case of TCP/UDP inputs, there are the persistent queue and memory queue which can deal with the network problem with indexers behind. What happen if the UF which runs streamfwd disconnected from indexer? Is there any buffer which can handle it? If there is, how can I manage the size of it?

Thank you.

[mdickey_splunk]

The Splunk UF has an in-memory event queue and HA capabilities that are controlled by outputs.conf. In particular, see the maxQueueSize, dropEventsOnQueueFull and dropClonedEventsOnQueueFull. Note that dropEventsOnQueueFull defaults to -1, which blocks the Stream TA and will cause its event queue to overflow (causing events to be dropped) if the UF is unable to contact an indexer. You can reduce the risk of dropped events by configuring groups of indexers in your outputs.conf.

[vshcherbakov_sp · Accepted solution]

hi kwchang,

Stream has a configurable in-memory event queue - see MaxEventQueueSize parameter in the docs. For all other data forwarding tasks, Stream relies on UF to manage the data forwarding. If UF Stream runs with is not properly configured (disconnected from indexers, etc.), Stream starts dropping events once its event queue is full - you'll see messages like "event queue overflow; dropping NNN events" in stream log when it occurs.