All Apps and Add-ons

Splunk App for Stream : if indexing queue blocked or the network connection with Indexer is disconnected what happen?

kwchang_splunk
Splunk Employee
Splunk Employee

Dear stream experts,

In case of TCP/UDP inputs, there are the persistent queue and memory queue which can deal with the network problem with indexers behind. What happen if the UF which runs streamfwd disconnected from indexer? Is there any buffer which can handle it? If there is, how can I manage the size of it?

Thank you.

Tags (1)
0 Karma
1 Solution

vshcherbakov_sp
Splunk Employee
Splunk Employee

hi kwchang,

Stream has a configurable in-memory event queue - see MaxEventQueueSize parameter in the docs. For all other data forwarding tasks, Stream relies on UF to manage the data forwarding. If UF Stream runs with is not properly configured (disconnected from indexers, etc.), Stream starts dropping events once its event queue is full - you'll see messages like "event queue overflow; dropping NNN events" in stream log when it occurs.

View solution in original post

mdickey_splunk
Splunk Employee
Splunk Employee

The Splunk UF has an in-memory event queue and HA capabilities that are controlled by outputs.conf. In particular, see the maxQueueSize, dropEventsOnQueueFull and dropClonedEventsOnQueueFull. Note that dropEventsOnQueueFull defaults to -1, which blocks the Stream TA and will cause its event queue to overflow (causing events to be dropped) if the UF is unable to contact an indexer. You can reduce the risk of dropped events by configuring groups of indexers in your outputs.conf.

vshcherbakov_sp
Splunk Employee
Splunk Employee

hi kwchang,

Stream has a configurable in-memory event queue - see MaxEventQueueSize parameter in the docs. For all other data forwarding tasks, Stream relies on UF to manage the data forwarding. If UF Stream runs with is not properly configured (disconnected from indexers, etc.), Stream starts dropping events once its event queue is full - you'll see messages like "event queue overflow; dropping NNN events" in stream log when it occurs.

Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...