topic Re: running down indexer congestion problems in Getting Data In

running down indexer congestion problems

tgiles — Tue, 09 Aug 2011 22:15:37 GMT

Hi,

I'm running into occasional errors from one of my indexers reporting "skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block."

I've run the following to monitor for any high values for the queues and don't see anything really actionable during timeframes I see the messages:

index="_internal" source="*metrics.log" group="queue" earliest=-4h | timechart max(current_size) span=30m by name

Checked for any forwarders flooding my indexer and nothing was obvious. So, nothing really actionable.

According to SPL-37407, this is a known issue in 4.2.1 "most often tcpout-queue", but there's no real info on how to get it addressed. in fact, that's the only place the tcpout-queue is mentioned. So, got some questions:

is there a search to query the status of the tcpout-queue on indexers?
would adjusting the maxQueueSize in the outputs.conf on the search heads give me a bigger default queue to work with before errors start?
Any tips on how to troubleshoot indexer congestion issues? there's not a lot of data out there about how to handle.

Thanks!

tom

Re: running down indexer congestion problems

splunk68 — Wed, 14 Aug 2013 14:14:31 GMT

You could try the "Splunk on Splunk" App, http://apps.splunk.com/app/748

It will provide you a good overview of what's happening on your indexer.

Re: running down indexer congestion problems

yannK — Wed, 14 Aug 2013 15:32:11 GMT

Good advice : install the SOS app on the indexer and check the indexing performance.
If the queues are full, then this can be :

slow disks (index queue) or congested rotation of buckets from homePath -> coldPath -> frozen
heavy parsing (parsing/aggregation queues) or non optimized events
heavy load, the usual suspect
too large metadata files -> upgrade to 4.3 ASAP

And remember that at one point, you will need more than 1 indexer to scale your volume.

Re: running down indexer congestion problems

vr2312 — Fri, 12 May 2017 09:48:51 GMT

@yannK , is it also possible for the congestion to occur due to a lot of searches targeting the indexer. We have premium apps (ITSI/ES) enabled in our environment. Could that be the case too ?