Monitoring Splunk

Indexer Queues are full



We've noticed an issue with our upgrade after upgrading Splunk from 7.3.2 to version 8.0.5. 
We're on a cluster environment, with 3 indexers and 3 SHs. We're forcing python 3.7 on all of the Splunk servers.

Since the upgrade, all 3 indexer Indexing queues have been full, as you can see in the screenshot below.

There have been no changes to the amount of data we're ingesting since the upgrade, however a few of the apps did need to also be upgraded to be python 3.7 compatible. 

Here is what we've tried:

  • Restarting - Alleviates the queues for a little bit, but inevitably gets blocked
  • Increasing the queue sizes - We've increased the queue sizes from the default to 80mb, and this increased the time until the queues were blocked. Noticeably, one indexer would block first, then the others would get blocked after some more minutes
  • Validated all the permissions and ownership

There's been two things of note that could be related to this issue:


This graph shows that the indexer pipe is directly correlating to the FwdDataReceiverThread. Unfortunately, doesn't seem to be much info concerning this thread out there. 
We've noticed that we've been getting the following errors concerning this thread.

  • ERROR Watchdog - No response received from IMonitoredThread=0x7fb47f7feb50 within 8000 ms. Looks like thread name='FwdDataReceiverThread' tid=6894 is busy !? Starting to trace with 8000 ms interval.

There have also been a number of crashlogs since the upgrade on the Indexers. These crashlogs include items like the following:


It seems to be related to a particular search, so not sure if this is related to the issue.


Does anyone have any ideas about these items?

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

<P style=" text-align: center; "><span class="lia-inline-image-display-wrapper lia-image-align-center" ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

<FONT size="5"><FONT size="5" color="#FF00FF">Get the latest news and updates from the Splunk Community ...