Indexer recv-q high and certain sources lagging. How do I fix that?

[vcarbona]

Noticing from netstat there are high recv-q numbers on the indexer. We also notice some sources lagging in the indexer. The ports with the high recv-q are from the forwarders which contain these particular sources.

Our setting in limits.conf on the indexer is the following:
[thruput]

throughput limiting at index time

maxKBps = 0

We don't have excessive cpu, memory, or io on the indexer. Indexers will typically have about 35 connections from forwarders open. We also have a ulimit set to 4096.

Our current workaround is to restart the indexer but it reappears on other indexers of the same server group. Obviously this is starting to become an annoyance.

Anything else we should be checking?

[vcarbona · Accepted solution]

We've done a few things since then to control this issue:
1) Set our ulimit to 10240 on all indexers (But this didn't in itself resolve the problem yet helped out with the load)
2) We've also added the following line to the offending forwarder's outputs.conf under the [tcpout] stanza and restarted the forwarder:
forceTimebasedAutoLB = true

This latter solution has helped out tremendously so far in forcing data to be load balanced. It appears the default behavior is to stream it to one indexer until the "batch" of data is done. The setting appears to force it to another indexer.
Thanks for asking by the way! Almost forgot I posted the question awhile back 🙂

[omgwut56k]

Did you find a solution?