Getting Data In

Indexer recv-q high and certain sources lagging. How do I fix that?

vcarbona
Path Finder

Noticing from netstat there are high recv-q numbers on the indexer. We also notice some sources lagging in the indexer. The ports with the high recv-q are from the forwarders which contain these particular sources.

Our setting in limits.conf on the indexer is the following:
[thruput]

throughput limiting at index time

maxKBps = 0

We don't have excessive cpu, memory, or io on the indexer. Indexers will typically have about 35 connections from forwarders open. We also have a ulimit set to 4096.

Our current workaround is to restart the indexer but it reappears on other indexers of the same server group. Obviously this is starting to become an annoyance.

Anything else we should be checking?

Tags (3)
1 Solution

vcarbona
Path Finder

We've done a few things since then to control this issue:
1) Set our ulimit to 10240 on all indexers (But this didn't in itself resolve the problem yet helped out with the load)
2) We've also added the following line to the offending forwarder's outputs.conf under the [tcpout] stanza and restarted the forwarder:
forceTimebasedAutoLB = true

This latter solution has helped out tremendously so far in forcing data to be load balanced. It appears the default behavior is to stream it to one indexer until the "batch" of data is done. The setting appears to force it to another indexer.
Thanks for asking by the way! Almost forgot I posted the question awhile back 🙂

View solution in original post

vcarbona
Path Finder

We've done a few things since then to control this issue:
1) Set our ulimit to 10240 on all indexers (But this didn't in itself resolve the problem yet helped out with the load)
2) We've also added the following line to the offending forwarder's outputs.conf under the [tcpout] stanza and restarted the forwarder:
forceTimebasedAutoLB = true

This latter solution has helped out tremendously so far in forcing data to be load balanced. It appears the default behavior is to stream it to one indexer until the "batch" of data is done. The setting appears to force it to another indexer.
Thanks for asking by the way! Almost forgot I posted the question awhile back 🙂

omgwut56k
Path Finder

Did you find a solution?

0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...