Are you a member of the Splunk Community?
- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Indexer recv-q high and certain sources lagging. H...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Noticing from netstat there are high recv-q numbers on the indexer. We also notice some sources lagging in the indexer. The ports with the high recv-q are from the forwarders which contain these particular sources.
Our setting in limits.conf on the indexer is the following:
[thruput]
throughput limiting at index time
maxKBps = 0
We don't have excessive cpu, memory, or io on the indexer. Indexers will typically have about 35 connections from forwarders open. We also have a ulimit set to 4096.
Our current workaround is to restart the indexer but it reappears on other indexers of the same server group. Obviously this is starting to become an annoyance.
Anything else we should be checking?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We've done a few things since then to control this issue:
1) Set our ulimit to 10240 on all indexers (But this didn't in itself resolve the problem yet helped out with the load)
2) We've also added the following line to the offending forwarder's outputs.conf under the [tcpout] stanza and restarted the forwarder:
forceTimebasedAutoLB = true
This latter solution has helped out tremendously so far in forcing data to be load balanced. It appears the default behavior is to stream it to one indexer until the "batch" of data is done. The setting appears to force it to another indexer.
Thanks for asking by the way! Almost forgot I posted the question awhile back 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We've done a few things since then to control this issue:
1) Set our ulimit to 10240 on all indexers (But this didn't in itself resolve the problem yet helped out with the load)
2) We've also added the following line to the offending forwarder's outputs.conf under the [tcpout] stanza and restarted the forwarder:
forceTimebasedAutoLB = true
This latter solution has helped out tremendously so far in forcing data to be load balanced. It appears the default behavior is to stream it to one indexer until the "batch" of data is done. The setting appears to force it to another indexer.
Thanks for asking by the way! Almost forgot I posted the question awhile back 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Did you find a solution?
Thanks for the Memories! Splunk University, .conf25, and our Community
Data Persistence in the OpenTelemetry Collector
Introducing Splunk 10.0: Smarter, Faster, and More Powerful Than Ever
