I have a small development environment with one search head and two indexers. I've noticed that the two indexers are not balancing properly. They seem to have been before and now they are way off balance. I tried the curl command to rebalance from the search head/master but no luck. Any guidance would be appreciated.
Remember that streamed inputs from the forwarder will not be load balanced:
- tcp inputs
- udp inputs
- WinEventLog and maybe other windows inputs
- some continuous scripted inputs
While regular monitor and batch will be loadbalanced every 30 sec
Always double check your outputs, maybe some forwarder only know one indexer.
Your issue likely comes from the synchronization that naturally happens when forwarders switch indexers every 30 seconds and your data coming in at regular times - chances are it'll mostly be the same indexer that's "on duty" at that point every time.
Consider changing your load balancing frequency to a less even number, e.g. to 31 seconds. That'll create an artificial disconnect between the two regular intervals, causing each data delivery to switch between indexers more often.
That won't distribute one delivery between both indexers, but will make it more likely that 24 hourly deliveries during a day don't end up as a 1:23 split.
In your inputs.conf are you using 'batch' or 'monitor' to forward the data. I've seen issues in the past on light and universal forwarders where 'batch' will grab on to one indexer in a pool and not switch to one of the other indexers until after the full content of a file being pushed has completed. This seemed to go away when I converted the 'batch' forwarding hosts to heavy forwarders.,In your inputs.conf are you using 'batch' to forward the data, or 'monitor'. I've seen issues with batch on light forwarders where the forwarding seems to latch on to a single indexer in a pool until that file that it's pushing is completed. When I changed my forwarding to a heavy forwarder the problem went away.
Are you talking about badly balanced primary buckets in a cluster (hence the rebalance attempt from the CLI), or about uneven incoming data with one indexer getting the bulk and the other getting very little?
I'm assuming the latter, if so then the cluster command to rebalance primaries isn't related to your issue.
What kind of data are you adding that isn't being balanced, and how is it added? Forwarders, how many? Any scripts? Scheduled data delivery?
martin_mueller, thanks for the feedback. Yes, I'm talking about the latter (one indexer getting the bulk of the data and the other very little).
All of our data is added from a handful of forwarders (maybe 5). The delivery is not scheduled, but the jobs we run to produce the data to be ingested into splunk is. Most run once per day, but a couple do run on a more regular schedule (hourly perhaps). Each of the jobs we run produces an output that it stores into a particular directory that we have being monitored.
Does this make it clearer? I apologize for the initial confusion. We are a small shop, learning and deploying Splunk at the same time which presents some interesting challenges.