Monitoring Splunk

Is there any way to equally distribute the storage load on all of the 4 indexers? Does data rebalancing option help here

sbhatnagar88
Path Finder

HI Experts,

 

we have 4 physical indexers in cluster and since few days /splunk file system storage has reached to threshold on 2 out of 4 indexers.

Is there any way to equally distribute the storage load on all of the 4 indexers? Does data rebalancing option help here?

Labels (1)
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

You can rebalance buckets and it will probable help somewhat for som short time but it's worth digging into why such unbalanced storage use occured in the first place.

Outputs in splunk components connect to a randomly chosen single output from a load-balancing group and send their events to this one output until the thresholds are reached (see @gcusello 's reply).

If you - for example - have just one forwarder and you get extremely unlucky, that forwarder may be pushing events to a single indexer for a longer period of time.

This is how splunk's load-balancing works.

Now what can you do about it?

Lowering thresholds is one thing - the more often forwarder chooses where to send data, the more probability that in the long run the distribution among single outputs will be relatively uniform. But that comes at a price of additional overhead of reconnection processing so you have to find a reasonable balance between longer connection duration for performance vs. shorter for load balancing.

The more distinct source forwarders you have, the more probability that as a whole group they will be hiting those indexers uniformly.

And the more indexers you have, the lower probability that the forwarder will hit the same indexer again and again.

So if you had, for example, several indexers ahd many uf's but ingested all events from uf's via a single hf, you'd be cripplimg your load balancing severly.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sbhatnagar88,

when there are more Indexers ,load should be balanced between Indexers (there's an automatic AutoLoadBalancing) but I found that usually this isn't true.

During a training, an instructor said that if there are less Heavy Forwarders than indexers this could be possible because when an HF starts to send logs to an Indexers it continues until it's available so some Indexers could be less used, but this topic wasn't confirmed by other Splunk people.

Anyway, you can change a parameter to distribute indexers between forwarders (HFs and UFs), you could use in outputs.conf:

connectionTTL = <integer>
* The time, in seconds, for a forwarder to keep a socket connection
  open with an existing indexer despite switching to a new indexer.
* This setting reduces the time required for indexer switching.
* Useful during frequent indexer switching potentially caused
  by using the 'autoLBVolume' setting.
* Default: 0 seconds

for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Outputsconf

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...