Getting Data In

Load balancing cold buckets

watsm10
Communicator

Hi all,

We currently have 4 indexers and 2 search heads running on VMs. We have two more physical servers on their way with faster disk which we will use as indexers. The plan is to use the two physical servers to index the data and store hot + warm buckets and the 4 indexers we have currently will store the cold data.

Firstly, would anyone recommend this type of setup?

Secondly, how do you configure the warm+hot indexers to load balance the cold data across the other 4 indexers? Looking in the documentation I can see that in the indexes.conf file examples (http://docs.splunk.com/Documentation/Splunk/5.0.5/Admin/Indexesconf) that you can specify a "volume", but this only seems to be one server and no more than that...

0 Karma
1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

Splunk will not balance cold buckets across 4 indexers, while leaving hot+warm on 2 other indexers. If your issue is storage space, you could set up a mount from each physical to a single virtual and place the cold buckets in the mount using the config options, I wouldn't recommend this because of the speed of mounting disk in this way. If I had this hardware setup, I would probably use all 6 for hot+warm+cold, and index across all 6. You will see an increase in speed of searches because you have scaled horizontally. Check out this guide, it covers why it's better to scale horizontally. http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Accommodatemanysimultaneoussearches

View solution in original post

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Splunk will not balance cold buckets across 4 indexers, while leaving hot+warm on 2 other indexers. If your issue is storage space, you could set up a mount from each physical to a single virtual and place the cold buckets in the mount using the config options, I wouldn't recommend this because of the speed of mounting disk in this way. If I had this hardware setup, I would probably use all 6 for hot+warm+cold, and index across all 6. You will see an increase in speed of searches because you have scaled horizontally. Check out this guide, it covers why it's better to scale horizontally. http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Accommodatemanysimultaneoussearches

View solution in original post

0 Karma

watsm10
Communicator

Hi thanks for your comments. It's just as I thought then.. I'll use your recommendations for your setup 🙂

0 Karma

gfuente
Motivator

Hello

I think, that what you want to achieve can not be done. All the peers in the cluster will index data from the forwarders and that data will go directly into hot buckets

Regards

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!