Deployment Architecture

What is the correct procedure to patch the OS and reboot host servers in an indexer clustering environment?

kornkid42
Explorer

We have 1 master and 3 peers all running RHEL 7. My question is, what is the correct procedure to patch the OS and reboot the host servers? Here are the steps I took:

Put cluster in maintenance mode
Patch indexer cluster master node and reboot
Put cluster back in maintenance mode
Patch the search peers
Restart the peer servers one at a time
Take cluster out of maintenance mode

Is this the correct procedure? When completed, the number of buckets rose from 13 to 29, is this expected?

woodcock
Esteemed Legend

The first step was not necessary but your process is fine otherwise. The reason for the increase in buckets is that whenever Splunk stops/restarts, it closes all open hot buckets (at least 1 for every index on every indexer). This is normal.

0 Karma

season88481
Contributor

You can patch your cluster master at last, so that you don't have to enable maintenance mode twice.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can follow the same sequence as for Splunk Upgrade of Indexer cluster. Details here http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/Upgradeacluster#Upgrade_to_a_new_maintenan...

0 Karma

kornkid42
Explorer

I followed the directions in the link you posted. Is it normal for the "Buckets" numbers to rise every time the servers are rebooted? Each peer had 12 buckets, and after rebooting twice, they now have 29.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I believe the reboot causes cluster master to rebalance the primary buckets. See this

http://docs.splunk.com/Documentation/Splunk/6.2.6/Indexer/Rebalancethecluster

Can you run the | dbinspect index=YourIndexName from the search to check different buckets and their type/sizes. This just to check if reboot has caused Hot buckets to roll to Warm before they reach the maximum size.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...