Installation

Difference between Splunk Maintenance Mode vs Splunk Offline mode?

jagadeeshm
Contributor

We are trying to upgrade couple of indexers from our multi site cluster to a better hardware (16 core to 24 core etc). We decided to simply swap the disk to the new boxes to avoid unnecessary fix-up activities and save network traffic.

What is the best way to perform this upgrade?

I am thinking -

  1. Initiate maintenance mode on the cluster by running “splunk enable maintenance-mode” command on the master node.
  2. We have 4 indexers to upgrade, so we stop the splunkd process running on each indexer, one at a time by running “splunk stop” command
  3. Move the disk to the new box
  4. Start splunkd process on this server by running “splunk start” command
  5. Repeat steps 2 to 4 for remaining indexers
  6. Finally, disable maintenance mode by running “splunk disable maintenance-mode” command on the master node.

Or am I supposed to use Splunk Offline mode by extending the default interval?

Any advice?

Labels (3)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

I would follow the cluster upgrade procedure (minus the upgrade tasks for cluster master and search head) to do this. The only addition from your list would to do run "splunk offline" on indexers/peer nodes before stopping them.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Upgradeacluster#Upgrade_to_a_maintenance_r...

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I would follow the cluster upgrade procedure (minus the upgrade tasks for cluster master and search head) to do this. The only addition from your list would to do run "splunk offline" on indexers/peer nodes before stopping them.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Upgradeacluster#Upgrade_to_a_maintenance_r...

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

splunk offline actually stops the indexer.

How long do you think the process will take for each indexer?

Before you put the cluster in maintenance mode, you might consider increasing the restart timeout value to some number of seconds longer that the process will take:

splunk edit cluster-config -restart_timeout 900

Also be sure to take the cluster out of maintenance mode once you are done with the process.

0 Karma

vermasa
New Member

""After the peer shuts down, you have 60 seconds (by default) to complete any maintenance work and bring the peer back online. If the peer does not return to the cluster within this time, the master initiates bucket-fixing activities to return the cluster to a complete state. If you need more time, you can extend the time that the master waits for the peer to come back online by configuring the restart_timeout attribute""

But why does "restart_timeout" matter here ? when you are already putting cluster into maintenance mode which does not allow any bucket fixup activity.

0 Karma

jagadeeshm
Contributor

As per https://answers.splunk.com/answers/464439/what-is-the-best-action-plan-during-hardwarefirmwa.html,
We don't even need to enable the maintenance mode? I am trying to avoid failed searches during this upgrade process.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Yes, the maintenance mode enable is not a requirement to upgrade the peers, but not enabling maintenance mode has certain effect on the cluster health (too many bucket rolling may occur). For short duration to which the peers will be down, I would enable the maintenance mode. See this for more information on effect of not enabling maintenance mode.

https://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Usemaintenancemode

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...