Deployment Architecture

How to perform maintenance on entire search head clusters ?

vgollapudi
Communicator

Hello Techies !!

I have to take snapshots of the EBS volumes that are attached to the cluster search heads, I have to do it by stopping the instance and perform the task. The reason behind stopping the instances is I wanted to deploy apps through deployer but, I don't know whether the deployer will override the existing apps under /opt/splunk/etc/apps on the cluster search heads so, I don't want to take any risk. I'm aware of file precedence related to apps that are installed on the cluster search heads. In worst case scenarios, I can revert back the changes through snapshots.

Any recommendations on how to perform the maintenance and also on what instances should I stop the splunk service during the downtime.

0 Karma

hardikJsheth
Motivator

You are right the Search Head Deployer will overwrite apps folder with the content from the Search Head Deployer but changes to local folder would remain. Also the data from the default folder won't be lost. Whenever you push folder from SHD, it will take backup of existing default folder by appending timestamp to the folder before coping folder from SHD.

You should not have any problem even if you push apps from SHD. In case you still want to take back up, close your search head one by one and perform the back up and restart the node. Repeat this process for all the nodes also ensure every time check the cluster status to verify SHC (Captain) node is getting selected.

0 Karma

vgollapudi
Communicator

The re-election of the captain has to get approval from the majority of the current cluster members that means if one of the Cluster member is down and only two of them are available to vote but the issue is one of two has to be captain. It won't be possible if the cluster has only 3 members.

A cluster should consist of a minimum of three members. A two-member cluster cannot tolerate any node failure. Failure of either member will prevent the cluster from electing a captain and continuing to function.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/SHCarchitecture#Captain_election_proces...

The other approach would be elect static captain and it becomes complicated because again you have make them dynamic once the maintenance is done. This process has to be repeated 3 times for each member.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/Staticcaptain

0 Karma

vgollapudi
Communicator

Thanks Sheth for your response. If we restart the node that has been stopped is it necessary to run the re-sync command "splunk resync shcluster-replicated-config" on each node after the node is up after the backup ?

I checked this link and the documentation said that it will be done automatically before the set time exceeds. I'm not aware of the default set time for the syncing process to kick on its own.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/HowconfrepoworksinSHC#Replication_synch...

0 Karma

vgollapudi
Communicator

Another question though, the re-election of the captain has to get approval from the majority of the current cluster members that means if one of the Cluster member is down and only two of them are available to vote but the issue is one of two has to be captain. It won't be possible if the cluster has only 3 members.

A cluster should consist of a minimum of three members. A two-member cluster cannot tolerate any node failure. Failure of either member will prevent the cluster from electing a captain and continuing to function.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/SHCarchitecture#Captain_election_proces...

The other approach would be elect static captain and it becomes complicated because again you have make them dynamic once the maintenance is done. This process has to be repeated 3 times for each member.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/Staticcaptain

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...