Deployment Architecture

How to revert "splunk apply shcluster-bundle" changes from deployer and what is master node for SH Cluster?

kamal_jagga
Contributor

Urgent Issue

We have a clustered environment and it seems that some changes were recently deployed but that deleted some of the changes previously made from UI in the ES app(ES clustered search heads). So, we need to revert the bundle of changes that were deployed recently.

Command used from Deployer to push changes:
splunk apply shcluster-bundle

I see some details given about the rollback from CLI in the documentation.
To rollback the configuration bundle, run this command from the master node:

splunk rollback cluster-bundle

You can use the splunk show cluster-bundle-status command to determine the current active bundle. You can use the cluster/master/info endpoint to get information about the current active and previous active bundles.

http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/Updatepeerconfigurations

But this can be done from master node. But what would be master node for Search-head cluster. And can this be done from Indexer Cluster Master.

Kindly advise.

0 Karma
1 Solution

somesoni2
Revered Legend

The link and command you've listed in your question is for Indexer clusters, and doesn't apply for Search Head clusters (SHC). There is no way to revert a deployed bundle from CLI in SHC. The only way to rollback changes, if you've backup being taken for your SH servers, is to
1) Rollback changes in deployer (etc/shcluster/apps directory),
2) Push updated bundle to SH
3) Restore backup of SH and restart Splunk on them.

On indexer clusters, generally, you don't make any configuration changes from UI, so you can just rollback whatever was previously deployed. In SHC, any change that users make will be lost if you're not taking regular backups.

FYI, this is the link for SHC deployment http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/PropagateSHCconfigurationchanges

View solution in original post

kamal_jagga
Contributor

Since we didn't had the backups, so we had to setup the environment again from scratch. Points have been awarded to Somesh.

Thanks.

0 Karma

davebo1896
Communicator

The question was correctly answered by somesoni2, albeit not questioner's the desired outcome. The karma points offered should have been awarded.

0 Karma

somesoni2
Revered Legend

The link and command you've listed in your question is for Indexer clusters, and doesn't apply for Search Head clusters (SHC). There is no way to revert a deployed bundle from CLI in SHC. The only way to rollback changes, if you've backup being taken for your SH servers, is to
1) Rollback changes in deployer (etc/shcluster/apps directory),
2) Push updated bundle to SH
3) Restore backup of SH and restart Splunk on them.

On indexer clusters, generally, you don't make any configuration changes from UI, so you can just rollback whatever was previously deployed. In SHC, any change that users make will be lost if you're not taking regular backups.

FYI, this is the link for SHC deployment http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/PropagateSHCconfigurationchanges

View solution in original post

kamal_jagga
Contributor

Unfortunately, we don't have backups. Is there a way, we can revert to the last version.

Otherwise, its going to be big issue.

0 Karma

kamal_jagga
Contributor

Thanks for replying Somesh.
I am thinking of doing the following things.
1. Restore the ES from the backup of ES Search-head.
2. Validate if it has all the changes (should have till the day the backup was taken).
3. Once confirmed, take that backup to ES Deployer and push the backup again from there. So that the deployer image of ES and search-head image of ES are also in sync.

Kindly advise.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!