Deployment Architecture

Cluster bundle stuck in validation mode?

rbal_splunk
Splunk Employee
Splunk Employee

We have 40 node Indexer cluster setup on Splunk version 6.6.
We push a bundle and only 3 of the 32 indexers have validated the package. We are stuck and cannot push out any updates to indexers.

alt text

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...