Deployment Architecture

Cluster bundle stuck in validation mode?

rbal_splunk
Splunk Employee
Splunk Employee

We have 40 node Indexer cluster setup on Splunk version 6.6.
We push a bundle and only 3 of the 32 indexers have validated the package. We are stuck and cannot push out any updates to indexers.

alt text

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

Get Updates on the Splunk Community!

Set Up More Secure Configurations in Splunk Enterprise With Config Assist

This blog post is part 3 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...