Deployment Architecture

Cluster bundle stuck in validation mode?

rbal_splunk
Splunk Employee
Splunk Employee

We have 40 node Indexer cluster setup on Splunk version 6.6.
We push a bundle and only 3 of the 32 indexers have validated the package. We are stuck and cannot push out any updates to indexers.

alt text

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...