Deployment Architecture

Cluster bundle stuck in validation mode?

rbal_splunk
Splunk Employee
Splunk Employee

We have 40 node Indexer cluster setup on Splunk version 6.6.
We push a bundle and only 3 of the 32 indexers have validated the package. We are stuck and cannot push out any updates to indexers.

alt text

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

Get Updates on the Splunk Community!

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...