Why am I unable to add a currently running standalone indexer to an indexer cluster using the cluster master?

[awilliams_splun]

Having a problem joining an indexer already in use to my cluster. This indexer is currently running as a standalone instance. I'm trying to standup a clustered environment and want to add this indexer as a member to the cluster. I built the cluster master (CM) and it's working fine. When I try to add my indexer to the cluster it fails. The indexer will get stuck in a hung state. If I view the server.conf file of the peer, I can see the clustering information was added correctly to the indexer's /local/server.conf file. However, the peer will not be listed in the clustering view on the CM. The peer is just hung. I can't log into the peer through splunkweb. When I try, it will give an error that splunkweb can not communicate with splunkd. The only way I can get the peer back is by removing the clustering information from /local/server.conf. If I build another instance of Splunk on the same hardware as the one failing, that system will join the cluster and come up just fine. Here are the entries in the splunkd.log file of the CM.

02-26-2015 15:31:37.193 -0500 INFO  CMBundleMgr - Downloaded bundle to /opt/splunk/var/run/splunk/cluster/remote-bundle/9e5648b83b7b2b5b1a0195316e3d8205-1424982697.bundle
02-26-2015 15:31:37.193 -0500 INFO  CMBundleMgr - untarring bundle=/opt/splunk/var/run/splunk/cluster/remote-bundle/9e5648b83b7b2b5b1a0195316e3d8205-1424982697.bundle
02-26-2015 15:31:37.196 -0500 INFO  ClusterBundleValidator - Validating bundle path=/opt/splunk/var/run/splunk/cluster/remote-bundle/9e5648b83b7b2b5b1a0195316e3d8205-1424982697/apps
02-26-2015 15:31:37.221 -0500 INFO  CMBundleMgr - Removed the untarred bundle folder=/opt/splunk/var/run/splunk/cluster/remote-bundle/9e5648b83b7b2b5b1a0195316e3d8205-1424982697
02-26-2015 15:31:37.221 -0500 INFO  CMBundleMgr - Removed the bundle downloaded from master to '/opt/splunk/var/run/splunk/cluster/remote-bundle/9e5648b83b7b2b5b1a0195316e3d8205-1424982697.bundle'

[awilliams_splun · Accepted solution]

This happens when the cluster master can not validate the app bundle that it is trying to push to the peer. This results in the initial app bundle from failing to get added to the cluster peer so the CM never adds the peer to the cluster. This can be resolved by connecting to the CM via CLI and typing the following command:

./splunk apply cluster-bundle --skip-validation

This will force the CM to push the app bundle and the peer come back to life.