Deployment Architecture

splunkd and splunkweb are up for a cluster peer, but why is it repetitively downloading the cluster bundle with the following errors?

Communicator

We have clustered environment
Version : v6.1.1

1 x CM (Multi-Site, rf=2, site_rf=origin:1,site1:1,site2:1,total:2, site_sf=origin:1,site1:1,site2:1,total:2)
2 x SHs (Originally Mounted Bundle was enabled. But, it was disabled when this problem happened. A misconfiguration still existed in SH.)
8 x CPs (Two Splunk instances per physical machine, one of the CP is a search head.)
41 x Indexes db

Trying to fix misconfigration of indexes.conf and applying a cluster bundle to peers, one of the Cluster Peers became the state of endlessly downloading
bundle with the errors shown below

07-11-2015 17:13:18.855 -0400 INFO  CMBundleMgr - Removed the untarred bundle folder=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/bce9740371ce42bcc4b9f4920bd99857-1436649198
07-11-2015 17:13:18.855 -0400 INFO  CMBundleMgr - Removed the bundle downloaded from master to '/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/bce9740371ce42bcc4b9f4920bd99857-1436649198.bundle'
07-11-2015 17:13:19.880 -0400 ERROR CMBundleMgr - failed to compute checksum err=Truncated tar archive
07-11-2015 17:13:19.935 -0400 INFO  CMBundleMgr - Downloaded bundle to /backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/3d76761010429034685696b0faeed1f5-1436649199.bundle
07-11-2015 17:13:19.935 -0400 INFO  CMBundleMgr - untarring bundle=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/3d76761010429034685696b0faeed1f5-1436649199.bundle
07-11-2015 17:13:19.944 -0400 INFO  ClusterBundleValidator - Validating bundle path=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/3d76761010429034685696b0faeed1f5-1436649199/apps
......
07-11-2015 17:13:20.206 -0400 INFO  CMBundleMgr - Removed the untarred bundle folder=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/3d76761010429034685696b0faeed1f5-1436649199
07-11-2015 17:13:20.206 -0400 INFO  CMBundleMgr - Removed the bundle downloaded from master to '/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/3d76761010429034685696b0faeed1f5-1436649199.bundle'
07-11-2015 17:13:21.234 -0400 ERROR CMBundleMgr - failed to compute checksum err=Truncated tar archive
07-11-2015 17:13:21.294 -0400 INFO  CMBundleMgr - Downloaded bundle to /backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/4685cbfb25fac1c4945bece7f3287b9d-1436649201.bundle
07-11-2015 17:13:21.294 -0400 INFO  CMBundleMgr - untarring bundle=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/4685cbfb25fac1c4945bece7f3287b9d-1436649201.bundle
07-11-2015 17:13:21.302 -0400 INFO  ClusterBundleValidator - Validating bundle path=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/4685cbfb25fac1c4945bece7f3287b9d-1436649201/apps
..........
07-11-2015 17:13:21.560 -0400 INFO  CMBundleMgr - Removed the untarred bundle folder=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/4685cbfb25fac1c4945bece7f3287b9d-1436649201
07-11-2015 17:13:21.560 -0400 INFO  CMBundleMgr - Removed the bundle downloaded from master to '/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/4685cbfb25fac1c4945bece7f3287b9d-1436649201.bundle'
07-11-2015 17:13:22.587 -0400 ERROR CMBundleMgr - failed to compute checksum err=Truncated tar archive
07-11-2015 17:13:22.644 -0400 INFO  CMBundleMgr - Downloaded bundle to /backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/da769c2cc82d7154a445f326ca44e621-1436649202.bundle
07-11-2015 17:13:22.645 -0400 INFO  CMBundleMgr - untarring bundle=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/da769c2cc82d7154a445f326ca44e621-1436649202.bundle
07-11-2015 17:13:22.651 -0400 INFO  ClusterBundleValidator - Validating bundle path=/backup/splunk1/splunk/var/run/splunk/cluster/remote-bundle/da769c2cc82d7154a445f326ca44e621-1436649202/apps

We are makinga change in the etc/slave-apps/all_indexes/local/indexes.conf to fix the follow some error messages. For the change to be in effect:

Restart CM.
Rolling restart of the peer.
But 1 out of the 8 peers' status is down, even though Splunk status is up.

Splunk Employee
Splunk Employee

Have you tried removing the peer from the cluster, restarting it, and then re-adding it?

Additionally, if you are editing the slave-apps, this can cause some issues. Be careful with the user you edit this as and what you edit. When you apply a bundle, all the peers will pull the master-apps/* from the Cluster Master. If file level permissions are wrong, then there will some issues...

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!