Why is this indexer unable to get latest bundle from master?

[test_splunk15]

Hi Team,

We are having an issue with Indexer not receiving updated code from master.

I could see when we are pushing code its getting deployed to master and the .bundle is getting created and code is pushed to search heads fine but not to indexers also the .bundle is not persisted in master (utility box). This started happening while we are trying to move Splunk from 7.1 to 7.3.4

Could you please let me know what could be possible wrong?

I have tried checking my puppet code for any errors but there are no errors with :
/opt/splunk/bin/splunk apply cluster-bundle --answer-yes -auth username:password

Post this step we are pushing code to SHs and this is working fine.
/opt/splunk/bin/splunk apply shcluster-bundle --answer-yes -target targerURL -auth username:password

As I mentioned above, during puppet apply (to push latest code running puppet to execute above commands) I see .bundle is getting created but after its applied I dont see the .bundle with latest timestamp is available under master (utility) /opt/splunk/var/run/splunk/cluster/remote-bundle.

I have tried to check the logs under /opt/splunk/var/log/ (splunkd , utility, audit and other logs but nothing concrete I could find), except sometimes getting bundle validation failed (not for each deployment though).

Any suggestions around this please? is this due to upgrade or some other issue?

[harsmarvania57 · Accepted solution]

How many indexer do you have and your bundle size is >200MB ? Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.4/Indexer/Configurationbundleissues

Check $SPLUNK_HOME/var/log/splunk/splunkd.log on Cluster Master with word CMBundleMgr , CMMaster, CMPeer and you will able to see what is happening during bundle creation, validation, reload/restart.

[test_splunk15]

This is working now 🙂

After changing max_peers_to_download_bundle value which was 5 in our settings.

[test_splunk15]

Thanks for you help @harsmarvania57

[harsmarvania57]

Great but you need to find out why bundle size increased from 187MB to 697MB after upgrade.

[test_splunk15]

However I will check the Link provided and come back

[test_splunk15]

in this environment we have 4 indexers.

I verified the size, before upgrade it was 187MB (4 days before) now it is showing 697 MB (how can I verify what is being added additionally - may be the issue with TAs which needs verifying?).

I verified logs and see below:

04-01-2020 16:04:49.972 +0100 INFO CMBundleMgr - setting latest bundle= to active bundle=Bundle-ID
04-01-2020 16:04:49.972 +0100 INFO CMBundleMgr - apply bundle status transitioning from='Bundle validation is in progress.', to='None'
04-01-2020 16:04:49.972 +0100 INFO CMRepJob - running job=CMBundleRemoveJob bundle=[id=, path=/$SPLUNK_HOME/var/run/splunk/cluster/remote-bundle/.bundle]
04-01-2020 16:04:51.938 +0100 INFO CMBundleMgr - Removed the untarred bundle folder=/$SPLUNK_HOME/var/run/splunk/cluster/remote-bundle/

One of the older logs I see:

04-01-2020 16:03:02.222 +0100 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 9 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=13911, tar_elapsed_ms= 2019 , for 9 peer(s), bundle_replication_mode="baseline", bundle_file_name=utility-server-1.bundle, bundle_size=155030KB, replication_id=111111111, replication_reason="async replication allowed"

Should I back-out the upgrade and test?

[harsmarvania57]

Have you tried to run /opt/splunk/bin/splunk apply cluster-bundle --answer-yes -auth username:password on Cluster Master manually ?

[test_splunk15]

yes, I have tried manually still it is not working unless I need to move existing .bundle file to bak file and generate new bundle by restarting Splunk service, post this when I am running my puppet code, it is then replicated (with updated bundle).

[harsmarvania57]

And you need to perform same activity during every bundle push ?

[test_splunk15]

It was not like this earlier, We never used to perform this activity, as said this is happening in only one environment where we upgraded Splunk to 7.3.4

Hence thinking whether upgrade is causing issue (current version expires in next 20 days so need to decide a way forward).

Code is updating in master-apps but not pushed to slave-apps, is it only from .bundle or something else I need to check (I am not a Splunk expert though 🙂

[test_splunk15]

yes - for this case but never it was like this. this should be done automatically (though I am not a splunk expert but this is what I have seen). Same is not happening with other environment where my splunk is running with 7.1

Also I just tried pushing code into indexer and see the same, .bundle getting created and removed automatically. Code is updated under master-apps but not been pushed to slave-apps (is id due to .bundle or something else I am missing)?

when I see the cluster status:

master
cluster_status=None
active_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
latest_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
last_validated_bundle
checksum= value2
last_validation_succeeded=1
timestamp=1585742519 (in localtime=Wed Apr 1 13:01:59 2020)

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_ML_Toolkit/default/experiments.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_imperva-waf/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_microsoft-sqlserver/default/sqlserver_dbx2.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_oracle/default/db_input_templates.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_sourcefire/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_squid/default/eventgen.conf

[test_splunk15]

Also I just modified the code and pushed it for indexer.

I see the .bundle created and then removed automatically cluster status command shows below:

master
cluster_status=None
active_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
latest_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
last_validated_bundle
checksum= value2
last_validation_succeeded=1
timestamp=1585742519 (in localtime=Wed Apr 1 13:01:59 2020)

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_ML_Toolkit/default/experiments.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_imperva-waf/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_microsoft-sqlserver/default/sqlserver_dbx2.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_oracle/default/db_input_templates.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_sourcefire/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_squid/default/eventgen.conf