Deployment Architecture

Why is this indexer unable to get latest bundle from master?

test_splunk15
Explorer

Hi Team,

We are having an issue with Indexer not receiving updated code from master.

I could see when we are pushing code its getting deployed to master and the .bundle is getting created and code is pushed to search heads fine but not to indexers also the .bundle is not persisted in master (utility box). This started happening while we are trying to move Splunk from 7.1 to 7.3.4

Could you please let me know what could be possible wrong?

I have tried checking my puppet code for any errors but there are no errors with :
/opt/splunk/bin/splunk apply cluster-bundle --answer-yes -auth username:password

Post this step we are pushing code to SHs and this is working fine.
/opt/splunk/bin/splunk apply shcluster-bundle --answer-yes -target targerURL -auth username:password

As I mentioned above, during puppet apply (to push latest code running puppet to execute above commands) I see .bundle is getting created but after its applied I dont see the .bundle with latest timestamp is available under master (utility) /opt/splunk/var/run/splunk/cluster/remote-bundle.

I have tried to check the logs under /opt/splunk/var/log/ (splunkd , utility, audit and other logs but nothing concrete I could find), except sometimes getting bundle validation failed (not for each deployment though).

Any suggestions around this please? is this due to upgrade or some other issue?

0 Karma
1 Solution

harsmarvania57
Ultra Champion

How many indexer do you have and your bundle size is >200MB ? Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.4/Indexer/Configurationbundleissues

Check $SPLUNK_HOME/var/log/splunk/splunkd.log on Cluster Master with word CMBundleMgr , CMMaster, CMPeer and you will able to see what is happening during bundle creation, validation, reload/restart.

View solution in original post

harsmarvania57
Ultra Champion

How many indexer do you have and your bundle size is >200MB ? Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.4/Indexer/Configurationbundleissues

Check $SPLUNK_HOME/var/log/splunk/splunkd.log on Cluster Master with word CMBundleMgr , CMMaster, CMPeer and you will able to see what is happening during bundle creation, validation, reload/restart.

test_splunk15
Explorer

This is working now 🙂

After changing max_peers_to_download_bundle value which was 5 in our settings.

0 Karma

test_splunk15
Explorer

Thanks for you help @harsmarvania57

0 Karma

harsmarvania57
Ultra Champion

Great but you need to find out why bundle size increased from 187MB to 697MB after upgrade.

0 Karma

test_splunk15
Explorer

However I will check the Link provided and come back

0 Karma

test_splunk15
Explorer

in this environment we have 4 indexers.

I verified the size, before upgrade it was 187MB (4 days before) now it is showing 697 MB (how can I verify what is being added additionally - may be the issue with TAs which needs verifying?).

I verified logs and see below:

04-01-2020 16:04:49.972 +0100 INFO CMBundleMgr - setting latest bundle= to active bundle=Bundle-ID
04-01-2020 16:04:49.972 +0100 INFO CMBundleMgr - apply bundle status transitioning from='Bundle validation is in progress.', to='None'
04-01-2020 16:04:49.972 +0100 INFO CMRepJob - running job=CMBundleRemoveJob bundle=[id=, path=/$SPLUNK_HOME/var/run/splunk/cluster/remote-bundle/.bundle]
04-01-2020 16:04:51.938 +0100 INFO CMBundleMgr - Removed the untarred bundle folder=/$SPLUNK_HOME/var/run/splunk/cluster/remote-bundle/

One of the older logs I see:

04-01-2020 16:03:02.222 +0100 WARN DistributedBundleReplicationManager - Asynchronous bundle replication to 9 peer(s) succeeded; however it took too long (longer than 10 seconds): elapsed_ms=13911, tar_elapsed_ms= 2019 , for 9 peer(s), bundle_replication_mode="baseline", bundle_file_name=utility-server-1.bundle, bundle_size=155030KB, replication_id=111111111, replication_reason="async replication allowed"

Should I back-out the upgrade and test?

0 Karma

harsmarvania57
Ultra Champion

Have you tried to run /opt/splunk/bin/splunk apply cluster-bundle --answer-yes -auth username:password on Cluster Master manually ?

0 Karma

test_splunk15
Explorer

yes, I have tried manually still it is not working unless I need to move existing .bundle file to bak file and generate new bundle by restarting Splunk service, post this when I am running my puppet code, it is then replicated (with updated bundle).

0 Karma

harsmarvania57
Ultra Champion

And you need to perform same activity during every bundle push ?

0 Karma

test_splunk15
Explorer

It was not like this earlier, We never used to perform this activity, as said this is happening in only one environment where we upgraded Splunk to 7.3.4

Hence thinking whether upgrade is causing issue (current version expires in next 20 days so need to decide a way forward).

Code is updating in master-apps but not pushed to slave-apps, is it only from .bundle or something else I need to check (I am not a Splunk expert though 🙂

0 Karma

test_splunk15
Explorer

yes - for this case but never it was like this. this should be done automatically (though I am not a splunk expert but this is what I have seen). Same is not happening with other environment where my splunk is running with 7.1

Also I just tried pushing code into indexer and see the same, .bundle getting created and removed automatically. Code is updated under master-apps but not been pushed to slave-apps (is id due to .bundle or something else I am missing)?

when I see the cluster status:

master
cluster_status=None
active_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
latest_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
last_validated_bundle
checksum= value2
last_validation_succeeded=1
timestamp=1585742519 (in localtime=Wed Apr 1 13:01:59 2020)

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_ML_Toolkit/default/experiments.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_imperva-waf/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_microsoft-sqlserver/default/sqlserver_dbx2.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_oracle/default/db_input_templates.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_sourcefire/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_squid/default/eventgen.conf

0 Karma

test_splunk15
Explorer

Also I just modified the code and pushed it for indexer.

I see the .bundle created and then removed automatically cluster status command shows below:

master
cluster_status=None
active_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
latest_bundle
checksum= value1
timestamp=1585595012 (in localtime=Mon Mar 30 20:03:32 2020)
last_validated_bundle
checksum= value2
last_validation_succeeded=1
timestamp=1585742519 (in localtime=Wed Apr 1 13:01:59 2020)

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_ML_Toolkit/default/experiments.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_imperva-waf/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_microsoft-sqlserver/default/sqlserver_dbx2.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_oracle/default/db_input_templates.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_sourcefire/default/eventgen.conf

[Not Critical]No spec file for: $Splunk_Home/etc/master-apps/Splunk_TA_squid/default/eventgen.conf

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...