topic Re: Error when pushing cluster bundle to SHs in Splunk Enterprise

Error when pushing cluster bundle to SHs

SplunkNinja — Fri, 20 Feb 2026 19:21:31 GMT

Seeing this error on SHC-D when attempting to push bundle to SHs:

Error while deploying apps to first member, aborting apps deployment to all members: Error while fetching apps baseline on target=https://<IP address of SH Cluster Captain>:8089: Network-layer error: Read Timeout

All servers are running Splunk Enterprise 9.4.8

Re: Error when pushing cluster bundle to SHs

SplunkNinja — Fri, 20 Feb 2026 19:43:21 GMT

Also - it seems that the SHs may be in a rolling restart loop? splunk show shcluster-status --verbose shows the following:

rolling_restart : restart

status : Up -> status : Pending -> status : Up

Re: Error when pushing cluster bundle to SHs

LAME-Creations — Sat, 21 Feb 2026 00:17:50 GMT

I am hoping this is of help, never really ran into this problem, but I would say the fact that you have a constant rolling restart of your search heads is probably not a good thing. So one of the things I had happen to me a long time ago, was I pushed a bad config to my search head cluster and it got stuck in that constant rolling restart.

I ultimately was able to fix it by shutting down the whole cluster, bringing up one machine, getting that config file fixed and then moving onto the next search head. Not sure if you do or do not have a bad config and I am sure this is not a great idea in a production environment.

If the search heads are still working - then you may not want to try this method, but if they aren't working, bringing them all down doesn't technically make things worse.

I wish I had a better answer for what exactly you need to change to get the machines to stop rolling restarting, but maybe my above suggesiton of bringing it down and then restarting it from the CLI and look for any errors in the start up or in the splunk internal logs.

Re: Error when pushing cluster bundle to SHs

PickleRick — Sat, 21 Feb 2026 15:35:50 GMT

Firstly, browse through all member SHs and check if they are indeed in a restart loop. If they are, you gotta dig into the logs why are the SH(s) restarting. But they might not be and the message might be just because of network (or TLS!) problems.

Re: Error when pushing cluster bundle to SHs

SplunkNinja — Tue, 24 Feb 2026 00:10:05 GMT

After reviewing again, the SHs were not restarting. There was an issue with the SH replication port which I since fixed. So now all the SHs show status: up. But I am still having an issue with pushing a bundle. I even removed all the current shcluster apps to a tmp directory and I created a dummy app and tried to push this with no luck.

mkdir -p /opt/splunk/etc/shcluster/apps/_dummy_app/local
echo "[ui]" > /opt/splunk/etc/shcluster/apps/_dummy_app/local/app.conf
echo "is_visible = false" >> /opt/splunk/etc/shcluster/apps/_dummy_app/local/app.conf

sudo chown -R splunk:splunk /opt/splunk

Error while deploying apps to target=https://<SH-02>:8089 with members=3: Error while fetching apps baseline on target=https://<SH-02>:8089: Network-layer error: Read Timeout, Error while fetching apps baseline on target=https://<SH-03>:8089: Network-layer error: Read Timeout

I also do not think this is a network or port issue since I can use this openssl command from the deployer to any of the SHs and I get what looks like a normal response. I can also use the same command on SHs to the other SHs.

openssl s_client -connect <IP address of SH>:8089

Not sure what to try next. Thanks for your help!

Re: Error when pushing cluster bundle to SHs

Zzyzx — Tue, 24 Feb 2026 13:19:29 GMT

Did you ever find a solution to this? Im having the same issue now when pushing sh cluster bundle after upgrading to Splunk 9.4.8.
The SHs are not restarting, status is up and ports are open. Everything appears to be normal.

Im using Ansible to trigger the bundle push, and the ansible job just "hangs" for 10 minutes before returning the error.

Re: Error when pushing cluster bundle to SHs

SplunkNinja — Thu, 26 Feb 2026 22:43:16 GMT

I did not find a fix to this.

Re: Error when pushing cluster bundle to SHs

JohnEGones — Thu, 26 Mar 2026 18:27:04 GMT

I have run into this issue, and support has been struggling a bit here. These all started with the upgrade from 9.4.4 to 9.4.8.

9.4.8 lists this in Known Issues

Splunk WebUI and bundle push are not working correctly on stacks that are unable to fetch application dumps Workaround: Edit $SPLUNK_HOME/etc/system/local/server.conf, and set the below parameters values as blank: [applicationsManagement] splunkbaseAppsDumpUrl= archivedSplunkbaseAppsDumpUrl= Then restart the modified instances (Splunk Web). Clearing those values prevents unreachable calls to the applications dumps and restores UI responsiveness and other functionalities that rely on apps listing.

I'll send an update when I get a firm resolution.

Re: Error when pushing cluster bundle to SHs

JohnEGones — Thu, 26 Mar 2026 19:02:03 GMT

This is the resolution:

Error 503 : Oops Splunkd daemon cannot be reached by Splunk Web after upgrade to 9.4.8/ 10.x versions | Splunk

Resolution The issue is resolved after updating the server.conf configuration, by leaving the two added parameter values as blank. [applicationsManagement] splunkbaseAppsDumpUrl= archivedSplunkbaseAppsDumpUrl= Fix is included in the following versions of Splunk Enterprise: 9.3.11, 9.4.10,10.0.5,10.2.2

Re: Error when pushing cluster bundle to SHs

SplunkNinja — Fri, 27 Mar 2026 15:30:33 GMT

@PickleRick @JohnEGones @Zzyzx @LAME-Creations I was able to follow this KB and it helped solve the issue for me

Deployer fails to push a bundle to Search Head Cluster Peers in 9.4.1 (default apps)

https://splunk.my.site.com/customer/s/article/Deployer-fails-to-push-a-bundle-to-Search-Head-Cluster-Peers-in-9-4-1

Changing the command arguments as per this Splunk KB Article prompted the Deployer to create a fresh staging directory, bypassing the stale/corrupt one that was causing the error.

$SPLUNK_HOME/bin/splunk apply shcluster-bundle -target <URI>:<management_port> -push-default-apps true

Also - I could have run this command to clear out the old/bad bundle...

rm -rf $SPLUNK_HOME/var/run/splunk/deploy*

Re: Error when pushing cluster bundle to SHs

SplunkNinja — Fri, 27 Mar 2026 15:20:56 GMT

See my latest message. Also - I could have run this command to clear out the old/bad bundle...

rm -rf $SPLUNK_HOME/var/run/splunk/deploy*