topic Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0) in Splunk Enterprise

Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

livehybrid — Mon, 16 Feb 2026 12:52:00 GMT

Good afternoon!

This week we upgraded a Splunk deployment from 9.4.x to 10.0.3, and whilst everything seemingly went well, we came to work the next day to a number of upset users who kept getting random 503 errors and the Oops page:

I was able to replicate this and found that approximately every 5 minutes certain pages in the Splunk UI gave this frustating error! Following further analysis we found that 30 seconds after the 503 error we saw the following error in the _internal logs:

Splunkd daemon is not responding: ('Error connecting to /services/apps/local: The read operation timed out')

After checking directly to https://splunkserver:8089/services/apps/local we found that most of the time it returned in 10-30milliseconds but would periodically take upto 180 seconds to respond.

I later found errors in _internal for the following web address: https://cdn.splunkbase.splunk.com/public/report/apps_dump.json - This environment does not have internet access and has allowInternetAccess=false in server.conf.

Following a chat with some other users in the Community on Slack we found others have had the same issue with different versions - we found that a setting was added in the latest releases (9.4.8/10.0.3/10.2.0) [see https://github.com/livehybrid/splunk-spec-files/blob/master/server.conf#L255] in server.conf/[applicationsManagement]/splunkbaseAppsDumpUrl which contains that SplunkBase URL.

These settings were not in previous 9.4/10.0 releases and I couldnt find any reference to it in the Release Notes either, however settings this to a blank value solved the problem for us.

# server.conf [applicationsManagement] splunkbaseAppsDumpUrl= archivedSplunkbaseAppsDumpUrl=

Ultimately we believe this to be a bug which was introduced in the latest releases and have raised with Support to see if it gets added to the Known Issue for 10.0.3 (https://help.splunk.com/en/splunk-enterprise/release-notes-and-updates/release-notes/10.0/known-issues-for-this-release/known-issues/splunk-enterprise-10.0.3-known-issues)

Hopefully if you come across this issue you can find this fix! 🙂

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

erlingen — Fri, 06 Feb 2026 09:34:13 GMT

Awesome find, thanks for sharing!

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

marcoscala — Mon, 09 Feb 2026 17:55:37 GMT

Hi!

Found same bug and errors upgrading to 9.4.8 (splunk-9.4.8-c543277b24fa-linux-amd64.tgz) on Customer's Environment.

Asked to appli the same fix and let's see if it works there too!

Thanks,

Marco

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

davisb221 — Mon, 09 Feb 2026 18:27:26 GMT

We are experiencing this issue. Splunk support was no help. It'd be nice to have accurate change logs.

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

teunlaan — Tue, 10 Feb 2026 12:56:51 GMT

Great a fix! Thanks you.

Added this post to my splunk support ticket

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

whar_garbl — Tue, 10 Feb 2026 22:06:58 GMT

Great find, I can confirm this solved this (very annoying) problem for me!

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

BenjaminFinck — Wed, 11 Feb 2026 13:58:53 GMT

Can confirm that this fix is working on Splunk Version 9.4.8!

Thanks for sharing this find.

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

joao_amorim — Fri, 13 Feb 2026 16:34:46 GMT

Applied the suggested fix on Splunk 9.3.9, it only converted

ERROR LocalAppsAdminHandler [846555 TcpChannelThread] - Failed to fetch and parse apps dump; aborting splunkbase apps cache update ERROR LocalAppsAdminHandler [846555 TcpChannelThread] - Failed to fetch apps from apps dump error at url=https://cdn.splunkbase.splunk.com/public/report/apps_dump.json code=502 desc=Error connecting: Connect Timeout

To (every time I click on the GUI now)

ERROR LocalAppsAdminHandler [1078631 TcpChannelThread] - Failed to parse apps dump uri from server.conf; aborting splunkbase apps cache update

Do you had the same issue?

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

esplunkuser — Tue, 17 Feb 2026 09:52:40 GMT

Same issue with Splunk 9.2.12

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

tjohnson — Tue, 17 Feb 2026 17:36:57 GMT

I'm seeing the same behavior on Splunk 9.4.8 after upgrading our lower environment.

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

livehybrid — Tue, 17 Feb 2026 18:08:02 GMT

There are now 2 Knowledge base articles about this so its an acknowledged issue with these releases and hopefully patched in the next release (I'll keep an eye out!)

The following KB links might be useful for others who find this, but it pretty much repeats what is in the original post:

https://splunk.my.site.com/customer/s/article/Splunk-Web-503-errors-after-9-4-8-upgrade-on-offline-deployments

https://splunk.my.site.com/customer/s/article/Error-503-Splunkd-daemon-cannot-be-reached-by-Splunk-Web-after-upgrade-to-9-4-8-10-x-versions

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

verbal_666 — Wed, 25 Feb 2026 17:24:37 GMT

Many many thanks again for resolving the issue.
I did the same analysis, and found all those CDN calls with errors, but couldn't figure out how to solve it!

https://community.splunk.com/t5/Splunk-Enterprise/9-3-6-to-9-3-9-frequently-WebUI-timeout/td-p/758767/jump-to/first-unread-message

Great 👍👍👍

PS. quite annoying SPLUNK introduces this really strange bug with 503/delays/timeouts for users 😫

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.4.8/10.2.0)

i_am_me — Tue, 03 Mar 2026 08:00:05 GMT

I did not have 503 errors, but instead 502 errors. Applying the fix here did fix that issue. The UI was also responding very slowly on random moments, the fix also remedied that. Unfortunately, I'm now seeing the "

Failed to parse apps dump uri from server.conf;

error as well. It seems to not affect the speed of the UI though, so for now we're keeping it like this.

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

computermathguy — Thu, 05 Mar 2026 15:16:26 GMT

I just started getting a 502 Proxy error (not a 503) after upgrading from 9.3.8 to 9.3.9. The addition of the below stanza to server.conf did not help.

[applicationsManagement]
splunkbaseAppsDumpUrl=
archivedSplunkbaseAppsDumpUrl=

We are using Apache as an SSL proxy to pass your user's CAC credentials through Splunk.

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: Error reading from remote server

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

computermathguy — Fri, 06 Mar 2026 19:57:40 GMT

Then our users start asking questions about the 502 proxy errors.

Re: Spike in 503 errors in the Splunk WebUI after upgrading to 10.0.3 (also affects 9.2.10/9.3.9/9.4.8/10.2.0)

livehybrid — Fri, 06 Mar 2026 19:59:15 GMT

Hi @computermathguy

This was a very specific 503 error, if you're getting Proxy error 502 then I would recommend creating a new thread so that someone can help look into it as it sounds like a different issue.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing